2024-09-18T13:43:47.919 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-18T13:43:47.933 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-18T13:43:48.003 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911065 branch: wip-bharath11-testing-2024-09-18-1115-quincy description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd} email: skanta@redhat.com first_in_suite: false job_id: '7911065' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath11-testing-2024-09-18-1115-quincy ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: enable_coprs: - ceph/el9 flavor: default sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 workunit: branch: wip-bharath11-testing-2024-09-18-1115-quincy sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 4631 sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath11-testing-2024-09-18-1115-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 targets: smithi047.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC/gh6T1e3/dJuGfz4vvHscW0EYQZD0X89UHZdVCnwW3buXi/XSIom4c2HoDbQrSm5K7UdpvDA5U1kY2Eg1EAd4= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-18_09:13:00 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-18T13:43:48.004 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa; will attempt to use it 2024-09-18T13:43:48.004 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa/tasks 2024-09-18T13:43:48.005 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-18T13:43:48.006 INFO:teuthology.task.internal:Checking packages... 2024-09-18T13:43:48.029 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '7061ea6ca50908d104e9f3979dc615e118d9e153' 2024-09-18T13:43:48.029 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-18T13:43:48.029 INFO:teuthology.packaging:ref: None 2024-09-18T13:43:48.029 INFO:teuthology.packaging:tag: None 2024-09-18T13:43:48.030 INFO:teuthology.packaging:branch: wip-bharath11-testing-2024-09-18-1115-quincy 2024-09-18T13:43:48.030 INFO:teuthology.packaging:sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T13:43:48.030 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath11-testing-2024-09-18-1115-quincy 2024-09-18T13:43:48.176 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1645.g7061ea6c 2024-09-18T13:43:48.178 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-18T13:43:48.187 INFO:teuthology.task.internal:no buildpackages task found 2024-09-18T13:43:48.187 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-18T13:43:48.227 INFO:teuthology.task.internal:Saving configuration 2024-09-18T13:43:48.237 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-18T13:43:48.261 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-18T13:43:48.283 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi047.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911065', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-18 13:39:08.894495', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC/gh6T1e3/dJuGfz4vvHscW0EYQZD0X89UHZdVCnwW3buXi/XSIom4c2HoDbQrSm5K7UdpvDA5U1kY2Eg1EAd4='} 2024-09-18T13:43:48.283 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-18T13:43:48.295 INFO:teuthology.task.internal:roles: ubuntu@smithi047.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-18T13:43:48.295 INFO:teuthology.run_tasks:Running task console_log... 2024-09-18T13:43:48.381 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fda793bfac0>, signals=[15]) 2024-09-18T13:43:48.381 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-18T13:43:48.392 INFO:teuthology.task.internal:Opening connections... 2024-09-18T13:43:48.392 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi047.front.sepia.ceph.com 2024-09-18T13:43:48.394 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T13:43:48.471 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-18T13:43:48.500 DEBUG:teuthology.orchestra.run.smithi047:> uname -m 2024-09-18T13:43:48.518 INFO:teuthology.orchestra.run.smithi047.stdout:x86_64 2024-09-18T13:43:48.518 DEBUG:teuthology.orchestra.run.smithi047:> cat /etc/os-release 2024-09-18T13:43:48.575 INFO:teuthology.orchestra.run.smithi047.stdout:NAME="CentOS Stream" 2024-09-18T13:43:48.575 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION="9" 2024-09-18T13:43:48.575 INFO:teuthology.orchestra.run.smithi047.stdout:ID="centos" 2024-09-18T13:43:48.575 INFO:teuthology.orchestra.run.smithi047.stdout:ID_LIKE="rhel fedora" 2024-09-18T13:43:48.575 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION_ID="9" 2024-09-18T13:43:48.575 INFO:teuthology.orchestra.run.smithi047.stdout:PLATFORM_ID="platform:el9" 2024-09-18T13:43:48.575 INFO:teuthology.orchestra.run.smithi047.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-18T13:43:48.576 INFO:teuthology.orchestra.run.smithi047.stdout:ANSI_COLOR="0;31" 2024-09-18T13:43:48.576 INFO:teuthology.orchestra.run.smithi047.stdout:LOGO="fedora-logo-icon" 2024-09-18T13:43:48.576 INFO:teuthology.orchestra.run.smithi047.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-18T13:43:48.576 INFO:teuthology.orchestra.run.smithi047.stdout:HOME_URL="https://centos.org/" 2024-09-18T13:43:48.576 INFO:teuthology.orchestra.run.smithi047.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-18T13:43:48.576 INFO:teuthology.orchestra.run.smithi047.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-18T13:43:48.576 INFO:teuthology.orchestra.run.smithi047.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-18T13:43:48.577 INFO:teuthology.lock.ops:Updating smithi047.front.sepia.ceph.com on lock server 2024-09-18T13:43:48.600 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-18T13:43:48.609 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-18T13:43:48.651 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-18T13:43:48.651 DEBUG:teuthology.orchestra.run.smithi047:> test '!' -e /home/ubuntu/cephtest 2024-09-18T13:43:48.667 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-18T13:43:48.686 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-18T13:43:48.686 DEBUG:teuthology.orchestra.run.smithi047:> test -z $(ls -A /var/lib/ceph) 2024-09-18T13:43:48.731 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-18T13:43:48.762 INFO:teuthology.run_tasks:Running task kernel... 2024-09-18T13:43:48.832 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-18T13:43:48.832 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-18T13:43:48.832 DEBUG:teuthology.orchestra.run.smithi047:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-18T13:43:48.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T13:43:48.848 DEBUG:teuthology.orchestra.run.smithi047:> uname -r 2024-09-18T13:43:48.905 INFO:teuthology.orchestra.run.smithi047.stdout:5.14.0-503.el9.x86_64 2024-09-18T13:43:48.906 INFO:teuthology.task.kernel:Running kernel on smithi047: 5.14.0-503.el9.x86_64 2024-09-18T13:43:48.906 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum install -y kernel 2024-09-18T13:43:52.876 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - BaseOS 3.6 MB/s | 8.3 MB 00:02 2024-09-18T13:43:54.869 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - AppStream 23 MB/s | 20 MB 00:00 2024-09-18T13:44:01.127 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - CRB 2.8 MB/s | 6.5 MB 00:02 2024-09-18T13:44:02.720 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 19 kB 00:00 2024-09-18T13:44:04.007 INFO:teuthology.orchestra.run.smithi047.stdout:Extra Packages for Enterprise Linux 19 MB/s | 23 MB 00:01 2024-09-18T13:44:09.672 INFO:teuthology.orchestra.run.smithi047.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-18T13:44:11.645 INFO:teuthology.orchestra.run.smithi047.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-18T13:44:11.645 INFO:teuthology.orchestra.run.smithi047.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-18T13:44:11.731 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-09-18T13:44:11.749 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:44:11.749 INFO:teuthology.orchestra.run.smithi047.stdout: Package Architecture Version Repository Size 2024-09-18T13:44:11.749 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:44:11.749 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-09-18T13:44:11.749 INFO:teuthology.orchestra.run.smithi047.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-18T13:44:11.749 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-09-18T13:44:11.749 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-18T13:44:11.749 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-18T13:44:11.750 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-18T13:44:11.750 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:44:11.750 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-09-18T13:44:11.750 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:44:11.750 INFO:teuthology.orchestra.run.smithi047.stdout:Install 4 Packages 2024-09-18T13:44:11.750 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:44:11.750 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 79 M 2024-09-18T13:44:11.751 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 126 M 2024-09-18T13:44:11.751 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-09-18T13:44:12.148 INFO:teuthology.orchestra.run.smithi047.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 218 kB/s | 36 kB 00:00 2024-09-18T13:44:13.490 INFO:teuthology.orchestra.run.smithi047.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 10 MB/s | 16 MB 00:01 2024-09-18T13:44:13.932 INFO:teuthology.orchestra.run.smithi047.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 16 MB/s | 29 MB 00:01 2024-09-18T13:44:14.932 INFO:teuthology.orchestra.run.smithi047.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 12 MB/s | 35 MB 00:02 2024-09-18T13:44:14.932 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-09-18T13:44:14.932 INFO:teuthology.orchestra.run.smithi047.stdout:Total 25 MB/s | 79 MB 00:03 2024-09-18T13:44:15.645 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-09-18T13:44:16.013 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-09-18T13:44:16.014 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-09-18T13:44:16.995 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-09-18T13:44:16.995 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-09-18T13:44:19.555 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-09-18T13:44:20.246 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-18T13:44:20.342 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-18T13:44:23.094 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-18T13:44:23.218 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-18T13:44:28.339 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-18T13:44:28.377 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-18T13:44:33.513 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-18T13:45:12.528 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-18T13:45:12.568 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-18T13:45:12.925 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-18T13:45:12.925 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-18T13:45:12.925 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-18T13:45:12.925 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-18T13:45:13.806 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-18T13:45:13.807 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:45:13.807 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-09-18T13:45:13.807 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-18T13:45:13.807 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-18T13:45:13.807 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-18T13:45:13.807 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-18T13:45:13.807 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:45:13.807 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-09-18T13:45:14.146 DEBUG:teuthology.orchestra.run.smithi047:> rpm -q kernel | sort -rV | head -n 1 2024-09-18T13:45:14.229 INFO:teuthology.orchestra.run.smithi047.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-18T13:45:14.230 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-18T13:45:14.230 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-18T13:45:14.230 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-18T13:45:14.230 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-18T13:45:14.230 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-18T13:45:14.230 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi047.front.sepia.ceph.com, path=None, version=distro) 2024-09-18T13:45:14.230 DEBUG:teuthology.orchestra.run.smithi047:> rpm -q kernel | sort -rV | head -n 1 2024-09-18T13:45:14.289 INFO:teuthology.orchestra.run.smithi047.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-18T13:45:14.289 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-18T13:45:14.289 DEBUG:teuthology.orchestra.run.smithi047:> sudo rpm -qi grub2-tools 2024-09-18T13:45:14.322 INFO:teuthology.orchestra.run.smithi047.stdout:Name : grub2-tools 2024-09-18T13:45:14.322 INFO:teuthology.orchestra.run.smithi047.stdout:Epoch : 1 2024-09-18T13:45:14.322 INFO:teuthology.orchestra.run.smithi047.stdout:Version : 2.06 2024-09-18T13:45:14.322 INFO:teuthology.orchestra.run.smithi047.stdout:Release : 82.el9 2024-09-18T13:45:14.322 INFO:teuthology.orchestra.run.smithi047.stdout:Architecture: x86_64 2024-09-18T13:45:14.322 INFO:teuthology.orchestra.run.smithi047.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-18T13:45:14.323 INFO:teuthology.orchestra.run.smithi047.stdout:Group : Unspecified 2024-09-18T13:45:14.323 INFO:teuthology.orchestra.run.smithi047.stdout:Size : 8274670 2024-09-18T13:45:14.323 INFO:teuthology.orchestra.run.smithi047.stdout:License : GPLv3+ 2024-09-18T13:45:14.323 INFO:teuthology.orchestra.run.smithi047.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-18T13:45:14.323 INFO:teuthology.orchestra.run.smithi047.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-18T13:45:14.323 INFO:teuthology.orchestra.run.smithi047.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-18T13:45:14.323 INFO:teuthology.orchestra.run.smithi047.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-18T13:45:14.323 INFO:teuthology.orchestra.run.smithi047.stdout:Packager : builder@centos.org 2024-09-18T13:45:14.323 INFO:teuthology.orchestra.run.smithi047.stdout:Vendor : CentOS 2024-09-18T13:45:14.323 INFO:teuthology.orchestra.run.smithi047.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-18T13:45:14.323 INFO:teuthology.orchestra.run.smithi047.stdout:Summary : Support tools for GRUB. 2024-09-18T13:45:14.323 INFO:teuthology.orchestra.run.smithi047.stdout:Description : 2024-09-18T13:45:14.323 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:45:14.324 INFO:teuthology.orchestra.run.smithi047.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-18T13:45:14.324 INFO:teuthology.orchestra.run.smithi047.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-18T13:45:14.324 INFO:teuthology.orchestra.run.smithi047.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-18T13:45:14.324 INFO:teuthology.orchestra.run.smithi047.stdout:hardware devices. 2024-09-18T13:45:14.324 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:45:14.324 INFO:teuthology.orchestra.run.smithi047.stdout:This subpackage provides tools for support of all platforms. 2024-09-18T13:45:14.325 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-18T13:45:14.325 INFO:teuthology.task.kernel:Updating grub on smithi047 to boot 5.14.0-508.el9.x86_64 2024-09-18T13:45:14.325 DEBUG:teuthology.orchestra.run.smithi047:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-18T13:45:15.024 INFO:teuthology.orchestra.run.smithi047.stderr:Generating grub configuration file ... 2024-09-18T13:45:16.369 INFO:teuthology.orchestra.run.smithi047.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-18T13:45:16.397 INFO:teuthology.orchestra.run.smithi047.stderr:done 2024-09-18T13:45:16.399 DEBUG:teuthology.orchestra.run.smithi047:> mktemp 2024-09-18T13:45:16.415 INFO:teuthology.orchestra.run.smithi047.stdout:/tmp/tmp.zx0MXi7pIl 2024-09-18T13:45:16.415 DEBUG:teuthology.orchestra.run.smithi047:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.zx0MXi7pIl 2024-09-18T13:45:16.486 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 0666 /tmp/tmp.zx0MXi7pIl 2024-09-18T13:45:16.623 DEBUG:teuthology.orchestra.remote:smithi047:/tmp/tmp.zx0MXi7pIl is 6KB 2024-09-18T13:45:16.672 DEBUG:teuthology.orchestra.run.smithi047:> rm -fr /tmp/tmp.zx0MXi7pIl 2024-09-18T13:45:16.687 DEBUG:teuthology.orchestra.run.smithi047:> sudo /bin/ls /boot/loader/entries || true 2024-09-18T13:45:16.748 INFO:teuthology.orchestra.run.smithi047.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-18T13:45:16.748 INFO:teuthology.orchestra.run.smithi047.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-18T13:45:16.749 INFO:teuthology.orchestra.run.smithi047.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-18T13:45:16.749 INFO:teuthology.orchestra.run.smithi047.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-18T13:45:16.750 DEBUG:teuthology.orchestra.run.smithi047:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-18T13:45:16.911 DEBUG:teuthology.orchestra.run.smithi047:> sudo shutdown -r now 2024-09-18T13:45:46.942 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-18T13:45:46.943 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-09-18T13:45:46.944 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T13:46:05.501 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.47 2024-09-18T13:46:14.509 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-09-18T13:46:14.510 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T13:47:14.570 DEBUG:teuthology.orchestra.remote:timed out 2024-09-18T13:47:26.581 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-09-18T13:47:26.582 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T13:47:26.945 DEBUG:teuthology.orchestra.run.smithi047:> true 2024-09-18T13:47:27.322 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-09-18T13:47:27.322 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-18T13:47:27.323 DEBUG:teuthology.orchestra.run.smithi047:> uname -r 2024-09-18T13:47:27.339 INFO:teuthology.orchestra.run.smithi047.stdout:5.14.0-508.el9.x86_64 2024-09-18T13:47:27.339 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-18T13:47:27.339 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-18T13:47:27.339 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-18T13:47:28.340 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-18T13:47:28.340 DEBUG:teuthology.orchestra.run.smithi047:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-18T13:47:28.408 INFO:teuthology.orchestra.run.smithi047.stdout:ttyS1 2024-09-18T13:47:28.430 DEBUG:teuthology.parallel:result is None 2024-09-18T13:47:28.431 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-18T13:47:28.439 INFO:teuthology.task.internal:Creating test directory... 2024-09-18T13:47:28.440 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-18T13:47:28.458 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-18T13:47:28.614 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-18T13:47:28.656 INFO:teuthology.task.internal:Creating archive directory... 2024-09-18T13:47:28.656 DEBUG:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-18T13:47:28.689 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-18T13:47:28.697 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-18T13:47:28.697 DEBUG:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-18T13:47:28.762 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T13:47:28.774 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T13:47:28.776 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-18T13:47:28.783 INFO:teuthology.task.internal:Configuring sudo... 2024-09-18T13:47:28.784 DEBUG:teuthology.orchestra.run.smithi047:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-18T13:47:28.845 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-18T13:47:28.855 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-18T13:47:28.856 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-18T13:47:28.902 DEBUG:teuthology.orchestra.run.smithi047:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-18T13:47:28.983 DEBUG:teuthology.orchestra.run.smithi047:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-18T13:47:29.042 DEBUG:teuthology.orchestra.run.smithi047:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-18T13:47:29.120 DEBUG:teuthology.orchestra.run.smithi047:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-18T13:47:29.168 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-09-18T13:47:29.168 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-18T13:47:29.235 DEBUG:teuthology.orchestra.run.smithi047:> sudo service rsyslog restart 2024-09-18T13:47:29.322 INFO:teuthology.orchestra.run.smithi047.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-18T13:47:29.771 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-18T13:47:29.779 INFO:teuthology.task.internal:Starting timer... 2024-09-18T13:47:29.779 INFO:teuthology.run_tasks:Running task pcp... 2024-09-18T13:47:29.826 INFO:teuthology.run_tasks:Running task selinux... 2024-09-18T13:47:29.859 DEBUG:teuthology.orchestra.run.smithi047:> sudo service auditd rotate 2024-09-18T13:47:29.924 INFO:teuthology.orchestra.run.smithi047.stdout:Rotating logs: 2024-09-18T13:47:29.926 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-18T13:47:29.926 DEBUG:teuthology.orchestra.run.smithi047:> /usr/sbin/getenforce 2024-09-18T13:47:29.958 INFO:teuthology.orchestra.run.smithi047.stdout:Permissive 2024-09-18T13:47:29.959 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi047.front.sepia.ceph.com': 'permissive'} 2024-09-18T13:47:29.959 DEBUG:teuthology.orchestra.run.smithi047:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-18T13:47:30.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T13:47:30.027 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-18T13:47:30.027 DEBUG:teuthology.orchestra.run.smithi047:> sudo /usr/sbin/setenforce permissive 2024-09-18T13:47:30.101 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-18T13:47:30.110 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-18T13:47:30.113 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-18T13:47:30.242 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-18T13:47:30.253 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-18T13:47:30.254 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi047.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-18T13:51:17.531 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi047.front.sepia.ceph.com')] 2024-09-18T13:51:17.532 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-09-18T13:51:17.534 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T13:51:17.612 DEBUG:teuthology.orchestra.run.smithi047:> true 2024-09-18T13:51:17.701 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-09-18T13:51:17.701 INFO:teuthology.run_tasks:Running task clock... 2024-09-18T13:51:17.712 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-18T13:51:17.713 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-18T13:51:17.713 DEBUG:teuthology.orchestra.run.smithi047:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T13:51:17.771 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-18T13:51:17.787 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-18T13:51:17.813 INFO:teuthology.orchestra.run.smithi047.stderr:sudo: ntpd: command not found 2024-09-18T13:51:17.824 INFO:teuthology.orchestra.run.smithi047.stdout:506 Cannot talk to daemon 2024-09-18T13:51:17.839 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-18T13:51:17.854 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-18T13:51:17.917 INFO:teuthology.orchestra.run.smithi047.stderr:bash: line 1: ntpq: command not found 2024-09-18T13:51:17.920 INFO:teuthology.orchestra.run.smithi047.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-18T13:51:17.920 INFO:teuthology.orchestra.run.smithi047.stdout:=============================================================================== 2024-09-18T13:51:17.920 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T13:51:17.920 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T13:51:17.920 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T13:51:17.920 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T13:51:17.921 INFO:teuthology.run_tasks:Running task install... 2024-09-18T13:51:17.933 DEBUG:teuthology.task.install:project ceph 2024-09-18T13:51:17.933 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'}} 2024-09-18T13:51:17.933 DEBUG:teuthology.task.install:config {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'} 2024-09-18T13:51:17.933 INFO:teuthology.task.install:Using flavor: default 2024-09-18T13:51:17.941 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-18T13:51:17.941 INFO:teuthology.task.install:extra packages: [] 2024-09-18T13:51:17.942 DEBUG:teuthology.orchestra.run.smithi047:> sudo dnf -y install 'dnf-command(copr)' 2024-09-18T13:51:18.563 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:00:44 ago on Wed 18 Sep 2024 01:50:34 PM UTC. 2024-09-18T13:51:18.652 INFO:teuthology.orchestra.run.smithi047.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-09-18T13:51:18.697 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-09-18T13:51:18.698 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-09-18T13:51:18.698 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-09-18T13:51:18.756 DEBUG:teuthology.orchestra.run.smithi047:> sudo dnf -y copr enable ceph/el9 2024-09-18T13:51:18.935 INFO:teuthology.orchestra.run.smithi047.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-09-18T13:51:18.935 INFO:teuthology.orchestra.run.smithi047.stderr:of the main distribution, and quality may vary. 2024-09-18T13:51:18.935 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-18T13:51:18.935 INFO:teuthology.orchestra.run.smithi047.stderr:The Fedora Project does not exercise any power over the contents of 2024-09-18T13:51:18.935 INFO:teuthology.orchestra.run.smithi047.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-09-18T13:51:18.935 INFO:teuthology.orchestra.run.smithi047.stderr:, 2024-09-18T13:51:18.935 INFO:teuthology.orchestra.run.smithi047.stderr:and packages are not held to any quality or security level. 2024-09-18T13:51:18.935 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-18T13:51:18.936 INFO:teuthology.orchestra.run.smithi047.stderr:Please do not file bug reports about these packages in Fedora 2024-09-18T13:51:18.936 INFO:teuthology.orchestra.run.smithi047.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-09-18T13:51:19.008 INFO:teuthology.orchestra.run.smithi047.stdout:Repository successfully enabled. 2024-09-18T13:51:19.044 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153', 'tag': None, 'wait_for_package': False} 2024-09-18T13:51:19.044 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T13:51:19.192 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/centos/9/flavors/default/ 2024-09-18T13:51:19.192 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1645.g7061ea6c 2024-09-18T13:51:19.307 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-18T13:51:19.308 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-09-18T13:51:19.308 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-18T13:51:19.335 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-18T13:51:19.335 DEBUG:teuthology.orchestra.run.smithi047:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/7061ea6ca50908d104e9f3979dc615e118d9e153/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-18T13:51:19.403 DEBUG:teuthology.orchestra.run.smithi047:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-18T13:51:19.481 DEBUG:teuthology.orchestra.run.smithi047:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-18T13:51:19.547 INFO:teuthology.orchestra.run.smithi047.stdout:check_obsoletes = 1 2024-09-18T13:51:19.548 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum clean all 2024-09-18T13:51:19.784 INFO:teuthology.orchestra.run.smithi047.stdout:45 files removed 2024-09-18T13:51:19.811 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-radosgw 2024-09-18T13:51:20.377 INFO:teuthology.orchestra.run.smithi047.stdout:ceph packages for x86_64 291 kB/s | 77 kB 00:00 2024-09-18T13:51:20.652 INFO:teuthology.orchestra.run.smithi047.stdout:ceph noarch packages 47 kB/s | 11 kB 00:00 2024-09-18T13:51:20.911 INFO:teuthology.orchestra.run.smithi047.stdout:ceph source packages 8.1 kB/s | 1.8 kB 00:00 2024-09-18T13:51:21.112 INFO:teuthology.orchestra.run.smithi047.stdout:Copr repo for el9 owned by ceph 47 kB/s | 7.6 kB 00:00 2024-09-18T13:51:22.912 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.3 MB 00:01 2024-09-18T13:51:24.713 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - AppStream 30 MB/s | 20 MB 00:00 2024-09-18T13:51:32.372 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - CRB 1.7 MB/s | 6.5 MB 00:03 2024-09-18T13:51:34.689 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 19 kB 00:01 2024-09-18T13:51:36.327 INFO:teuthology.orchestra.run.smithi047.stdout:Extra Packages for Enterprise Linux 15 MB/s | 23 MB 00:01 2024-09-18T13:51:41.982 INFO:teuthology.orchestra.run.smithi047.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-18T13:51:44.443 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-09-18T13:51:44.457 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:51:44.457 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-09-18T13:51:44.457 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:51:44.457 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-09-18T13:51:44.457 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-radosgw x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 14 M 2024-09-18T13:51:44.457 INFO:teuthology.orchestra.run.smithi047.stdout:Upgrading: 2024-09-18T13:51:44.458 INFO:teuthology.orchestra.run.smithi047.stdout: librados2 x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 3.1 M 2024-09-18T13:51:44.458 INFO:teuthology.orchestra.run.smithi047.stdout: librbd1 x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 2.9 M 2024-09-18T13:51:44.458 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-09-18T13:51:44.458 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-base x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 5.0 M 2024-09-18T13:51:44.458 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-common x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 22 M 2024-09-18T13:51:44.458 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-selinux x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 25 k 2024-09-18T13:51:44.458 INFO:teuthology.orchestra.run.smithi047.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-18T13:51:44.458 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs2 x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 637 k 2024-09-18T13:51:44.458 INFO:teuthology.orchestra.run.smithi047.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-18T13:51:44.458 INFO:teuthology.orchestra.run.smithi047.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-18T13:51:44.458 INFO:teuthology.orchestra.run.smithi047.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-18T13:51:44.458 INFO:teuthology.orchestra.run.smithi047.stdout: libradosstriper1 x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 458 k 2024-09-18T13:51:44.458 INFO:teuthology.orchestra.run.smithi047.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-18T13:51:44.459 INFO:teuthology.orchestra.run.smithi047.stdout: librgw2 x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 4.9 M 2024-09-18T13:51:44.459 INFO:teuthology.orchestra.run.smithi047.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-18T13:51:44.459 INFO:teuthology.orchestra.run.smithi047.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-18T13:51:44.459 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-argparse x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 45 k 2024-09-18T13:51:44.459 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-common x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 120 k 2024-09-18T13:51:44.459 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cephfs x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 156 k 2024-09-18T13:51:44.459 INFO:teuthology.orchestra.run.smithi047.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-18T13:51:44.459 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rados x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 320 k 2024-09-18T13:51:44.459 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rbd x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 296 k 2024-09-18T13:51:44.460 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rgw x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 99 k 2024-09-18T13:51:44.460 INFO:teuthology.orchestra.run.smithi047.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-18T13:51:44.460 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:51:44.460 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-09-18T13:51:44.460 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:51:44.460 INFO:teuthology.orchestra.run.smithi047.stdout:Install 22 Packages 2024-09-18T13:51:44.460 INFO:teuthology.orchestra.run.smithi047.stdout:Upgrade 2 Packages 2024-09-18T13:51:44.460 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:51:44.460 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 58 M 2024-09-18T13:51:44.461 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-09-18T13:51:45.201 INFO:teuthology.orchestra.run.smithi047.stdout:(1/24): ceph-base-17.2.7-1645.g7061ea6c.el9.x86 7.9 MB/s | 5.0 MB 00:00 2024-09-18T13:51:45.259 INFO:teuthology.orchestra.run.smithi047.stdout:(2/24): ceph-selinux-17.2.7-1645.g7061ea6c.el9. 432 kB/s | 25 kB 00:00 2024-09-18T13:51:45.343 INFO:teuthology.orchestra.run.smithi047.stdout:(3/24): libcephfs2-17.2.7-1645.g7061ea6c.el9.x8 7.5 MB/s | 637 kB 00:00 2024-09-18T13:51:45.443 INFO:teuthology.orchestra.run.smithi047.stdout:(4/24): libradosstriper1-17.2.7-1645.g7061ea6c. 4.5 MB/s | 458 kB 00:00 2024-09-18T13:51:45.585 INFO:teuthology.orchestra.run.smithi047.stdout:(5/24): ceph-radosgw-17.2.7-1645.g7061ea6c.el9. 14 MB/s | 14 MB 00:01 2024-09-18T13:51:45.627 INFO:teuthology.orchestra.run.smithi047.stdout:(6/24): python3-ceph-argparse-17.2.7-1645.g7061 1.1 MB/s | 45 kB 00:00 2024-09-18T13:51:45.852 INFO:teuthology.orchestra.run.smithi047.stdout:(7/24): ceph-common-17.2.7-1645.g7061ea6c.el9.x 18 MB/s | 22 MB 00:01 2024-09-18T13:51:45.878 INFO:teuthology.orchestra.run.smithi047.stdout:(8/24): python3-ceph-common-17.2.7-1645.g7061ea 480 kB/s | 120 kB 00:00 2024-09-18T13:51:45.953 INFO:teuthology.orchestra.run.smithi047.stdout:(9/24): librgw2-17.2.7-1645.g7061ea6c.el9.x86_6 9.7 MB/s | 4.9 MB 00:00 2024-09-18T13:51:45.978 INFO:teuthology.orchestra.run.smithi047.stdout:(10/24): python3-cephfs-17.2.7-1645.g7061ea6c.e 1.2 MB/s | 156 kB 00:00 2024-09-18T13:51:46.003 INFO:teuthology.orchestra.run.smithi047.stdout:(11/24): python3-rados-17.2.7-1645.g7061ea6c.el 2.5 MB/s | 320 kB 00:00 2024-09-18T13:51:46.028 INFO:teuthology.orchestra.run.smithi047.stdout:(12/24): python3-rbd-17.2.7-1645.g7061ea6c.el9. 3.8 MB/s | 296 kB 00:00 2024-09-18T13:51:46.045 INFO:teuthology.orchestra.run.smithi047.stdout:(13/24): python3-rgw-17.2.7-1645.g7061ea6c.el9. 1.4 MB/s | 99 kB 00:00 2024-09-18T13:51:46.112 INFO:teuthology.orchestra.run.smithi047.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 542 kB/s | 45 kB 00:00 2024-09-18T13:51:46.154 INFO:teuthology.orchestra.run.smithi047.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-09-18T13:51:46.196 INFO:teuthology.orchestra.run.smithi047.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-18T13:51:46.221 INFO:teuthology.orchestra.run.smithi047.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-09-18T13:51:46.255 INFO:teuthology.orchestra.run.smithi047.stdout:(18/24): librdkafka-1.6.1-102.el9.x86_64.rpm 3.1 MB/s | 662 kB 00:00 2024-09-18T13:51:46.314 INFO:teuthology.orchestra.run.smithi047.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 529 kB/s | 49 kB 00:00 2024-09-18T13:51:46.347 INFO:teuthology.orchestra.run.smithi047.stdout:(20/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.0 MB/s | 308 kB 00:00 2024-09-18T13:51:46.372 INFO:teuthology.orchestra.run.smithi047.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 666 kB/s | 67 kB 00:00 2024-09-18T13:51:46.439 INFO:teuthology.orchestra.run.smithi047.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-09-18T13:51:46.523 INFO:teuthology.orchestra.run.smithi047.stdout:(23/24): librados2-17.2.7-1645.g7061ea6c.el9.x8 18 MB/s | 3.1 MB 00:00 2024-09-18T13:51:46.581 INFO:teuthology.orchestra.run.smithi047.stdout:(24/24): librbd1-17.2.7-1645.g7061ea6c.el9.x86_ 14 MB/s | 2.9 MB 00:00 2024-09-18T13:51:46.587 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-09-18T13:51:46.587 INFO:teuthology.orchestra.run.smithi047.stdout:Total 27 MB/s | 58 MB 00:02 2024-09-18T13:51:46.650 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-09-18T13:51:46.688 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-09-18T13:51:46.688 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-09-18T13:51:47.369 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-09-18T13:51:47.369 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-09-18T13:51:48.661 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-09-18T13:51:49.071 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-18T13:51:49.134 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : librados2-2:17.2.7-1645.g7061ea6c.el9.x86_64 2/26 2024-09-18T13:51:49.235 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librados2-2:17.2.7-1645.g7061ea6c.el9.x86_64 2/26 2024-09-18T13:51:49.324 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rados-2:17.2.7-1645.g7061ea6c.el9.x86_64 3/26 2024-09-18T13:51:49.352 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libcephfs2-2:17.2.7-1645.g7061ea6c.el9.x86_64 4/26 2024-09-18T13:51:49.441 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libcephfs2-2:17.2.7-1645.g7061ea6c.el9.x86_64 4/26 2024-09-18T13:51:49.496 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-18T13:51:49.532 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-18T13:51:49.994 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-18T13:51:50.030 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librgw2-2:17.2.7-1645.g7061ea6c.el9.x86_64 8/26 2024-09-18T13:51:50.076 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librgw2-2:17.2.7-1645.g7061ea6c.el9.x86_64 8/26 2024-09-18T13:51:50.113 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-18T13:51:50.156 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-ceph-argparse-2:17.2.7-1645.g7061ea6c.el9. 10/26 2024-09-18T13:51:50.194 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cephfs-2:17.2.7-1645.g7061ea6c.el9.x86_64 11/26 2024-09-18T13:51:50.257 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 12/26 2024-09-18T13:51:50.283 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libradosstriper1-2:17.2.7-1645.g7061ea6c.el9.x86_6 13/26 2024-09-18T13:51:50.332 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1645.g7061ea6c.el9.x86_6 13/26 2024-09-18T13:51:50.372 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-18T13:51:50.408 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-18T13:51:50.441 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-18T13:51:50.750 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-18T13:51:50.795 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : librbd1-2:17.2.7-1645.g7061ea6c.el9.x86_64 18/26 2024-09-18T13:51:50.886 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librbd1-2:17.2.7-1645.g7061ea6c.el9.x86_64 18/26 2024-09-18T13:51:50.962 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 19/26 2024-09-18T13:51:50.997 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-ceph-common-2:17.2.7-1645.g7061ea6c.el9.x8 20/26 2024-09-18T13:51:53.597 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-common-2:17.2.7-1645.g7061ea6c.el9.x86_64 21/26 2024-09-18T13:51:53.636 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-common-2:17.2.7-1645.g7061ea6c.el9.x86_64 21/26 2024-09-18T13:51:54.177 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-common-2:17.2.7-1645.g7061ea6c.el9.x86_64 21/26 2024-09-18T13:51:54.222 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-base-2:17.2.7-1645.g7061ea6c.el9.x86_64 22/26 2024-09-18T13:51:54.319 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-base-2:17.2.7-1645.g7061ea6c.el9.x86_64 22/26 2024-09-18T13:51:54.319 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-18T13:51:54.319 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-18T13:51:54.319 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:51:54.348 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-selinux-2:17.2.7-1645.g7061ea6c.el9.x86_64 23/26 2024-09-18T13:52:04.794 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1645.g7061ea6c.el9.x86_64 23/26 2024-09-18T13:52:04.794 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /sys 2024-09-18T13:52:04.794 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /proc 2024-09-18T13:52:04.795 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /mnt 2024-09-18T13:52:04.795 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /var/tmp 2024-09-18T13:52:04.795 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /home 2024-09-18T13:52:04.795 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /root 2024-09-18T13:52:04.795 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /tmp 2024-09-18T13:52:04.795 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:52:06.167 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-radosgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 24/26 2024-09-18T13:52:06.247 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 24/26 2024-09-18T13:52:06.247 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T13:52:06.248 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-18T13:52:06.248 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-18T13:52:06.248 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-18T13:52:06.248 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:52:06.249 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-18T13:52:06.337 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-18T13:52:06.337 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-18T13:52:07.575 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-18T13:52:07.575 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-base-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/26 2024-09-18T13:52:07.575 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-common-2:17.2.7-1645.g7061ea6c.el9.x86_64 2/26 2024-09-18T13:52:07.576 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-radosgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 3/26 2024-09-18T13:52:07.576 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-selinux-2:17.2.7-1645.g7061ea6c.el9.x86_64 4/26 2024-09-18T13:52:07.576 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libcephfs2-2:17.2.7-1645.g7061ea6c.el9.x86_64 5/26 2024-09-18T13:52:07.576 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libradosstriper1-2:17.2.7-1645.g7061ea6c.el9.x86_6 6/26 2024-09-18T13:52:07.576 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librgw2-2:17.2.7-1645.g7061ea6c.el9.x86_64 7/26 2024-09-18T13:52:07.576 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1645.g7061ea6c.el9. 8/26 2024-09-18T13:52:07.576 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-ceph-common-2:17.2.7-1645.g7061ea6c.el9.x8 9/26 2024-09-18T13:52:07.576 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cephfs-2:17.2.7-1645.g7061ea6c.el9.x86_64 10/26 2024-09-18T13:52:07.576 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rados-2:17.2.7-1645.g7061ea6c.el9.x86_64 11/26 2024-09-18T13:52:07.576 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 12/26 2024-09-18T13:52:07.576 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 13/26 2024-09-18T13:52:07.576 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-18T13:52:07.576 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-18T13:52:07.576 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-18T13:52:07.577 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-18T13:52:07.577 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-18T13:52:07.577 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-18T13:52:07.577 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-18T13:52:07.577 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-18T13:52:07.577 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-18T13:52:07.577 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librados2-2:17.2.7-1645.g7061ea6c.el9.x86_64 23/26 2024-09-18T13:52:07.577 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-18T13:52:07.577 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librbd1-2:17.2.7-1645.g7061ea6c.el9.x86_64 25/26 2024-09-18T13:52:08.345 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-18T13:52:08.345 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:52:08.345 INFO:teuthology.orchestra.run.smithi047.stdout:Upgraded: 2024-09-18T13:52:08.345 INFO:teuthology.orchestra.run.smithi047.stdout: librados2-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T13:52:08.345 INFO:teuthology.orchestra.run.smithi047.stdout: librbd1-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T13:52:08.345 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-09-18T13:52:08.345 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-base-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T13:52:08.345 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-common-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T13:52:08.346 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-radosgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T13:52:08.346 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-selinux-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T13:52:08.346 INFO:teuthology.orchestra.run.smithi047.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-18T13:52:08.346 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs2-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T13:52:08.346 INFO:teuthology.orchestra.run.smithi047.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-18T13:52:08.346 INFO:teuthology.orchestra.run.smithi047.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-18T13:52:08.346 INFO:teuthology.orchestra.run.smithi047.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-18T13:52:08.346 INFO:teuthology.orchestra.run.smithi047.stdout: libradosstriper1-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T13:52:08.346 INFO:teuthology.orchestra.run.smithi047.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-18T13:52:08.346 INFO:teuthology.orchestra.run.smithi047.stdout: librgw2-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T13:52:08.346 INFO:teuthology.orchestra.run.smithi047.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-18T13:52:08.346 INFO:teuthology.orchestra.run.smithi047.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-18T13:52:08.347 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-argparse-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T13:52:08.347 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-common-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T13:52:08.347 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cephfs-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T13:52:08.347 INFO:teuthology.orchestra.run.smithi047.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-18T13:52:08.347 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rados-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T13:52:08.347 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T13:52:08.347 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T13:52:08.347 INFO:teuthology.orchestra.run.smithi047.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-18T13:52:08.347 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:52:08.347 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-09-18T13:52:08.729 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-test 2024-09-18T13:52:09.357 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:00:28 ago on Wed 18 Sep 2024 01:51:41 PM UTC. 2024-09-18T13:52:09.505 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-09-18T13:52:09.505 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:52:09.506 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-09-18T13:52:09.506 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:52:09.506 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-09-18T13:52:09.506 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-test x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 48 M 2024-09-18T13:52:09.506 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-09-18T13:52:09.506 INFO:teuthology.orchestra.run.smithi047.stdout: libcephsqlite x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 163 k 2024-09-18T13:52:09.506 INFO:teuthology.orchestra.run.smithi047.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-18T13:52:09.506 INFO:teuthology.orchestra.run.smithi047.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-18T13:52:09.506 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:52:09.506 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-09-18T13:52:09.506 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:52:09.506 INFO:teuthology.orchestra.run.smithi047.stdout:Install 4 Packages 2024-09-18T13:52:09.507 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:52:09.507 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 48 M 2024-09-18T13:52:09.507 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 193 M 2024-09-18T13:52:09.507 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-09-18T13:52:09.833 INFO:teuthology.orchestra.run.smithi047.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.3 MB/s | 304 kB 00:00 2024-09-18T13:52:09.858 INFO:teuthology.orchestra.run.smithi047.stdout:(2/4): libcephsqlite-17.2.7-1645.g7061ea6c.el9. 650 kB/s | 163 kB 00:00 2024-09-18T13:52:09.883 INFO:teuthology.orchestra.run.smithi047.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-18T13:52:11.109 INFO:teuthology.orchestra.run.smithi047.stdout:(4/4): ceph-test-17.2.7-1645.g7061ea6c.el9.x86_ 32 MB/s | 48 MB 00:01 2024-09-18T13:52:11.111 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-09-18T13:52:11.111 INFO:teuthology.orchestra.run.smithi047.stdout:Total 30 MB/s | 48 MB 00:01 2024-09-18T13:52:11.140 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-09-18T13:52:11.161 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-09-18T13:52:11.161 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-09-18T13:52:11.546 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-09-18T13:52:11.547 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-09-18T13:52:12.095 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-09-18T13:52:12.177 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-18T13:52:12.241 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-18T13:52:12.260 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libcephsqlite-2:17.2.7-1645.g7061ea6c.el9.x86_64 3/4 2024-09-18T13:52:16.917 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1645.g7061ea6c.el9.x86_64 3/4 2024-09-18T13:52:17.012 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-test-2:17.2.7-1645.g7061ea6c.el9.x86_64 4/4 2024-09-18T13:52:19.315 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-test-2:17.2.7-1645.g7061ea6c.el9.x86_64 4/4 2024-09-18T13:52:19.316 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-test-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/4 2024-09-18T13:52:19.316 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libcephsqlite-2:17.2.7-1645.g7061ea6c.el9.x86_64 2/4 2024-09-18T13:52:19.316 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-18T13:52:19.621 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-18T13:52:19.621 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:52:19.621 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-09-18T13:52:19.621 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-test-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T13:52:19.621 INFO:teuthology.orchestra.run.smithi047.stdout: libcephsqlite-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T13:52:19.621 INFO:teuthology.orchestra.run.smithi047.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-18T13:52:19.621 INFO:teuthology.orchestra.run.smithi047.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-18T13:52:19.621 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:52:19.621 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-09-18T13:52:19.910 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph 2024-09-18T13:52:20.511 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:00:39 ago on Wed 18 Sep 2024 01:51:41 PM UTC. 2024-09-18T13:52:20.657 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-09-18T13:52:20.659 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:52:20.660 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-09-18T13:52:20.660 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:52:20.660 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-09-18T13:52:20.660 INFO:teuthology.orchestra.run.smithi047.stdout: ceph x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 6.5 k 2024-09-18T13:52:20.660 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-09-18T13:52:20.660 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mds x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 2.1 M 2024-09-18T13:52:20.660 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 1.4 M 2024-09-18T13:52:20.660 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 240 k 2024-09-18T13:52:20.660 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mon x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 3.9 M 2024-09-18T13:52:20.660 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-osd x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 15 M 2024-09-18T13:52:20.660 INFO:teuthology.orchestra.run.smithi047.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-18T13:52:20.660 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-18T13:52:20.660 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-18T13:52:20.661 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-18T13:52:20.662 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-18T13:52:20.662 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-18T13:52:20.662 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-18T13:52:20.662 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-collections 2024-09-18T13:52:20.662 INFO:teuthology.orchestra.run.smithi047.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-18T13:52:20.662 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-18T13:52:20.662 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-18T13:52:20.662 INFO:teuthology.orchestra.run.smithi047.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-18T13:52:20.662 INFO:teuthology.orchestra.run.smithi047.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-18T13:52:20.662 INFO:teuthology.orchestra.run.smithi047.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-18T13:52:20.662 INFO:teuthology.orchestra.run.smithi047.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-18T13:52:20.662 INFO:teuthology.orchestra.run.smithi047.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-18T13:52:20.662 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-18T13:52:20.663 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-18T13:52:20.663 INFO:teuthology.orchestra.run.smithi047.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-18T13:52:20.664 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-18T13:52:20.664 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-18T13:52:20.664 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-18T13:52:20.664 INFO:teuthology.orchestra.run.smithi047.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-18T13:52:20.664 INFO:teuthology.orchestra.run.smithi047.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-18T13:52:20.665 INFO:teuthology.orchestra.run.smithi047.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-18T13:52:20.665 INFO:teuthology.orchestra.run.smithi047.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-18T13:52:20.665 INFO:teuthology.orchestra.run.smithi047.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-18T13:52:20.667 INFO:teuthology.orchestra.run.smithi047.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-18T13:52:20.667 INFO:teuthology.orchestra.run.smithi047.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-18T13:52:20.667 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:52:20.667 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-09-18T13:52:20.667 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:52:20.667 INFO:teuthology.orchestra.run.smithi047.stdout:Install 34 Packages 2024-09-18T13:52:20.667 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:52:20.668 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 27 M 2024-09-18T13:52:20.668 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 96 M 2024-09-18T13:52:20.668 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-09-18T13:52:20.909 INFO:teuthology.orchestra.run.smithi047.stdout:(1/34): ceph-17.2.7-1645.g7061ea6c.el9.x86_64.r 45 kB/s | 6.5 kB 00:00 2024-09-18T13:52:21.085 INFO:teuthology.orchestra.run.smithi047.stdout:(2/34): ceph-mgr-17.2.7-1645.g7061ea6c.el9.x86_ 4.4 MB/s | 1.4 MB 00:00 2024-09-18T13:52:21.127 INFO:teuthology.orchestra.run.smithi047.stdout:(3/34): ceph-mds-17.2.7-1645.g7061ea6c.el9.x86_ 5.7 MB/s | 2.1 MB 00:00 2024-09-18T13:52:21.194 INFO:teuthology.orchestra.run.smithi047.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1645.g7061 3.5 MB/s | 240 kB 00:00 2024-09-18T13:52:21.269 INFO:teuthology.orchestra.run.smithi047.stdout:(5/34): ceph-mon-17.2.7-1645.g7061ea6c.el9.x86_ 11 MB/s | 3.9 MB 00:00 2024-09-18T13:52:21.603 INFO:teuthology.orchestra.run.smithi047.stdout:(6/34): ceph-osd-17.2.7-1645.g7061ea6c.el9.x86_ 30 MB/s | 15 MB 00:00 2024-09-18T13:52:21.961 INFO:teuthology.orchestra.run.smithi047.stdout:(7/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 330 kB/s | 253 kB 00:00 2024-09-18T13:52:22.061 INFO:teuthology.orchestra.run.smithi047.stdout:(8/34): python3-ply-3.11-14.el9.noarch.rpm 232 kB/s | 106 kB 00:00 2024-09-18T13:52:22.087 INFO:teuthology.orchestra.run.smithi047.stdout:(9/34): python3-pycparser-2.20-6.el9.noarch.rpm 1.1 MB/s | 135 kB 00:00 2024-09-18T13:52:22.196 INFO:teuthology.orchestra.run.smithi047.stdout:(10/34): python3-urllib3-1.26.5-6.el9.noarch.rp 1.9 MB/s | 215 kB 00:00 2024-09-18T13:52:22.229 INFO:teuthology.orchestra.run.smithi047.stdout:(11/34): python3-cryptography-36.0.1-4.el9.x86_ 1.3 MB/s | 1.2 MB 00:00 2024-09-18T13:52:22.262 INFO:teuthology.orchestra.run.smithi047.stdout:(12/34): python3-requests-2.25.1-8.el9.noarch.r 622 kB/s | 125 kB 00:00 2024-09-18T13:52:22.304 INFO:teuthology.orchestra.run.smithi047.stdout:(13/34): python3-mako-1.1.4-6.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-09-18T13:52:22.330 INFO:teuthology.orchestra.run.smithi047.stdout:(14/34): python3-markupsafe-1.1.1-12.el9.x86_64 345 kB/s | 35 kB 00:00 2024-09-18T13:52:22.363 INFO:teuthology.orchestra.run.smithi047.stdout:(15/34): python3-packaging-20.9-5.el9.noarch.rp 769 kB/s | 77 kB 00:00 2024-09-18T13:52:22.405 INFO:teuthology.orchestra.run.smithi047.stdout:(16/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.0 MB/s | 43 kB 00:00 2024-09-18T13:52:22.422 INFO:teuthology.orchestra.run.smithi047.stdout:(17/34): python3-pytz-2021.1-5.el9.noarch.rpm 434 kB/s | 51 kB 00:00 2024-09-18T13:52:22.447 INFO:teuthology.orchestra.run.smithi047.stdout:(18/34): python3-toml-0.10.2-6.el9.noarch.rpm 355 kB/s | 42 kB 00:00 2024-09-18T13:52:22.473 INFO:teuthology.orchestra.run.smithi047.stdout:(19/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-09-18T13:52:22.498 INFO:teuthology.orchestra.run.smithi047.stdout:(20/34): python3-jaraco-classes-3.2.1-5.el9.noa 1.2 MB/s | 18 kB 00:00 2024-09-18T13:52:22.515 INFO:teuthology.orchestra.run.smithi047.stdout:(21/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 180 kB/s | 11 kB 00:00 2024-09-18T13:52:22.540 INFO:teuthology.orchestra.run.smithi047.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 576 kB/s | 23 kB 00:00 2024-09-18T13:52:22.565 INFO:teuthology.orchestra.run.smithi047.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-09-18T13:52:22.591 INFO:teuthology.orchestra.run.smithi047.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-09-18T13:52:22.616 INFO:teuthology.orchestra.run.smithi047.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 921 kB/s | 46 kB 00:00 2024-09-18T13:52:22.633 INFO:teuthology.orchestra.run.smithi047.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-09-18T13:52:22.658 INFO:teuthology.orchestra.run.smithi047.stdout:(27/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-09-18T13:52:22.684 INFO:teuthology.orchestra.run.smithi047.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-09-18T13:52:22.709 INFO:teuthology.orchestra.run.smithi047.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-18T13:52:22.734 INFO:teuthology.orchestra.run.smithi047.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2024-09-18T13:52:22.760 INFO:teuthology.orchestra.run.smithi047.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-09-18T13:52:22.785 INFO:teuthology.orchestra.run.smithi047.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 805 kB/s | 20 kB 00:00 2024-09-18T13:52:22.810 INFO:teuthology.orchestra.run.smithi047.stdout:(33/34): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-18T13:52:22.835 INFO:teuthology.orchestra.run.smithi047.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-18T13:52:22.840 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-09-18T13:52:22.840 INFO:teuthology.orchestra.run.smithi047.stdout:Total 13 MB/s | 27 MB 00:02 2024-09-18T13:52:22.979 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-09-18T13:52:23.021 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-09-18T13:52:23.021 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-09-18T13:52:23.451 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-09-18T13:52:23.451 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-09-18T13:52:24.719 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-09-18T13:52:24.797 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-18T13:52:24.829 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-18T13:52:24.862 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-18T13:52:24.959 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-18T13:52:25.036 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-18T13:52:25.082 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-18T13:52:25.116 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-18T13:52:25.146 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-18T13:52:25.181 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-18T13:52:25.214 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-18T13:52:25.255 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-18T13:52:25.298 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-18T13:52:25.327 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-18T13:52:25.376 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-18T13:52:25.426 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-18T13:52:25.509 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-18T13:52:25.661 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-18T13:52:25.744 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-18T13:52:25.795 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-18T13:52:25.833 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-18T13:52:25.891 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-18T13:52:25.955 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-18T13:52:26.153 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-18T13:52:26.211 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-18T13:52:26.293 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-18T13:52:26.391 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-18T13:52:26.437 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-18T13:52:26.558 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-18T13:52:26.721 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1645.g7061ea6c.el9. 29/34 2024-09-18T13:52:26.753 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-2:17.2.7-1645.g7061ea6c.el9.x86_64 30/34 2024-09-18T13:52:26.780 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1645.g7061ea6c.el9.x86_64 30/34 2024-09-18T13:52:26.781 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T13:52:26.781 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-18T13:52:26.781 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-18T13:52:26.781 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-18T13:52:26.781 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:52:28.188 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-osd-2:17.2.7-1645.g7061ea6c.el9.x86_64 31/34 2024-09-18T13:52:28.220 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-osd-2:17.2.7-1645.g7061ea6c.el9.x86_64 31/34 2024-09-18T13:52:28.220 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T13:52:28.220 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-18T13:52:28.220 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-18T13:52:28.220 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-18T13:52:28.220 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:52:28.622 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mon-2:17.2.7-1645.g7061ea6c.el9.x86_64 32/34 2024-09-18T13:52:28.648 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mon-2:17.2.7-1645.g7061ea6c.el9.x86_64 32/34 2024-09-18T13:52:28.648 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T13:52:28.648 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-18T13:52:28.648 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-18T13:52:28.649 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-18T13:52:28.649 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:52:28.856 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mds-2:17.2.7-1645.g7061ea6c.el9.x86_64 33/34 2024-09-18T13:52:28.881 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mds-2:17.2.7-1645.g7061ea6c.el9.x86_64 33/34 2024-09-18T13:52:28.881 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T13:52:28.881 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-18T13:52:28.881 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-18T13:52:28.882 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-18T13:52:28.882 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:52:28.948 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-2:17.2.7-1645.g7061ea6c.el9.x86_64 34/34 2024-09-18T13:52:30.625 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-2:17.2.7-1645.g7061ea6c.el9.x86_64 34/34 2024-09-18T13:52:30.626 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/34 2024-09-18T13:52:30.626 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mds-2:17.2.7-1645.g7061ea6c.el9.x86_64 2/34 2024-09-18T13:52:30.626 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-2:17.2.7-1645.g7061ea6c.el9.x86_64 3/34 2024-09-18T13:52:30.626 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mon-2:17.2.7-1645.g7061ea6c.el9.x86_64 4/34 2024-09-18T13:52:30.626 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-osd-2:17.2.7-1645.g7061ea6c.el9.x86_64 5/34 2024-09-18T13:52:30.626 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1645.g7061ea6c.el9. 6/34 2024-09-18T13:52:30.626 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-18T13:52:30.626 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-18T13:52:30.626 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-18T13:52:30.626 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-18T13:52:30.626 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-18T13:52:30.627 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-18T13:52:30.627 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-18T13:52:30.627 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-18T13:52:30.627 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-18T13:52:30.627 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-18T13:52:30.627 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-18T13:52:30.627 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-18T13:52:30.627 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-18T13:52:30.627 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-18T13:52:30.627 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-18T13:52:30.627 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-18T13:52:30.627 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-18T13:52:30.627 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-18T13:52:30.627 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-18T13:52:30.628 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-18T13:52:30.629 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-18T13:52:30.629 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-18T13:52:30.629 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-18T13:52:30.629 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-18T13:52:30.629 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-18T13:52:30.629 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-18T13:52:30.629 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-18T13:52:31.420 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-18T13:52:31.420 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:52:31.420 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-09-18T13:52:31.420 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T13:52:31.421 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mds-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T13:52:31.421 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T13:52:31.421 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-modules-core-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T13:52:31.421 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mon-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T13:52:31.421 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-osd-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T13:52:31.421 INFO:teuthology.orchestra.run.smithi047.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-18T13:52:31.421 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-18T13:52:31.421 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-18T13:52:31.421 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-18T13:52:31.421 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-18T13:52:31.421 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-18T13:52:31.421 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-18T13:52:31.422 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-18T13:52:31.422 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-18T13:52:31.422 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-18T13:52:31.422 INFO:teuthology.orchestra.run.smithi047.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-18T13:52:31.422 INFO:teuthology.orchestra.run.smithi047.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-18T13:52:31.422 INFO:teuthology.orchestra.run.smithi047.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-18T13:52:31.422 INFO:teuthology.orchestra.run.smithi047.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-18T13:52:31.422 INFO:teuthology.orchestra.run.smithi047.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-18T13:52:31.422 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-18T13:52:31.422 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ply-3.11-14.el9.noarch 2024-09-18T13:52:31.422 INFO:teuthology.orchestra.run.smithi047.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-18T13:52:31.422 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-18T13:52:31.422 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-18T13:52:31.422 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-18T13:52:31.423 INFO:teuthology.orchestra.run.smithi047.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-18T13:52:31.423 INFO:teuthology.orchestra.run.smithi047.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-18T13:52:31.423 INFO:teuthology.orchestra.run.smithi047.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-18T13:52:31.423 INFO:teuthology.orchestra.run.smithi047.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-18T13:52:31.423 INFO:teuthology.orchestra.run.smithi047.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-18T13:52:31.423 INFO:teuthology.orchestra.run.smithi047.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-18T13:52:31.423 INFO:teuthology.orchestra.run.smithi047.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-18T13:52:31.423 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:52:31.423 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-09-18T13:52:31.769 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-base 2024-09-18T13:52:32.374 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:00:51 ago on Wed 18 Sep 2024 01:51:41 PM UTC. 2024-09-18T13:52:32.465 INFO:teuthology.orchestra.run.smithi047.stdout:Package ceph-base-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T13:52:32.509 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-09-18T13:52:32.510 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-09-18T13:52:32.510 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-09-18T13:52:32.581 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install cephadm 2024-09-18T13:52:33.187 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:00:52 ago on Wed 18 Sep 2024 01:51:41 PM UTC. 2024-09-18T13:52:33.323 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-09-18T13:52:33.324 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:52:33.324 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-09-18T13:52:33.324 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:52:33.324 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-09-18T13:52:33.324 INFO:teuthology.orchestra.run.smithi047.stdout: cephadm noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 83 k 2024-09-18T13:52:33.324 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:52:33.325 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-09-18T13:52:33.325 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:52:33.325 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-09-18T13:52:33.325 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:52:33.325 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 83 k 2024-09-18T13:52:33.325 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 375 k 2024-09-18T13:52:33.325 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-09-18T13:52:33.524 INFO:teuthology.orchestra.run.smithi047.stdout:cephadm-17.2.7-1645.g7061ea6c.el9.noarch.rpm 416 kB/s | 83 kB 00:00 2024-09-18T13:52:33.525 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-09-18T13:52:33.525 INFO:teuthology.orchestra.run.smithi047.stdout:Total 413 kB/s | 83 kB 00:00 2024-09-18T13:52:33.525 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-09-18T13:52:33.530 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-09-18T13:52:33.530 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-09-18T13:52:33.533 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-09-18T13:52:33.533 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-09-18T13:52:33.609 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-09-18T13:52:34.035 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 1/1 2024-09-18T13:52:34.139 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 1/1 2024-09-18T13:52:34.498 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 1/1 2024-09-18T13:52:34.753 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 1/1 2024-09-18T13:52:34.754 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:52:34.754 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-09-18T13:52:34.754 INFO:teuthology.orchestra.run.smithi047.stdout: cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T13:52:34.754 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:52:34.754 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-09-18T13:52:34.991 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-immutable-object-cache 2024-09-18T13:52:35.604 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:00:54 ago on Wed 18 Sep 2024 01:51:41 PM UTC. 2024-09-18T13:52:35.741 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-09-18T13:52:35.742 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:52:35.742 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-09-18T13:52:35.742 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:52:35.742 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-09-18T13:52:35.742 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 137 k 2024-09-18T13:52:35.742 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:52:35.742 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-09-18T13:52:35.742 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:52:35.742 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-09-18T13:52:35.742 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:52:35.743 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 137 k 2024-09-18T13:52:35.743 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 379 k 2024-09-18T13:52:35.743 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-09-18T13:52:35.955 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-immutable-object-cache-17.2.7-1645.g7061ea 649 kB/s | 137 kB 00:00 2024-09-18T13:52:35.955 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-09-18T13:52:35.956 INFO:teuthology.orchestra.run.smithi047.stdout:Total 644 kB/s | 137 kB 00:00 2024-09-18T13:52:35.956 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-09-18T13:52:35.964 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-09-18T13:52:35.964 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-09-18T13:52:36.024 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-09-18T13:52:36.024 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-09-18T13:52:36.167 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-09-18T13:52:36.223 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1645.g7061ea6c. 1/1 2024-09-18T13:52:36.249 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1645.g7061ea6c. 1/1 2024-09-18T13:52:36.249 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T13:52:36.249 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-18T13:52:36.250 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:52:37.289 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1645.g7061ea6c. 1/1 2024-09-18T13:52:37.289 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:52:37.289 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-09-18T13:52:37.289 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-immutable-object-cache-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T13:52:37.289 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:52:37.289 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-09-18T13:52:37.518 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mgr 2024-09-18T13:52:38.125 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:00:57 ago on Wed 18 Sep 2024 01:51:41 PM UTC. 2024-09-18T13:52:38.219 INFO:teuthology.orchestra.run.smithi047.stdout:Package ceph-mgr-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T13:52:38.263 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-09-18T13:52:38.263 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-09-18T13:52:38.264 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-09-18T13:52:38.335 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mgr-dashboard 2024-09-18T13:52:38.941 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:00:57 ago on Wed 18 Sep 2024 01:51:41 PM UTC. 2024-09-18T13:52:39.080 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-09-18T13:52:39.081 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:52:39.081 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-09-18T13:52:39.081 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:52:39.081 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-09-18T13:52:39.081 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 3.4 M 2024-09-18T13:52:39.081 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-09-18T13:52:39.081 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 24 k 2024-09-18T13:52:39.081 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 15 k 2024-09-18T13:52:39.081 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-18T13:52:39.081 INFO:teuthology.orchestra.run.smithi047.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-18T13:52:39.081 INFO:teuthology.orchestra.run.smithi047.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-18T13:52:39.081 INFO:teuthology.orchestra.run.smithi047.stdout:Installing weak dependencies: 2024-09-18T13:52:39.082 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-18T13:52:39.082 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:52:39.082 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-09-18T13:52:39.082 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:52:39.082 INFO:teuthology.orchestra.run.smithi047.stdout:Install 7 Packages 2024-09-18T13:52:39.082 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:52:39.082 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 3.7 M 2024-09-18T13:52:39.083 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 80 M 2024-09-18T13:52:39.083 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-09-18T13:52:39.400 INFO:teuthology.orchestra.run.smithi047.stdout:(1/7): ceph-prometheus-alerts-17.2.7-1645.g7061 98 kB/s | 15 kB 00:00 2024-09-18T13:52:39.425 INFO:teuthology.orchestra.run.smithi047.stdout:(2/7): ceph-grafana-dashboards-17.2.7-1645.g706 140 kB/s | 24 kB 00:00 2024-09-18T13:52:39.467 INFO:teuthology.orchestra.run.smithi047.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 135 kB/s | 9.0 kB 00:00 2024-09-18T13:52:39.492 INFO:teuthology.orchestra.run.smithi047.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-09-18T13:52:39.517 INFO:teuthology.orchestra.run.smithi047.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 444 kB/s | 41 kB 00:00 2024-09-18T13:52:39.534 INFO:teuthology.orchestra.run.smithi047.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 4.4 MB/s | 188 kB 00:00 2024-09-18T13:52:39.618 INFO:teuthology.orchestra.run.smithi047.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1645.g7061ea6c 9.1 MB/s | 3.4 MB 00:00 2024-09-18T13:52:39.621 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-09-18T13:52:39.621 INFO:teuthology.orchestra.run.smithi047.stdout:Total 6.8 MB/s | 3.7 MB 00:00 2024-09-18T13:52:39.622 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-09-18T13:52:39.631 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-09-18T13:52:39.631 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-09-18T13:52:39.722 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-09-18T13:52:39.722 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-09-18T13:52:40.062 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-09-18T13:52:40.128 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-18T13:52:40.162 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-18T13:52:40.210 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-18T13:52:40.240 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-18T13:52:40.278 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1645.g7061ea6c.el9.n 5/7 2024-09-18T13:52:41.328 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1645.g7061ea6c.el9. 6/7 2024-09-18T13:52:41.373 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1645.g7061ea6c.el9.noarc 7/7 2024-09-18T13:52:42.361 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1645.g7061ea6c.el9.noarc 7/7 2024-09-18T13:52:42.361 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1645.g7061ea6c.el9. 1/7 2024-09-18T13:52:42.361 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1645.g7061ea6c.el9.noarc 2/7 2024-09-18T13:52:42.361 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1645.g7061ea6c.el9.n 3/7 2024-09-18T13:52:42.361 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-18T13:52:42.362 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-18T13:52:42.362 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-18T13:52:42.644 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-18T13:52:42.645 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:52:42.645 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-09-18T13:52:42.645 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-grafana-dashboards-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T13:52:42.645 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-dashboard-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T13:52:42.645 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-prometheus-alerts-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T13:52:42.645 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-18T13:52:42.645 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-18T13:52:42.645 INFO:teuthology.orchestra.run.smithi047.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-18T13:52:42.645 INFO:teuthology.orchestra.run.smithi047.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-18T13:52:42.645 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:52:42.645 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-09-18T13:52:42.851 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-18T13:52:43.457 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:02 ago on Wed 18 Sep 2024 01:51:41 PM UTC. 2024-09-18T13:52:43.596 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-09-18T13:52:43.597 INFO:teuthology.orchestra.run.smithi047.stdout:======================================================================================= 2024-09-18T13:52:43.597 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-09-18T13:52:43.597 INFO:teuthology.orchestra.run.smithi047.stdout:======================================================================================= 2024-09-18T13:52:43.597 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-09-18T13:52:43.597 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 7.4 M 2024-09-18T13:52:43.597 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-09-18T13:52:43.597 INFO:teuthology.orchestra.run.smithi047.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-18T13:52:43.597 INFO:teuthology.orchestra.run.smithi047.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-18T13:52:43.598 INFO:teuthology.orchestra.run.smithi047.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-18T13:52:43.598 INFO:teuthology.orchestra.run.smithi047.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-18T13:52:43.598 INFO:teuthology.orchestra.run.smithi047.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-18T13:52:43.598 INFO:teuthology.orchestra.run.smithi047.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-18T13:52:43.598 INFO:teuthology.orchestra.run.smithi047.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-18T13:52:43.598 INFO:teuthology.orchestra.run.smithi047.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-18T13:52:43.598 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-18T13:52:43.598 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-18T13:52:43.598 INFO:teuthology.orchestra.run.smithi047.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-18T13:52:43.598 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:52:43.598 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-09-18T13:52:43.598 INFO:teuthology.orchestra.run.smithi047.stdout:======================================================================================= 2024-09-18T13:52:43.598 INFO:teuthology.orchestra.run.smithi047.stdout:Install 12 Packages 2024-09-18T13:52:43.598 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:52:43.599 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 38 M 2024-09-18T13:52:43.599 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 212 M 2024-09-18T13:52:43.599 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-09-18T13:52:44.551 INFO:teuthology.orchestra.run.smithi047.stdout:(1/12): ceph-mgr-diskprediction-local-17.2.7-16 16 MB/s | 7.4 MB 00:00 2024-09-18T13:52:44.576 INFO:teuthology.orchestra.run.smithi047.stdout:(2/12): libquadmath-11.5.0-2.el9.x86_64.rpm 400 kB/s | 191 kB 00:00 2024-09-18T13:52:44.702 INFO:teuthology.orchestra.run.smithi047.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 1.3 MB/s | 801 kB 00:00 2024-09-18T13:52:45.127 INFO:teuthology.orchestra.run.smithi047.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 57 kB/s | 33 kB 00:00 2024-09-18T13:52:45.177 INFO:teuthology.orchestra.run.smithi047.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 38 kB/s | 18 kB 00:00 2024-09-18T13:52:45.619 INFO:teuthology.orchestra.run.smithi047.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.9 MB/s | 3.0 MB 00:01 2024-09-18T13:52:45.686 INFO:teuthology.orchestra.run.smithi047.stdout:(7/12): openblas-0.3.26-2.el9.x86_64.rpm 69 kB/s | 38 kB 00:00 2024-09-18T13:52:46.303 INFO:teuthology.orchestra.run.smithi047.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 4.4 MB/s | 4.9 MB 00:01 2024-09-18T13:52:46.386 INFO:teuthology.orchestra.run.smithi047.stdout:(9/12): python3-devel-3.9.19-8.el9.x86_64.rpm 319 kB/s | 245 kB 00:00 2024-09-18T13:52:46.820 INFO:teuthology.orchestra.run.smithi047.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.5 MB/s | 5.1 MB 00:01 2024-09-18T13:52:47.112 INFO:teuthology.orchestra.run.smithi047.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 310 kB/s | 250 kB 00:00 2024-09-18T13:52:47.795 INFO:teuthology.orchestra.run.smithi047.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 12 MB/s | 16 MB 00:01 2024-09-18T13:52:47.797 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-09-18T13:52:47.797 INFO:teuthology.orchestra.run.smithi047.stdout:Total 9.1 MB/s | 38 MB 00:04 2024-09-18T13:52:48.170 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-09-18T13:52:48.198 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-09-18T13:52:48.198 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-09-18T13:52:48.742 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-09-18T13:52:48.742 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-09-18T13:52:49.531 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-09-18T13:52:49.601 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-18T13:52:49.652 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-18T13:52:49.757 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-18T13:52:49.804 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-18T13:52:49.995 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-18T13:52:50.027 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-18T13:52:50.106 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-18T13:52:50.829 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-18T13:52:50.947 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-18T13:52:52.289 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-18T13:52:53.383 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-18T13:52:53.418 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1645.g7061e 12/12 2024-09-18T13:52:55.970 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1645.g7061e 12/12 2024-09-18T13:52:55.970 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1645.g7061e 1/12 2024-09-18T13:52:55.971 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-18T13:52:55.971 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-18T13:52:55.971 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-18T13:52:55.971 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-18T13:52:55.971 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-18T13:52:55.971 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-18T13:52:55.971 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-18T13:52:55.971 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-18T13:52:55.971 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-18T13:52:55.971 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-18T13:52:56.377 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-18T13:52:56.377 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:52:56.377 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-09-18T13:52:56.377 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T13:52:56.377 INFO:teuthology.orchestra.run.smithi047.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-18T13:52:56.377 INFO:teuthology.orchestra.run.smithi047.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-18T13:52:56.377 INFO:teuthology.orchestra.run.smithi047.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-18T13:52:56.378 INFO:teuthology.orchestra.run.smithi047.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-18T13:52:56.378 INFO:teuthology.orchestra.run.smithi047.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-18T13:52:56.378 INFO:teuthology.orchestra.run.smithi047.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-18T13:52:56.378 INFO:teuthology.orchestra.run.smithi047.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-18T13:52:56.378 INFO:teuthology.orchestra.run.smithi047.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-18T13:52:56.378 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-18T13:52:56.378 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-18T13:52:56.378 INFO:teuthology.orchestra.run.smithi047.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-18T13:52:56.378 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:52:56.378 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-09-18T13:52:56.731 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mgr-rook 2024-09-18T13:52:57.338 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:16 ago on Wed 18 Sep 2024 01:51:41 PM UTC. 2024-09-18T13:52:57.477 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-09-18T13:52:57.479 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:52:57.479 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-09-18T13:52:57.479 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:52:57.479 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-09-18T13:52:57.479 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-rook noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 47 k 2024-09-18T13:52:57.479 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-09-18T13:52:57.479 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-18T13:52:57.479 INFO:teuthology.orchestra.run.smithi047.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-18T13:52:57.479 INFO:teuthology.orchestra.run.smithi047.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-18T13:52:57.479 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-18T13:52:57.480 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-18T13:52:57.480 INFO:teuthology.orchestra.run.smithi047.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-18T13:52:57.480 INFO:teuthology.orchestra.run.smithi047.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-18T13:52:57.480 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-18T13:52:57.480 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-18T13:52:57.480 INFO:teuthology.orchestra.run.smithi047.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-18T13:52:57.480 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-18T13:52:57.480 INFO:teuthology.orchestra.run.smithi047.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-18T13:52:57.480 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:52:57.480 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-09-18T13:52:57.480 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:52:57.480 INFO:teuthology.orchestra.run.smithi047.stdout:Install 13 Packages 2024-09-18T13:52:57.480 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:52:57.481 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 2.2 M 2024-09-18T13:52:57.481 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 26 M 2024-09-18T13:52:57.481 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-09-18T13:52:57.918 INFO:teuthology.orchestra.run.smithi047.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 371 kB/s | 26 kB 00:00 2024-09-18T13:52:57.943 INFO:teuthology.orchestra.run.smithi047.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 203 kB/s | 19 kB 00:00 2024-09-18T13:52:58.019 INFO:teuthology.orchestra.run.smithi047.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-09-18T13:52:58.044 INFO:teuthology.orchestra.run.smithi047.stdout:(4/13): ceph-mgr-rook-17.2.7-1645.g7061ea6c.el9 244 kB/s | 47 kB 00:00 2024-09-18T13:52:58.069 INFO:teuthology.orchestra.run.smithi047.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.4 MB/s | 222 kB 00:00 2024-09-18T13:52:58.094 INFO:teuthology.orchestra.run.smithi047.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-18T13:52:58.119 INFO:teuthology.orchestra.run.smithi047.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 279 kB 00:00 2024-09-18T13:52:58.170 INFO:teuthology.orchestra.run.smithi047.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 321 kB/s | 32 kB 00:00 2024-09-18T13:52:58.195 INFO:teuthology.orchestra.run.smithi047.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 140 kB/s | 14 kB 00:00 2024-09-18T13:52:58.229 INFO:teuthology.orchestra.run.smithi047.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2024-09-18T13:52:58.254 INFO:teuthology.orchestra.run.smithi047.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 1.8 MB/s | 220 kB 00:00 2024-09-18T13:52:58.287 INFO:teuthology.orchestra.run.smithi047.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-09-18T13:52:58.338 INFO:teuthology.orchestra.run.smithi047.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.2 MB/s | 1.0 MB 00:00 2024-09-18T13:52:58.340 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-09-18T13:52:58.341 INFO:teuthology.orchestra.run.smithi047.stdout:Total 2.6 MB/s | 2.2 MB 00:00 2024-09-18T13:52:58.409 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-09-18T13:52:58.424 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-09-18T13:52:58.424 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-09-18T13:52:58.521 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-09-18T13:52:58.521 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-09-18T13:52:59.069 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-09-18T13:52:59.243 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-18T13:52:59.316 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-18T13:52:59.372 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-18T13:52:59.407 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-18T13:52:59.449 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-18T13:52:59.555 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-18T13:52:59.674 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-18T13:52:59.735 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-18T13:53:00.497 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-18T13:53:00.547 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-18T13:53:00.581 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-18T13:53:00.636 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-18T13:53:00.659 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-rook-2:17.2.7-1645.g7061ea6c.el9.noarch 13/13 2024-09-18T13:53:01.501 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1645.g7061ea6c.el9.noarch 13/13 2024-09-18T13:53:01.501 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1645.g7061ea6c.el9.noarch 1/13 2024-09-18T13:53:01.501 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-18T13:53:01.501 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-18T13:53:01.501 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-18T13:53:01.501 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-18T13:53:01.501 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-18T13:53:01.501 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-18T13:53:01.501 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-18T13:53:01.501 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-18T13:53:01.501 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-18T13:53:01.501 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-18T13:53:01.502 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-18T13:53:01.934 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-18T13:53:01.934 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:01.934 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-09-18T13:53:01.934 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-rook-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T13:53:01.934 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-18T13:53:01.934 INFO:teuthology.orchestra.run.smithi047.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-18T13:53:01.935 INFO:teuthology.orchestra.run.smithi047.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-18T13:53:01.935 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-18T13:53:01.935 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-18T13:53:01.935 INFO:teuthology.orchestra.run.smithi047.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-18T13:53:01.935 INFO:teuthology.orchestra.run.smithi047.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-18T13:53:01.935 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-18T13:53:01.935 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-18T13:53:01.935 INFO:teuthology.orchestra.run.smithi047.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-18T13:53:01.935 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-18T13:53:01.935 INFO:teuthology.orchestra.run.smithi047.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-18T13:53:01.935 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:01.935 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-09-18T13:53:02.191 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mgr-cephadm 2024-09-18T13:53:02.797 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:21 ago on Wed 18 Sep 2024 01:51:41 PM UTC. 2024-09-18T13:53:02.934 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-09-18T13:53:02.935 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:53:02.935 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-09-18T13:53:02.935 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:53:02.935 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-09-18T13:53:02.935 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 124 k 2024-09-18T13:53:02.935 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-09-18T13:53:02.935 INFO:teuthology.orchestra.run.smithi047.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-18T13:53:02.935 INFO:teuthology.orchestra.run.smithi047.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-18T13:53:02.935 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-18T13:53:02.935 INFO:teuthology.orchestra.run.smithi047.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-18T13:53:02.935 INFO:teuthology.orchestra.run.smithi047.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-18T13:53:02.936 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:02.936 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-09-18T13:53:02.936 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:53:02.936 INFO:teuthology.orchestra.run.smithi047.stdout:Install 6 Packages 2024-09-18T13:53:02.936 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:02.936 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 7.0 M 2024-09-18T13:53:02.936 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 33 M 2024-09-18T13:53:02.936 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-09-18T13:53:03.206 INFO:teuthology.orchestra.run.smithi047.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-09-18T13:53:03.231 INFO:teuthology.orchestra.run.smithi047.stdout:(2/6): ceph-mgr-cephadm-17.2.7-1645.g7061ea6c.e 576 kB/s | 124 kB 00:00 2024-09-18T13:53:03.281 INFO:teuthology.orchestra.run.smithi047.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-18T13:53:03.315 INFO:teuthology.orchestra.run.smithi047.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.2 MB/s | 74 kB 00:00 2024-09-18T13:53:03.340 INFO:teuthology.orchestra.run.smithi047.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.0 MB/s | 548 kB 00:00 2024-09-18T13:53:03.474 INFO:teuthology.orchestra.run.smithi047.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 6.0 MB 00:00 2024-09-18T13:53:03.476 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-09-18T13:53:03.476 INFO:teuthology.orchestra.run.smithi047.stdout:Total 13 MB/s | 7.0 MB 00:00 2024-09-18T13:53:03.550 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-09-18T13:53:03.561 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-09-18T13:53:03.561 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-09-18T13:53:03.666 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-09-18T13:53:03.666 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-09-18T13:53:03.984 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-09-18T13:53:04.149 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-18T13:53:04.202 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-18T13:53:04.763 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-18T13:53:04.857 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-18T13:53:04.919 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-18T13:53:04.946 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 6/6 2024-09-18T13:53:05.701 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 6/6 2024-09-18T13:53:05.701 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 1/6 2024-09-18T13:53:05.701 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-18T13:53:05.701 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-18T13:53:05.701 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-18T13:53:05.701 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-18T13:53:05.976 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-18T13:53:05.976 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:05.976 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-09-18T13:53:05.976 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-cephadm-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T13:53:05.976 INFO:teuthology.orchestra.run.smithi047.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-18T13:53:05.976 INFO:teuthology.orchestra.run.smithi047.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-18T13:53:05.976 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-18T13:53:05.976 INFO:teuthology.orchestra.run.smithi047.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-18T13:53:05.976 INFO:teuthology.orchestra.run.smithi047.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-18T13:53:05.977 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:05.977 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-09-18T13:53:06.234 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-fuse 2024-09-18T13:53:06.841 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:25 ago on Wed 18 Sep 2024 01:51:41 PM UTC. 2024-09-18T13:53:06.980 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-09-18T13:53:06.981 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:53:06.981 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-09-18T13:53:06.981 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:53:06.981 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-09-18T13:53:06.981 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-fuse x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 813 k 2024-09-18T13:53:06.981 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-09-18T13:53:06.981 INFO:teuthology.orchestra.run.smithi047.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-18T13:53:06.981 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:06.981 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-09-18T13:53:06.981 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:53:06.981 INFO:teuthology.orchestra.run.smithi047.stdout:Install 2 Packages 2024-09-18T13:53:06.982 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:06.982 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 893 k 2024-09-18T13:53:06.982 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 2.6 M 2024-09-18T13:53:06.982 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-09-18T13:53:07.442 INFO:teuthology.orchestra.run.smithi047.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 440 kB/s | 80 kB 00:00 2024-09-18T13:53:07.517 INFO:teuthology.orchestra.run.smithi047.stdout:(2/2): ceph-fuse-17.2.7-1645.g7061ea6c.el9.x86_ 3.1 MB/s | 813 kB 00:00 2024-09-18T13:53:07.517 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-09-18T13:53:07.518 INFO:teuthology.orchestra.run.smithi047.stdout:Total 1.6 MB/s | 893 kB 00:00 2024-09-18T13:53:07.530 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-09-18T13:53:07.554 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-09-18T13:53:07.554 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-09-18T13:53:07.638 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-09-18T13:53:07.638 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-09-18T13:53:07.832 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-09-18T13:53:08.001 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-18T13:53:08.095 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 2/2 2024-09-18T13:53:08.788 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 2/2 2024-09-18T13:53:08.788 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/2 2024-09-18T13:53:09.049 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-18T13:53:09.049 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:09.049 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-09-18T13:53:09.049 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-18T13:53:09.049 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:09.049 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-09-18T13:53:09.299 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-volume 2024-09-18T13:53:09.909 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:28 ago on Wed 18 Sep 2024 01:51:41 PM UTC. 2024-09-18T13:53:10.049 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-09-18T13:53:10.050 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:53:10.050 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-09-18T13:53:10.050 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:53:10.050 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-09-18T13:53:10.050 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-volume noarch 2:17.2.7-1645.g7061ea6c.el9 ceph-noarch 270 k 2024-09-18T13:53:10.050 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:10.050 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-09-18T13:53:10.050 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:53:10.050 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-09-18T13:53:10.050 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:10.050 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 270 k 2024-09-18T13:53:10.050 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 1.3 M 2024-09-18T13:53:10.050 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-09-18T13:53:10.272 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-volume-17.2.7-1645.g7061ea6c.el9.noarch.rp 1.2 MB/s | 270 kB 00:00 2024-09-18T13:53:10.273 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-09-18T13:53:10.273 INFO:teuthology.orchestra.run.smithi047.stdout:Total 1.2 MB/s | 270 kB 00:00 2024-09-18T13:53:10.273 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-09-18T13:53:10.282 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-09-18T13:53:10.283 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-09-18T13:53:10.315 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-09-18T13:53:10.316 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-09-18T13:53:10.538 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-09-18T13:53:10.591 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-volume-2:17.2.7-1645.g7061ea6c.el9.noarch 1/1 2024-09-18T13:53:10.608 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-volume-2:17.2.7-1645.g7061ea6c.el9.noarch 1/1 2024-09-18T13:53:10.608 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T13:53:10.608 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-18T13:53:10.608 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:11.581 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-volume-2:17.2.7-1645.g7061ea6c.el9.noarch 1/1 2024-09-18T13:53:11.581 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:11.581 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-09-18T13:53:11.581 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-volume-2:17.2.7-1645.g7061ea6c.el9.noarch 2024-09-18T13:53:11.581 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:11.582 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-09-18T13:53:11.847 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install librados-devel 2024-09-18T13:53:12.453 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:31 ago on Wed 18 Sep 2024 01:51:41 PM UTC. 2024-09-18T13:53:12.591 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-09-18T13:53:12.592 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:53:12.592 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-09-18T13:53:12.592 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:53:12.592 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-09-18T13:53:12.592 INFO:teuthology.orchestra.run.smithi047.stdout: librados-devel x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 126 k 2024-09-18T13:53:12.592 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:12.593 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-09-18T13:53:12.593 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:53:12.593 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-09-18T13:53:12.593 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:12.593 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 126 k 2024-09-18T13:53:12.593 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 448 k 2024-09-18T13:53:12.593 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-09-18T13:53:12.802 INFO:teuthology.orchestra.run.smithi047.stdout:librados-devel-17.2.7-1645.g7061ea6c.el9.x86_64 604 kB/s | 126 kB 00:00 2024-09-18T13:53:12.803 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-09-18T13:53:12.803 INFO:teuthology.orchestra.run.smithi047.stdout:Total 599 kB/s | 126 kB 00:00 2024-09-18T13:53:12.803 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-09-18T13:53:12.809 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-09-18T13:53:12.809 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-09-18T13:53:12.875 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-09-18T13:53:12.876 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-09-18T13:53:13.039 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-09-18T13:53:13.152 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librados-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T13:53:13.577 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librados-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T13:53:13.840 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librados-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T13:53:13.840 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:13.840 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-09-18T13:53:13.840 INFO:teuthology.orchestra.run.smithi047.stdout: librados-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T13:53:13.840 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:13.840 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-09-18T13:53:14.062 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install libcephfs2 2024-09-18T13:53:14.670 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:33 ago on Wed 18 Sep 2024 01:51:41 PM UTC. 2024-09-18T13:53:14.763 INFO:teuthology.orchestra.run.smithi047.stdout:Package libcephfs2-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T13:53:14.808 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-09-18T13:53:14.809 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-09-18T13:53:14.809 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-09-18T13:53:14.876 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install libcephfs-devel 2024-09-18T13:53:15.481 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:34 ago on Wed 18 Sep 2024 01:51:41 PM UTC. 2024-09-18T13:53:15.619 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-09-18T13:53:15.620 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:53:15.620 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-09-18T13:53:15.620 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:53:15.620 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-09-18T13:53:15.620 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs-devel x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 25 k 2024-09-18T13:53:15.620 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:15.620 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-09-18T13:53:15.620 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:53:15.621 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-09-18T13:53:15.621 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:15.621 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 25 k 2024-09-18T13:53:15.621 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 110 k 2024-09-18T13:53:15.621 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-09-18T13:53:15.781 INFO:teuthology.orchestra.run.smithi047.stdout:libcephfs-devel-17.2.7-1645.g7061ea6c.el9.x86_6 156 kB/s | 25 kB 00:00 2024-09-18T13:53:15.782 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-09-18T13:53:15.782 INFO:teuthology.orchestra.run.smithi047.stdout:Total 154 kB/s | 25 kB 00:00 2024-09-18T13:53:15.782 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-09-18T13:53:15.785 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-09-18T13:53:15.785 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-09-18T13:53:15.787 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-09-18T13:53:15.787 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-09-18T13:53:15.873 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-09-18T13:53:15.997 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libcephfs-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T13:53:16.367 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T13:53:16.606 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libcephfs-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T13:53:16.607 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:16.607 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-09-18T13:53:16.607 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs-devel-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T13:53:16.607 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:16.607 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-09-18T13:53:16.790 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install librados2 2024-09-18T13:53:17.395 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:36 ago on Wed 18 Sep 2024 01:51:41 PM UTC. 2024-09-18T13:53:17.487 INFO:teuthology.orchestra.run.smithi047.stdout:Package librados2-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T13:53:17.532 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-09-18T13:53:17.533 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-09-18T13:53:17.533 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-09-18T13:53:17.600 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install librbd1 2024-09-18T13:53:18.196 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:37 ago on Wed 18 Sep 2024 01:51:41 PM UTC. 2024-09-18T13:53:18.289 INFO:teuthology.orchestra.run.smithi047.stdout:Package librbd1-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T13:53:18.334 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-09-18T13:53:18.335 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-09-18T13:53:18.335 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-09-18T13:53:18.405 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install python3-rados 2024-09-18T13:53:19.004 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:38 ago on Wed 18 Sep 2024 01:51:41 PM UTC. 2024-09-18T13:53:19.095 INFO:teuthology.orchestra.run.smithi047.stdout:Package python3-rados-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T13:53:19.142 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-09-18T13:53:19.143 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-09-18T13:53:19.144 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-09-18T13:53:19.209 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install python3-rgw 2024-09-18T13:53:19.813 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:38 ago on Wed 18 Sep 2024 01:51:41 PM UTC. 2024-09-18T13:53:19.906 INFO:teuthology.orchestra.run.smithi047.stdout:Package python3-rgw-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T13:53:19.951 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-09-18T13:53:19.952 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-09-18T13:53:19.952 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-09-18T13:53:20.014 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install python3-cephfs 2024-09-18T13:53:20.619 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:39 ago on Wed 18 Sep 2024 01:51:41 PM UTC. 2024-09-18T13:53:20.712 INFO:teuthology.orchestra.run.smithi047.stdout:Package python3-cephfs-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T13:53:20.758 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-09-18T13:53:20.759 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-09-18T13:53:20.760 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-09-18T13:53:20.826 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install python3-rbd 2024-09-18T13:53:21.431 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:40 ago on Wed 18 Sep 2024 01:51:41 PM UTC. 2024-09-18T13:53:21.525 INFO:teuthology.orchestra.run.smithi047.stdout:Package python3-rbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 is already installed. 2024-09-18T13:53:21.570 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-09-18T13:53:21.571 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-09-18T13:53:21.571 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-09-18T13:53:21.641 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install rbd-fuse 2024-09-18T13:53:22.236 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:41 ago on Wed 18 Sep 2024 01:51:41 PM UTC. 2024-09-18T13:53:22.374 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-09-18T13:53:22.375 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:53:22.375 INFO:teuthology.orchestra.run.smithi047.stdout: Package Architecture Version Repository Size 2024-09-18T13:53:22.375 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:53:22.375 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-09-18T13:53:22.376 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-fuse x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 82 k 2024-09-18T13:53:22.376 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:22.376 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-09-18T13:53:22.376 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:53:22.376 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-09-18T13:53:22.376 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:22.376 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 82 k 2024-09-18T13:53:22.376 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 222 k 2024-09-18T13:53:22.376 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-09-18T13:53:22.567 INFO:teuthology.orchestra.run.smithi047.stdout:rbd-fuse-17.2.7-1645.g7061ea6c.el9.x86_64.rpm 428 kB/s | 82 kB 00:00 2024-09-18T13:53:22.568 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-09-18T13:53:22.568 INFO:teuthology.orchestra.run.smithi047.stdout:Total 425 kB/s | 82 kB 00:00 2024-09-18T13:53:22.569 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-09-18T13:53:22.576 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-09-18T13:53:22.576 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-09-18T13:53:22.641 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-09-18T13:53:22.642 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-09-18T13:53:22.799 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-09-18T13:53:22.892 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : rbd-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T13:53:23.250 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T13:53:23.518 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : rbd-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T13:53:23.518 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:23.518 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-09-18T13:53:23.518 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-fuse-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T13:53:23.518 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:23.518 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-09-18T13:53:23.786 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install rbd-mirror 2024-09-18T13:53:24.397 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:43 ago on Wed 18 Sep 2024 01:51:41 PM UTC. 2024-09-18T13:53:24.537 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-09-18T13:53:24.538 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:53:24.538 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-09-18T13:53:24.538 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:53:24.538 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-09-18T13:53:24.538 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-mirror x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 2.9 M 2024-09-18T13:53:24.538 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:24.538 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-09-18T13:53:24.538 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:53:24.538 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-09-18T13:53:24.538 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:24.539 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 2.9 M 2024-09-18T13:53:24.539 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 12 M 2024-09-18T13:53:24.539 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-09-18T13:53:24.854 INFO:teuthology.orchestra.run.smithi047.stdout:rbd-mirror-17.2.7-1645.g7061ea6c.el9.x86_64.rpm 9.3 MB/s | 2.9 MB 00:00 2024-09-18T13:53:24.854 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-09-18T13:53:24.855 INFO:teuthology.orchestra.run.smithi047.stdout:Total 9.3 MB/s | 2.9 MB 00:00 2024-09-18T13:53:24.855 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-09-18T13:53:24.865 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-09-18T13:53:24.865 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-09-18T13:53:24.948 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-09-18T13:53:24.948 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-09-18T13:53:25.384 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-09-18T13:53:25.446 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : rbd-mirror-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T13:53:25.473 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T13:53:25.474 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T13:53:25.474 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-18T13:53:25.474 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-18T13:53:25.474 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-18T13:53:25.474 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:26.581 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : rbd-mirror-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T13:53:26.581 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:26.581 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-09-18T13:53:26.581 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-mirror-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T13:53:26.581 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:26.581 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-09-18T13:53:26.842 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install rbd-nbd 2024-09-18T13:53:27.454 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:46 ago on Wed 18 Sep 2024 01:51:41 PM UTC. 2024-09-18T13:53:27.595 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-09-18T13:53:27.596 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:53:27.596 INFO:teuthology.orchestra.run.smithi047.stdout: Package Architecture Version Repository Size 2024-09-18T13:53:27.596 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:53:27.596 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-09-18T13:53:27.597 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-nbd x86_64 2:17.2.7-1645.g7061ea6c.el9 ceph 165 k 2024-09-18T13:53:27.597 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:27.597 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-09-18T13:53:27.597 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-09-18T13:53:27.597 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-09-18T13:53:27.597 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:27.597 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 165 k 2024-09-18T13:53:27.597 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 477 k 2024-09-18T13:53:27.597 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-09-18T13:53:27.817 INFO:teuthology.orchestra.run.smithi047.stdout:rbd-nbd-17.2.7-1645.g7061ea6c.el9.x86_64.rpm 756 kB/s | 165 kB 00:00 2024-09-18T13:53:27.817 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-09-18T13:53:27.817 INFO:teuthology.orchestra.run.smithi047.stdout:Total 749 kB/s | 165 kB 00:00 2024-09-18T13:53:27.817 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-09-18T13:53:27.826 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-09-18T13:53:27.826 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-09-18T13:53:27.893 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-09-18T13:53:27.893 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-09-18T13:53:28.051 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-09-18T13:53:28.149 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : rbd-nbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T13:53:28.577 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T13:53:28.812 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : rbd-nbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 1/1 2024-09-18T13:53:28.813 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:28.813 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-09-18T13:53:28.813 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-nbd-2:17.2.7-1645.g7061ea6c.el9.x86_64 2024-09-18T13:53:28.813 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-18T13:53:28.813 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-09-18T13:53:29.064 DEBUG:teuthology.parallel:result is None 2024-09-18T13:53:29.065 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T13:53:29.209 DEBUG:teuthology.orchestra.run.smithi047:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-18T13:53:29.231 INFO:teuthology.orchestra.run.smithi047.stdout:17.2.7-1645.g7061ea6c.el9 2024-09-18T13:53:29.232 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1645.g7061ea6c.el9 2024-09-18T13:53:29.232 INFO:teuthology.task.install:The correct ceph version 17.2.7-1645.g7061ea6c is installed. 2024-09-18T13:53:29.234 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-18T13:53:29.234 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-09-18T13:53:29.234 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-18T13:53:29.298 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-18T13:53:29.299 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-09-18T13:53:29.299 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/daemon-helper 2024-09-18T13:53:29.364 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-18T13:53:29.429 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-18T13:53:29.429 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-09-18T13:53:29.430 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-18T13:53:29.495 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-18T13:53:29.559 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-18T13:53:29.560 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-09-18T13:53:29.560 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/stdin-killer 2024-09-18T13:53:29.625 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-18T13:53:29.691 INFO:teuthology.run_tasks:Running task workunit... 2024-09-18T13:53:29.707 INFO:tasks.workunit:Pulling workunits from ref 7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T13:53:29.707 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-18T13:53:29.707 INFO:tasks.workunit:timeout=3h 2024-09-18T13:53:29.707 INFO:tasks.workunit:cleanup=True 2024-09-18T13:53:29.707 DEBUG:teuthology.orchestra.run.smithi047:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-18T13:53:29.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T13:53:29.748 INFO:teuthology.orchestra.run.smithi047.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-18T13:53:29.748 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-18T13:53:29.804 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-18T13:53:29.804 DEBUG:teuthology.orchestra.run.smithi047:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-18T13:53:29.860 DEBUG:teuthology.orchestra.run.smithi047:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T13:53:29.916 INFO:tasks.workunit.client.0.smithi047.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-18T13:54:20.498 INFO:tasks.workunit.client.0.smithi047.stderr:Updating files: 98% (11894/12088) Updating files: 99% (11968/12088) Updating files: 100% (12088/12088) Updating files: 100% (12088/12088), done. 2024-09-18T13:54:21.403 INFO:tasks.workunit.client.0.smithi047.stderr:Note: switching to '7061ea6ca50908d104e9f3979dc615e118d9e153'. 2024-09-18T13:54:21.403 INFO:tasks.workunit.client.0.smithi047.stderr: 2024-09-18T13:54:21.403 INFO:tasks.workunit.client.0.smithi047.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-18T13:54:21.403 INFO:tasks.workunit.client.0.smithi047.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-18T13:54:21.403 INFO:tasks.workunit.client.0.smithi047.stderr:state without impacting any branches by switching back to a branch. 2024-09-18T13:54:21.403 INFO:tasks.workunit.client.0.smithi047.stderr: 2024-09-18T13:54:21.403 INFO:tasks.workunit.client.0.smithi047.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-18T13:54:21.403 INFO:tasks.workunit.client.0.smithi047.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-18T13:54:21.403 INFO:tasks.workunit.client.0.smithi047.stderr: 2024-09-18T13:54:21.403 INFO:tasks.workunit.client.0.smithi047.stderr: git switch -c 2024-09-18T13:54:21.403 INFO:tasks.workunit.client.0.smithi047.stderr: 2024-09-18T13:54:21.404 INFO:tasks.workunit.client.0.smithi047.stderr:Or undo this operation with: 2024-09-18T13:54:21.404 INFO:tasks.workunit.client.0.smithi047.stderr: 2024-09-18T13:54:21.404 INFO:tasks.workunit.client.0.smithi047.stderr: git switch - 2024-09-18T13:54:21.404 INFO:tasks.workunit.client.0.smithi047.stderr: 2024-09-18T13:54:21.404 INFO:tasks.workunit.client.0.smithi047.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-18T13:54:21.404 INFO:tasks.workunit.client.0.smithi047.stderr: 2024-09-18T13:54:21.404 INFO:tasks.workunit.client.0.smithi047.stderr:HEAD is now at 7061ea6ca50 Merge branch 'wip-65946-quincy' of https://github.com/amathuria/ceph into wip-bharath11-testing-2024-09-18-1115-quincy 2024-09-18T13:54:21.408 DEBUG:teuthology.orchestra.run.smithi047:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-18T13:54:21.466 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-09-18T13:54:21.466 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-18T13:54:21.522 INFO:tasks.workunit:Running workunits matching osd on client.0... 2024-09-18T13:54:21.522 INFO:tasks.workunit:Running workunit osd/bad-inc-map.sh... 2024-09-18T13:54:21.522 DEBUG:teuthology.orchestra.run.smithi047:workunit test osd/bad-inc-map.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh 2024-09-18T13:54:21.590 INFO:tasks.workunit.client.0.smithi047.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T13:54:21.619 INFO:tasks.workunit.client.0.smithi047.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T13:54:21.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T13:54:21.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T13:54:21.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T13:54:21.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T13:54:21.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T13:54:21.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T13:54:21.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T13:54:21.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T13:54:21.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/bad-inc-map 2024-09-18T13:54:21.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:8: run: local dir=td/bad-inc-map 2024-09-18T13:54:21.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:9: run: shift 2024-09-18T13:54:21.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: export CEPH_MON=127.0.0.1:48237 2024-09-18T13:54:21.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: CEPH_MON=127.0.0.1:48237 2024-09-18T13:54:21.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:12: run: export CEPH_ARGS 2024-09-18T13:54:21.621 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: uuidgen 2024-09-18T13:54:21.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: CEPH_ARGS+='--fsid=ee7adae8-8693-4ea1-bf62-0a97390f9838 --auth-supported=none ' 2024-09-18T13:54:21.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:48237 ' 2024-09-18T13:54:21.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:15: run: set -e 2024-09-18T13:54:21.625 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: set 2024-09-18T13:54:21.625 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-18T13:54:21.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: local funcs=TEST_bad_inc_map 2024-09-18T13:54:21.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:18: run: for func in $funcs 2024-09-18T13:54:21.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:19: run: setup td/bad-inc-map 2024-09-18T13:54:21.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/bad-inc-map 2024-09-18T13:54:21.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/bad-inc-map 2024-09-18T13:54:21.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2024-09-18T13:54:21.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T13:54:21.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2024-09-18T13:54:21.628 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T13:54:21.628 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T13:54:21.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T13:54:21.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T13:54:21.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T13:54:21.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T13:54:21.629 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T13:54:21.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T13:54:21.630 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T13:54:21.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T13:54:21.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T13:54:21.631 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T13:54:21.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T13:54:21.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T13:54:21.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T13:54:21.633 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T13:54:21.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T13:54:21.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T13:54:21.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2024-09-18T13:54:21.635 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T13:54:21.635 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T13:54:21.635 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26745 2024-09-18T13:54:21.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26745 2024-09-18T13:54:21.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T13:54:21.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T13:54:21.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/bad-inc-map 2024-09-18T13:54:21.637 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T13:54:21.637 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T13:54:21.638 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26745 2024-09-18T13:54:21.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26745 2024-09-18T13:54:21.640 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T13:54:21.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T13:54:21.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T13:54:21.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T13:54:21.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/bad-inc-map 1' TERM HUP INT 2024-09-18T13:54:21.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:20: run: TEST_bad_inc_map td/bad-inc-map 2024-09-18T13:54:21.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:26: TEST_bad_inc_map: local dir=td/bad-inc-map 2024-09-18T13:54:21.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:28: TEST_bad_inc_map: run_mon td/bad-inc-map a 2024-09-18T13:54:21.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/bad-inc-map 2024-09-18T13:54:21.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T13:54:21.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T13:54:21.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T13:54:21.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/bad-inc-map/a 2024-09-18T13:54:21.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/bad-inc-map/a --run-dir=td/bad-inc-map 2024-09-18T13:54:22.047 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T13:54:22.047 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T13:54:22.047 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T13:54:22.047 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T13:54:22.047 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T13:54:22.047 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26745 2024-09-18T13:54:22.047 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26745/$cluster-$name.asok' 2024-09-18T13:54:22.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/bad-inc-map/a '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.26745/$cluster-$name.asok' --mon-cluster-log-file=td/bad-inc-map/log --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T13:54:22.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T13:54:22.316 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T13:54:22.316 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T13:54:22.316 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T13:54:22.316 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T13:54:22.316 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T13:54:22.316 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T13:54:22.316 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T13:54:22.316 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T13:54:22.317 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T13:54:22.317 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T13:54:22.317 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26745 2024-09-18T13:54:22.318 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26745/ceph-mon.a.asok 2024-09-18T13:54:22.318 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T13:54:22.318 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26745/ceph-mon.a.asok config get fsid 2024-09-18T13:54:22.399 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T13:54:22.399 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T13:54:22.399 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T13:54:22.399 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T13:54:22.399 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T13:54:22.399 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T13:54:22.399 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T13:54:22.400 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T13:54:22.400 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T13:54:22.400 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T13:54:22.400 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26745 2024-09-18T13:54:22.400 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26745/ceph-mon.a.asok 2024-09-18T13:54:22.400 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T13:54:22.400 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26745/ceph-mon.a.asok config get mon_host 2024-09-18T13:54:22.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:29: TEST_bad_inc_map: run_mgr td/bad-inc-map x 2024-09-18T13:54:22.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/bad-inc-map 2024-09-18T13:54:22.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T13:54:22.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T13:54:22.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T13:54:22.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/bad-inc-map/x 2024-09-18T13:54:22.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T13:54:23.328 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T13:54:23.328 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T13:54:23.329 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T13:54:23.329 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T13:54:23.329 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T13:54:23.329 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26745 2024-09-18T13:54:23.329 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26745/$cluster-$name.asok' 2024-09-18T13:54:23.329 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T13:54:23.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/bad-inc-map/x '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.26745/$cluster-$name.asok' --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T13:54:23.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:30: TEST_bad_inc_map: run_osd td/bad-inc-map 0 2024-09-18T13:54:23.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2024-09-18T13:54:23.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T13:54:23.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T13:54:23.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T13:54:23.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/0 2024-09-18T13:54:23.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ee7adae8-8693-4ea1-bf62-0a97390f9838 --auth-supported=none --mon-host=127.0.0.1:48237 ' 2024-09-18T13:54:23.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T13:54:23.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T13:54:23.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T13:54:23.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/0' 2024-09-18T13:54:23.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/0/journal' 2024-09-18T13:54:23.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T13:54:23.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T13:54:23.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-09-18T13:54:23.357 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T13:54:23.357 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T13:54:23.357 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T13:54:23.357 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T13:54:23.357 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T13:54:23.357 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26745 2024-09-18T13:54:23.357 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26745/$cluster-$name.asok' 2024-09-18T13:54:23.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26745/$cluster-$name.asok' 2024-09-18T13:54:23.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T13:54:23.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T13:54:23.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T13:54:23.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-09-18T13:54:23.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-09-18T13:54:23.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T13:54:23.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T13:54:23.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T13:54:23.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T13:54:23.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T13:54:23.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T13:54:23.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/0 2024-09-18T13:54:23.359 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T13:54:23.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a34ffdb7-4b85-47cd-bbee-cbc8bf0c9187 2024-09-18T13:54:23.359 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 a34ffdb7-4b85-47cd-bbee-cbc8bf0c9187 2024-09-18T13:54:23.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a34ffdb7-4b85-47cd-bbee-cbc8bf0c9187' 2024-09-18T13:54:23.359 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T13:54:23.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCP2+pmnG8AFhAAeC2w3i1pCcToQTEVomULXw== 2024-09-18T13:54:23.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCP2+pmnG8AFhAAeC2w3i1pCcToQTEVomULXw=="}' 2024-09-18T13:54:23.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a34ffdb7-4b85-47cd-bbee-cbc8bf0c9187 -i td/bad-inc-map/0/new.json 2024-09-18T13:54:23.646 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T13:54:23.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/0/new.json 2024-09-18T13:54:23.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=ee7adae8-8693-4ea1-bf62-0a97390f9838 --auth-supported=none --mon-host=127.0.0.1:48237 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.26745/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCP2+pmnG8AFhAAeC2w3i1pCcToQTEVomULXw== --osd-uuid a34ffdb7-4b85-47cd-bbee-cbc8bf0c9187 2024-09-18T13:54:23.711 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:54:23.711+0000 7f76ab155440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:54:23.713 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:54:23.713+0000 7f76ab155440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:54:23.714 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:54:23.714+0000 7f76ab155440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:54:23.714 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:54:23.714+0000 7f76ab155440 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2024-09-18T13:54:23.714 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:54:23.714+0000 7f76ab155440 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2024-09-18T13:54:23.715 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:54:23.714+0000 7f76ab155440 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2024-09-18T13:54:23.774 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:54:23.773+0000 7f76ab155440 -1 bluestore(td/bad-inc-map/0) _read_fsid unparsable uuid 2024-09-18T13:54:29.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/0/keyring 2024-09-18T13:54:29.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T13:54:29.525 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T13:54:29.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T13:54:29.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T13:54:30.033 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T13:54:30.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T13:54:30.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=ee7adae8-8693-4ea1-bf62-0a97390f9838 --auth-supported=none --mon-host=127.0.0.1:48237 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.26745/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T13:54:30.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T13:54:30.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T13:54:30.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T13:54:30.054 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:54:30.054+0000 7f843bc11440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:54:30.057 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:54:30.056+0000 7f843bc11440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:54:30.057 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:54:30.057+0000 7f843bc11440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:54:30.420 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T13:54:30.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T13:54:30.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T13:54:30.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T13:54:30.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T13:54:30.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:54:30.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T13:54:30.421 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T13:54:30.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:54:30.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:54:30.575 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:54:30.574+0000 7f843bc11440 -1 Falling back to public interface 2024-09-18T13:54:30.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:54:31.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:54:31.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:54:31.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T13:54:31.788 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T13:54:31.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:54:31.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:54:32.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:54:33.072 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:54:33.071+0000 7f843bc11440 -1 osd.0 0 log_to_monitors true 2024-09-18T13:54:33.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:54:33.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:54:33.165 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T13:54:33.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T13:54:33.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:54:33.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:54:33.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:54:34.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:54:34.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:54:34.666 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T13:54:34.666 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T13:54:34.666 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:54:34.666 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:54:35.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:54:36.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:54:36.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:54:36.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T13:54:36.070 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T13:54:36.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:54:36.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:54:36.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:54:37.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:54:37.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:54:37.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T13:54:37.475 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T13:54:37.476 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:54:37.476 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:54:37.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:54:38.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:54:38.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:54:38.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T13:54:38.891 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T13:54:38.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:54:38.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:54:39.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:54:40.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:54:40.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:54:40.274 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T13:54:40.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T13:54:40.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:54:40.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:54:40.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:54:41.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:54:41.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:54:41.803 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T13:54:41.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T13:54:41.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:54:41.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:54:42.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:54:43.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:54:43.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:54:43.180 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T13:54:43.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T13:54:43.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:54:43.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:54:43.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:54:44.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:54:44.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:54:44.556 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T13:54:44.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T13:54:44.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:54:44.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:54:45.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:54:46.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:54:46.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:54:46.517 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T13:54:46.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T13:54:46.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:54:46.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:54:47.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:54:48.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:54:48.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:54:48.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T13:54:48.028 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T13:54:48.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:54:48.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:54:48.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:54:49.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:54:49.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:54:49.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T13:54:49.855 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T13:54:49.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:54:49.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:54:50.746 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:54:51.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:54:51.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:54:51.748 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T13:54:51.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T13:54:51.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:54:51.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:54:52.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:54:53.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:54:53.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:54:53.296 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T13:54:53.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T13:54:53.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:54:53.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:54:54.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:54:55.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:54:55.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:54:55.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T13:54:55.238 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T13:54:55.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:54:55.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:54:56.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:54:57.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:54:57.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:54:57.365 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T13:54:57.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T13:54:57.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:54:57.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:54:57.792 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/27838,v1:127.0.0.1:6803/27838] [v2:127.0.0.1:6804/27838,v1:127.0.0.1:6805/27838] exists,up a34ffdb7-4b85-47cd-bbee-cbc8bf0c9187 2024-09-18T13:54:57.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T13:54:57.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T13:54:57.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T13:54:57.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:31: TEST_bad_inc_map: run_osd td/bad-inc-map 1 2024-09-18T13:54:57.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2024-09-18T13:54:57.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T13:54:57.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T13:54:57.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T13:54:57.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/1 2024-09-18T13:54:57.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ee7adae8-8693-4ea1-bf62-0a97390f9838 --auth-supported=none --mon-host=127.0.0.1:48237 ' 2024-09-18T13:54:57.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T13:54:57.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T13:54:57.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T13:54:57.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/1' 2024-09-18T13:54:57.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/1/journal' 2024-09-18T13:54:57.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T13:54:57.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T13:54:57.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-09-18T13:54:57.797 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T13:54:57.797 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T13:54:57.797 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T13:54:57.797 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T13:54:57.797 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T13:54:57.798 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26745 2024-09-18T13:54:57.798 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26745/$cluster-$name.asok' 2024-09-18T13:54:57.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26745/$cluster-$name.asok' 2024-09-18T13:54:57.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T13:54:57.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T13:54:57.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T13:54:57.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-09-18T13:54:57.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-09-18T13:54:57.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T13:54:57.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T13:54:57.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T13:54:57.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T13:54:57.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T13:54:57.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T13:54:57.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/1 2024-09-18T13:54:57.799 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T13:54:57.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2e57d9bc-8233-4bf3-9355-dc0cb7a51e2f 2024-09-18T13:54:57.799 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 2e57d9bc-8233-4bf3-9355-dc0cb7a51e2f 2024-09-18T13:54:57.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 2e57d9bc-8233-4bf3-9355-dc0cb7a51e2f' 2024-09-18T13:54:57.799 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T13:54:57.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCx2+pmN40xMBAA/MOFypMdswBxlzB4Xx25Xw== 2024-09-18T13:54:57.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCx2+pmN40xMBAA/MOFypMdswBxlzB4Xx25Xw=="}' 2024-09-18T13:54:57.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2e57d9bc-8233-4bf3-9355-dc0cb7a51e2f -i td/bad-inc-map/1/new.json 2024-09-18T13:54:58.325 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T13:54:58.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/1/new.json 2024-09-18T13:54:58.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=ee7adae8-8693-4ea1-bf62-0a97390f9838 --auth-supported=none --mon-host=127.0.0.1:48237 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.26745/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCx2+pmN40xMBAA/MOFypMdswBxlzB4Xx25Xw== --osd-uuid 2e57d9bc-8233-4bf3-9355-dc0cb7a51e2f 2024-09-18T13:54:58.359 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:54:58.359+0000 7fbab777d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:54:58.361 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:54:58.361+0000 7fbab777d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:54:58.362 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:54:58.362+0000 7fbab777d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:54:58.363 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:54:58.362+0000 7fbab777d440 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2024-09-18T13:54:58.363 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:54:58.362+0000 7fbab777d440 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2024-09-18T13:54:58.363 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:54:58.362+0000 7fbab777d440 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2024-09-18T13:54:58.407 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:54:58.407+0000 7fbab777d440 -1 bluestore(td/bad-inc-map/1) _read_fsid unparsable uuid 2024-09-18T13:55:03.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/1/keyring 2024-09-18T13:55:03.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T13:55:03.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T13:55:03.941 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T13:55:03.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T13:55:04.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T13:55:04.451 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T13:55:04.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=ee7adae8-8693-4ea1-bf62-0a97390f9838 --auth-supported=none --mon-host=127.0.0.1:48237 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.26745/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T13:55:04.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T13:55:04.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T13:55:04.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T13:55:04.472 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:55:04.471+0000 7fa7ccb21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:55:04.479 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:55:04.479+0000 7fa7ccb21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:55:04.481 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:55:04.480+0000 7fa7ccb21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:55:04.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T13:55:04.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T13:55:04.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T13:55:04.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T13:55:04.852 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T13:55:04.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T13:55:04.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:55:04.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T13:55:04.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:55:04.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:55:05.006 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:55:05.006+0000 7fa7ccb21440 -1 Falling back to public interface 2024-09-18T13:55:05.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:55:06.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:55:06.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:55:06.254 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T13:55:06.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T13:55:06.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:55:06.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:55:06.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:55:07.568 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:55:07.568+0000 7fa7ccb21440 -1 osd.1 0 log_to_monitors true 2024-09-18T13:55:07.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:55:07.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:55:07.630 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T13:55:07.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T13:55:07.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:55:07.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:55:08.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:55:09.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:55:09.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:55:09.221 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T13:55:09.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T13:55:09.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:55:09.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:55:09.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:55:10.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:55:10.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:55:10.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T13:55:10.635 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T13:55:10.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:55:10.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:55:11.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:55:12.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:55:12.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:55:12.040 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T13:55:12.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T13:55:12.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:55:12.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:55:12.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:55:13.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:55:13.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:55:13.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T13:55:13.455 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T13:55:13.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:55:13.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:55:13.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:55:14.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:55:14.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:55:14.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T13:55:14.937 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T13:55:14.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:55:14.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:55:15.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:55:16.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:55:16.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:55:16.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T13:55:16.494 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T13:55:16.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:55:16.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:55:16.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:55:17.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:55:17.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:55:17.805 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T13:55:17.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T13:55:17.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:55:17.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:55:18.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:55:19.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:55:19.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:55:19.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T13:55:19.310 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T13:55:19.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:55:19.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:55:19.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:55:20.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:55:20.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:55:20.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T13:55:20.925 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T13:55:20.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:55:20.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:55:21.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:55:22.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:55:22.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:55:22.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T13:55:22.999 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T13:55:23.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:55:23.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:55:23.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:55:24.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:55:24.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:55:24.871 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T13:55:24.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T13:55:24.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:55:24.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:55:25.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:55:26.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:55:26.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:55:26.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T13:55:26.729 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T13:55:26.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:55:26.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:55:27.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:55:28.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:55:28.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:55:28.765 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T13:55:28.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T13:55:28.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:55:28.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:55:29.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:55:30.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:55:30.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:55:30.220 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T13:55:30.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T13:55:30.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:55:30.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:55:30.622 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/29653,v1:127.0.0.1:6811/29653] [v2:127.0.0.1:6812/29653,v1:127.0.0.1:6813/29653] exists,up 2e57d9bc-8233-4bf3-9355-dc0cb7a51e2f 2024-09-18T13:55:30.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T13:55:30.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T13:55:30.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T13:55:30.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:32: TEST_bad_inc_map: run_osd td/bad-inc-map 2 2024-09-18T13:55:30.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2024-09-18T13:55:30.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T13:55:30.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T13:55:30.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T13:55:30.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/2 2024-09-18T13:55:30.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ee7adae8-8693-4ea1-bf62-0a97390f9838 --auth-supported=none --mon-host=127.0.0.1:48237 ' 2024-09-18T13:55:30.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T13:55:30.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T13:55:30.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T13:55:30.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/2' 2024-09-18T13:55:30.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/2/journal' 2024-09-18T13:55:30.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T13:55:30.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T13:55:30.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-09-18T13:55:30.627 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T13:55:30.628 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T13:55:30.628 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T13:55:30.628 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T13:55:30.628 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T13:55:30.628 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26745 2024-09-18T13:55:30.628 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26745/$cluster-$name.asok' 2024-09-18T13:55:30.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26745/$cluster-$name.asok' 2024-09-18T13:55:30.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T13:55:30.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T13:55:30.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T13:55:30.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-09-18T13:55:30.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-09-18T13:55:30.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T13:55:30.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T13:55:30.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T13:55:30.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T13:55:30.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T13:55:30.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T13:55:30.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/2 2024-09-18T13:55:30.629 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T13:55:30.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8cdc34aa-46d6-4880-9795-222a6dc423ef 2024-09-18T13:55:30.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 8cdc34aa-46d6-4880-9795-222a6dc423ef' 2024-09-18T13:55:30.629 INFO:tasks.workunit.client.0.smithi047.stdout:add osd2 8cdc34aa-46d6-4880-9795-222a6dc423ef 2024-09-18T13:55:30.630 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T13:55:30.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDS2+pmeTUSJhAA7hx+Uy8EiKbwb6hKJyqgcA== 2024-09-18T13:55:30.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDS2+pmeTUSJhAA7hx+Uy8EiKbwb6hKJyqgcA=="}' 2024-09-18T13:55:30.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8cdc34aa-46d6-4880-9795-222a6dc423ef -i td/bad-inc-map/2/new.json 2024-09-18T13:55:31.139 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T13:55:31.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/2/new.json 2024-09-18T13:55:31.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=ee7adae8-8693-4ea1-bf62-0a97390f9838 --auth-supported=none --mon-host=127.0.0.1:48237 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.26745/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDS2+pmeTUSJhAA7hx+Uy8EiKbwb6hKJyqgcA== --osd-uuid 8cdc34aa-46d6-4880-9795-222a6dc423ef 2024-09-18T13:55:31.174 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:55:31.173+0000 7f9788409440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:55:31.175 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:55:31.175+0000 7f9788409440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:55:31.176 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:55:31.176+0000 7f9788409440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:55:31.177 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:55:31.176+0000 7f9788409440 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2024-09-18T13:55:31.177 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:55:31.176+0000 7f9788409440 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2024-09-18T13:55:31.177 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:55:31.176+0000 7f9788409440 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2024-09-18T13:55:31.230 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:55:31.230+0000 7f9788409440 -1 bluestore(td/bad-inc-map/2) _read_fsid unparsable uuid 2024-09-18T13:55:36.993 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/2/keyring 2024-09-18T13:55:36.993 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T13:55:36.994 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd2 key to auth repository 2024-09-18T13:55:36.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T13:55:36.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T13:55:37.505 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T13:55:37.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T13:55:37.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=ee7adae8-8693-4ea1-bf62-0a97390f9838 --auth-supported=none --mon-host=127.0.0.1:48237 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.26745/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T13:55:37.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T13:55:37.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T13:55:37.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T13:55:37.526 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:55:37.525+0000 7f9dbce23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:55:37.553 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:55:37.552+0000 7f9dbce23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:55:37.554 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:55:37.553+0000 7f9dbce23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:55:37.949 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T13:55:37.949 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T13:55:37.949 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T13:55:37.949 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T13:55:37.949 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T13:55:37.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T13:55:37.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:55:37.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T13:55:37.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:55:37.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:55:38.091 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:55:38.091+0000 7f9dbce23440 -1 Falling back to public interface 2024-09-18T13:55:38.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:55:39.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:55:39.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:55:39.314 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T13:55:39.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T13:55:39.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:55:39.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:55:39.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:55:40.566 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:55:40.566+0000 7f9dbce23440 -1 osd.2 0 log_to_monitors true 2024-09-18T13:55:40.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:55:40.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:55:40.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T13:55:40.729 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T13:55:40.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:55:40.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:55:41.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:55:42.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:55:42.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:55:42.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T13:55:42.237 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T13:55:42.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:55:42.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:55:42.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:55:43.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:55:43.666 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:55:43.666 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T13:55:43.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T13:55:43.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:55:43.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:55:44.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:55:45.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:55:45.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:55:45.106 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T13:55:45.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T13:55:45.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:55:45.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:55:45.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:55:46.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:55:46.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:55:46.503 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T13:55:46.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T13:55:46.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:55:46.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:55:46.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:55:47.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:55:47.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:55:47.901 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T13:55:47.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T13:55:47.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:55:47.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:55:48.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:55:49.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:55:49.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:55:49.326 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T13:55:49.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T13:55:49.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:55:49.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:55:49.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:55:50.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:55:50.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:55:50.793 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T13:55:50.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T13:55:50.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:55:50.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:55:51.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:55:52.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:55:52.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:55:52.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T13:55:52.169 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T13:55:52.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:55:52.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:55:52.714 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:55:53.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:55:53.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:55:53.716 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T13:55:53.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T13:55:53.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:55:53.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:55:54.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:55:55.135 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:55:55.135 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:55:55.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T13:55:55.136 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T13:55:55.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:55:55.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:55:55.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:55:56.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:55:56.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:55:56.849 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T13:55:56.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T13:55:56.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:55:56.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:55:57.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:55:58.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:55:58.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:55:58.541 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T13:55:58.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T13:55:58.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:55:58.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:55:59.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:56:00.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:56:00.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:56:00.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T13:56:00.013 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T13:56:00.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:56:00.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:56:00.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:56:01.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:56:01.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:56:01.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T13:56:01.708 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T13:56:01.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:56:01.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:56:02.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:56:03.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:56:03.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:56:03.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T13:56:03.458 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T13:56:03.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:56:03.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:56:03.843 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/31382,v1:127.0.0.1:6819/31382] [v2:127.0.0.1:6820/31382,v1:127.0.0.1:6821/31382] exists,up 8cdc34aa-46d6-4880-9795-222a6dc423ef 2024-09-18T13:56:03.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T13:56:03.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T13:56:03.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T13:56:03.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:34: TEST_bad_inc_map: ceph config set osd.2 osd_inject_bad_map_crc_probability 1 2024-09-18T13:56:04.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:37: TEST_bad_inc_map: create_pool foo 8 2024-09-18T13:56:04.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 8 2024-09-18T13:56:04.854 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'foo' already exists 2024-09-18T13:56:04.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T13:56:05.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:38: TEST_bad_inc_map: ceph osd pool set foo min_size 1 2024-09-18T13:56:07.193 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 1 min_size to 1 2024-09-18T13:56:07.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:39: TEST_bad_inc_map: ceph osd pool set foo min_size 2 2024-09-18T13:56:08.577 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 1 min_size to 2 2024-09-18T13:56:08.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:41: TEST_bad_inc_map: sleep 5 2024-09-18T13:56:13.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: TIMEOUT=10 2024-09-18T13:56:13.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: wait_for_osd up 0 2024-09-18T13:56:13.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T13:56:13.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T13:56:13.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T13:56:13.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T13:56:13.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2024-09-18T13:56:13.593 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T13:56:13.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T13:56:13.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:56:13.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:56:14.006 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/27838,v1:127.0.0.1:6803/27838] [v2:127.0.0.1:6804/27838,v1:127.0.0.1:6805/27838] exists,up a34ffdb7-4b85-47cd-bbee-cbc8bf0c9187 2024-09-18T13:56:14.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T13:56:14.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T13:56:14.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T13:56:14.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: TIMEOUT=10 2024-09-18T13:56:14.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: wait_for_osd up 1 2024-09-18T13:56:14.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T13:56:14.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T13:56:14.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T13:56:14.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T13:56:14.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2024-09-18T13:56:14.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T13:56:14.008 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T13:56:14.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:56:14.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:56:14.415 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 10 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/29653,v1:127.0.0.1:6811/29653] [v2:127.0.0.1:6812/29653,v1:127.0.0.1:6813/29653] exists,up 2e57d9bc-8233-4bf3-9355-dc0cb7a51e2f 2024-09-18T13:56:14.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T13:56:14.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T13:56:14.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T13:56:14.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: TIMEOUT=10 2024-09-18T13:56:14.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: wait_for_osd up 2 2024-09-18T13:56:14.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T13:56:14.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T13:56:14.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T13:56:14.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T13:56:14.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2024-09-18T13:56:14.418 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T13:56:14.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T13:56:14.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:56:14.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:56:14.798 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 15 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/31382,v1:127.0.0.1:6819/31382] [v2:127.0.0.1:6820/31382,v1:127.0.0.1:6821/31382] exists,up 8cdc34aa-46d6-4880-9795-222a6dc423ef 2024-09-18T13:56:14.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T13:56:14.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T13:56:14.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T13:56:14.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:49: TEST_bad_inc_map: grep 'injecting map crc failure' td/bad-inc-map/osd.2.log 2024-09-18T13:56:14.800 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T13:56:04.692+0000 7f9dad304640 -1 osd.2 16 handle_osd_map injecting map crc failure 2024-09-18T13:56:14.800 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T13:56:05.230+0000 7f9dad304640 -1 osd.2 17 handle_osd_map injecting map crc failure 2024-09-18T13:56:14.800 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T13:56:05.482+0000 7f9dad304640 -1 osd.2 18 handle_osd_map injecting map crc failure 2024-09-18T13:56:14.800 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T13:56:06.194+0000 7f9dad304640 -1 osd.2 19 handle_osd_map injecting map crc failure 2024-09-18T13:56:14.800 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T13:56:06.790+0000 7f9dab300640 -1 osd.2 20 handle_osd_map injecting map crc failure 2024-09-18T13:56:14.800 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T13:56:07.283+0000 7f9dab300640 -1 osd.2 22 handle_osd_map injecting map crc failure 2024-09-18T13:56:14.801 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T13:56:07.284+0000 7f9dab300640 -1 osd.2 22 handle_osd_map injecting map crc failure 2024-09-18T13:56:14.801 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T13:56:07.595+0000 7f9dad304640 -1 osd.2 23 handle_osd_map injecting map crc failure 2024-09-18T13:56:14.801 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T13:56:07.963+0000 7f9dad304640 -1 osd.2 24 handle_osd_map injecting map crc failure 2024-09-18T13:56:14.801 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T13:56:08.575+0000 7f9dad304640 -1 osd.2 25 handle_osd_map injecting map crc failure 2024-09-18T13:56:14.801 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T13:56:08.577+0000 7f9dad304640 -1 osd.2 25 handle_osd_map injecting map crc failure 2024-09-18T13:56:14.801 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T13:56:09.503+0000 7f9dad304640 -1 osd.2 26 handle_osd_map injecting map crc failure 2024-09-18T13:56:14.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:50: TEST_bad_inc_map: grep 'bailing because last' td/bad-inc-map/osd.2.log 2024-09-18T13:56:14.802 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T13:56:04.693+0000 7f9dad304640 10 osd.2 16 handle_osd_map bailing because last < start (16<17) 2024-09-18T13:56:14.802 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T13:56:05.230+0000 7f9dad304640 10 osd.2 17 handle_osd_map bailing because last < start (17<18) 2024-09-18T13:56:14.802 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T13:56:05.482+0000 7f9dad304640 10 osd.2 18 handle_osd_map bailing because last < start (18<19) 2024-09-18T13:56:14.802 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T13:56:06.194+0000 7f9dad304640 10 osd.2 19 handle_osd_map bailing because last < start (19<20) 2024-09-18T13:56:14.802 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T13:56:06.790+0000 7f9dab300640 10 osd.2 20 handle_osd_map bailing because last < start (20<21) 2024-09-18T13:56:14.803 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T13:56:07.284+0000 7f9dab300640 10 osd.2 22 handle_osd_map bailing because last < start (22<23) 2024-09-18T13:56:14.803 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T13:56:07.284+0000 7f9dab300640 10 osd.2 22 handle_osd_map bailing because last < start (22<23) 2024-09-18T13:56:14.803 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T13:56:07.595+0000 7f9dad304640 10 osd.2 23 handle_osd_map bailing because last < start (23<24) 2024-09-18T13:56:14.803 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T13:56:07.963+0000 7f9dad304640 10 osd.2 24 handle_osd_map bailing because last < start (24<25) 2024-09-18T13:56:14.803 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T13:56:08.576+0000 7f9dad304640 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2024-09-18T13:56:14.803 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T13:56:08.577+0000 7f9dad304640 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2024-09-18T13:56:14.803 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T13:56:09.503+0000 7f9dad304640 10 osd.2 26 handle_osd_map bailing because last < start (26<27) 2024-09-18T13:56:14.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:52: TEST_bad_inc_map: echo success 2024-09-18T13:56:14.803 INFO:tasks.workunit.client.0.smithi047.stdout:success 2024-09-18T13:56:14.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:54: TEST_bad_inc_map: delete_pool foo 2024-09-18T13:56:14.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=foo 2024-09-18T13:56:14.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete foo foo --yes-i-really-really-mean-it 2024-09-18T13:56:15.359 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'foo' does not exist 2024-09-18T13:56:15.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:55: TEST_bad_inc_map: kill_daemons td/bad-inc-map 2024-09-18T13:56:15.370 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T13:56:15.371 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T13:56:15.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T13:56:15.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T13:56:15.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T13:56:20.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T13:56:20.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:21: run: teardown td/bad-inc-map 2024-09-18T13:56:20.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2024-09-18T13:56:20.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T13:56:20.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2024-09-18T13:56:20.686 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T13:56:20.686 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T13:56:20.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T13:56:20.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T13:56:20.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T13:56:20.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T13:56:20.692 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T13:56:20.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T13:56:20.693 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T13:56:20.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T13:56:20.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T13:56:20.695 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T13:56:20.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T13:56:20.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T13:56:20.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T13:56:20.696 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T13:56:20.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T13:56:20.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T13:56:20.699 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2024-09-18T13:56:20.714 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T13:56:20.714 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T13:56:20.714 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26745 2024-09-18T13:56:20.714 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26745 2024-09-18T13:56:20.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T13:56:20.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T13:56:20.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T13:56:20.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/bad-inc-map 0 2024-09-18T13:56:20.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2024-09-18T13:56:20.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T13:56:20.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2024-09-18T13:56:20.716 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T13:56:20.716 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T13:56:20.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T13:56:20.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T13:56:20.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T13:56:20.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T13:56:20.717 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T13:56:20.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T13:56:20.719 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T13:56:20.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T13:56:20.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T13:56:20.720 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T13:56:20.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T13:56:20.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T13:56:20.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T13:56:20.721 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T13:56:20.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T13:56:20.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T13:56:20.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2024-09-18T13:56:20.725 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T13:56:20.725 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T13:56:20.725 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26745 2024-09-18T13:56:20.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26745 2024-09-18T13:56:20.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T13:56:20.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T13:56:20.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T13:56:20.727 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T13:56:20.727 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T13:56:20.797 INFO:tasks.workunit:Running workunit osd/divergent-priors.sh... 2024-09-18T13:56:20.797 DEBUG:teuthology.orchestra.run.smithi047:workunit test osd/divergent-priors.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2024-09-18T13:56:20.864 INFO:tasks.workunit.client.0.smithi047.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T13:56:20.868 INFO:tasks.workunit.client.0.smithi047.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T13:56:20.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T13:56:20.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T13:56:20.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T13:56:20.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T13:56:20.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T13:56:20.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T13:56:20.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T13:56:20.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T13:56:20.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/divergent-priors 2024-09-18T13:56:20.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:21: run: local dir=td/divergent-priors 2024-09-18T13:56:20.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:22: run: shift 2024-09-18T13:56:20.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: export loglen=12 2024-09-18T13:56:20.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: loglen=12 2024-09-18T13:56:20.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: export divisor=3 2024-09-18T13:56:20.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: divisor=3 2024-09-18T13:56:20.869 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: expr 12 / 2 2024-09-18T13:56:20.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: export trim=6 2024-09-18T13:56:20.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: trim=6 2024-09-18T13:56:20.870 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: expr 6 / 3 2024-09-18T13:56:20.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: export DIVERGENT_WRITE=2 2024-09-18T13:56:20.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: DIVERGENT_WRITE=2 2024-09-18T13:56:20.870 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: expr 6 / 3 2024-09-18T13:56:20.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: export DIVERGENT_REMOVE=2 2024-09-18T13:56:20.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: DIVERGENT_REMOVE=2 2024-09-18T13:56:20.871 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: expr 6 / 3 2024-09-18T13:56:20.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: export DIVERGENT_CREATE=2 2024-09-18T13:56:20.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: DIVERGENT_CREATE=2 2024-09-18T13:56:20.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: export poolname=test 2024-09-18T13:56:20.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: poolname=test 2024-09-18T13:56:20.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: export testobjects=100 2024-09-18T13:56:20.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: testobjects=100 2024-09-18T13:56:20.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: export CEPH_MON=127.0.0.1:7115 2024-09-18T13:56:20.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: CEPH_MON=127.0.0.1:7115 2024-09-18T13:56:20.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:35: run: export CEPH_ARGS 2024-09-18T13:56:20.873 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: uuidgen 2024-09-18T13:56:20.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: CEPH_ARGS+='--fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none ' 2024-09-18T13:56:20.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:37: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2024-09-18T13:56:20.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:39: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T13:56:20.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:40: run: CEPH_ARGS+='--osd_debug_pg_log_writeout=true ' 2024-09-18T13:56:20.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:41: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T13:56:20.874 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: set 2024-09-18T13:56:20.875 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-18T13:56:20.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: local 'funcs=TEST_divergent 2024-09-18T13:56:20.877 INFO:tasks.workunit.client.0.smithi047.stderr:TEST_divergent_2 2024-09-18T13:56:20.877 INFO:tasks.workunit.client.0.smithi047.stderr:TEST_divergent_3 2024-09-18T13:56:20.877 INFO:tasks.workunit.client.0.smithi047.stderr:TEST_divergent_ec' 2024-09-18T13:56:20.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-18T13:56:20.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-18T13:56:20.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-09-18T13:56:20.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-09-18T13:56:20.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-18T13:56:20.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T13:56:20.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-18T13:56:20.878 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T13:56:20.878 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T13:56:20.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T13:56:20.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T13:56:20.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T13:56:20.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T13:56:20.879 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T13:56:20.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T13:56:20.880 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T13:56:20.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T13:56:20.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T13:56:20.881 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T13:56:20.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T13:56:20.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T13:56:20.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T13:56:20.883 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T13:56:20.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T13:56:20.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T13:56:20.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-18T13:56:20.885 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T13:56:20.886 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T13:56:20.886 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T13:56:20.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.32609 2024-09-18T13:56:20.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T13:56:20.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T13:56:20.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-09-18T13:56:20.888 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T13:56:20.888 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T13:56:20.888 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T13:56:20.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.32609 2024-09-18T13:56:20.890 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T13:56:20.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T13:56:20.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T13:56:20.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T13:56:20.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-18T13:56:20.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent td/divergent-priors 2024-09-18T13:56:20.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:57: TEST_divergent: local dir=td/divergent-priors 2024-09-18T13:56:20.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:60: TEST_divergent: local dummyfile=/etc/fstab 2024-09-18T13:56:20.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:61: TEST_divergent: local dummyfile2=/etc/resolv.conf 2024-09-18T13:56:20.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:63: TEST_divergent: local num_osds=3 2024-09-18T13:56:20.891 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: expr 3 - 1 2024-09-18T13:56:20.891 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: seq 0 2 2024-09-18T13:56:20.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: local 'osds=0 2024-09-18T13:56:20.900 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T13:56:20.900 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T13:56:20.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:65: TEST_divergent: run_mon td/divergent-priors a 2024-09-18T13:56:20.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-09-18T13:56:20.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T13:56:20.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T13:56:20.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T13:56:20.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-09-18T13:56:20.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-18T13:56:21.258 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T13:56:21.258 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T13:56:21.258 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T13:56:21.258 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T13:56:21.258 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T13:56:21.259 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T13:56:21.259 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T13:56:21.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T13:56:21.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T13:56:21.496 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T13:56:21.496 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T13:56:21.496 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T13:56:21.496 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T13:56:21.496 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T13:56:21.496 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T13:56:21.496 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T13:56:21.496 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T13:56:21.496 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T13:56:21.496 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T13:56:21.496 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T13:56:21.496 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32609/ceph-mon.a.asok 2024-09-18T13:56:21.497 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T13:56:21.497 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.32609/ceph-mon.a.asok config get fsid 2024-09-18T13:56:21.574 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T13:56:21.574 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T13:56:21.575 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T13:56:21.575 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T13:56:21.575 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T13:56:21.575 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T13:56:21.575 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T13:56:21.575 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T13:56:21.575 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T13:56:21.575 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T13:56:21.575 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T13:56:21.575 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32609/ceph-mon.a.asok 2024-09-18T13:56:21.575 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T13:56:21.575 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.32609/ceph-mon.a.asok config get mon_host 2024-09-18T13:56:21.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:66: TEST_divergent: run_mgr td/divergent-priors x 2024-09-18T13:56:21.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-09-18T13:56:21.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T13:56:21.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T13:56:21.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T13:56:21.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-09-18T13:56:21.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T13:56:22.517 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T13:56:22.517 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T13:56:22.517 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T13:56:22.517 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T13:56:22.517 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T13:56:22.517 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T13:56:22.517 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T13:56:22.517 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T13:56:22.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T13:56:22.549 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-09-18T13:56:22.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 0 2024-09-18T13:56:22.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-18T13:56:22.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T13:56:22.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T13:56:22.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T13:56:22.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-09-18T13:56:22.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T13:56:22.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T13:56:22.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T13:56:22.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T13:56:22.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-18T13:56:22.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-18T13:56:22.551 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T13:56:22.551 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T13:56:22.551 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-18T13:56:22.551 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T13:56:22.551 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T13:56:22.551 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T13:56:22.551 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T13:56:22.551 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T13:56:22.551 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T13:56:22.551 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T13:56:22.551 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T13:56:22.551 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T13:56:22.551 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T13:56:22.551 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T13:56:22.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-18T13:56:22.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-18T13:56:22.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T13:56:22.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T13:56:22.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T13:56:22.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T13:56:22.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T13:56:22.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T13:56:22.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-09-18T13:56:22.552 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T13:56:22.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=120d8a51-9d44-40cc-b688-4581426ff01d 2024-09-18T13:56:22.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 120d8a51-9d44-40cc-b688-4581426ff01d' 2024-09-18T13:56:22.553 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 120d8a51-9d44-40cc-b688-4581426ff01d 2024-09-18T13:56:22.553 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T13:56:22.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAG3OpmhhruIRAA/Jc8p4bi0UWCqdVRKuQNmA== 2024-09-18T13:56:22.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAG3OpmhhruIRAA/Jc8p4bi0UWCqdVRKuQNmA=="}' 2024-09-18T13:56:22.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 120d8a51-9d44-40cc-b688-4581426ff01d -i td/divergent-priors/0/new.json 2024-09-18T13:56:22.867 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T13:56:22.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-09-18T13:56:22.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAG3OpmhhruIRAA/Jc8p4bi0UWCqdVRKuQNmA== --osd-uuid 120d8a51-9d44-40cc-b688-4581426ff01d 2024-09-18T13:56:22.910 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:56:22.909+0000 7fd75cc0c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:56:22.912 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:56:22.911+0000 7fd75cc0c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:56:22.913 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:56:22.912+0000 7fd75cc0c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:56:22.913 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:56:22.913+0000 7fd75cc0c440 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-18T13:56:22.913 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:56:22.913+0000 7fd75cc0c440 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-18T13:56:22.913 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:56:22.913+0000 7fd75cc0c440 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-18T13:56:22.951 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:56:22.951+0000 7fd75cc0c440 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-18T13:56:28.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-18T13:56:28.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T13:56:28.309 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T13:56:28.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T13:56:28.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T13:56:28.857 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T13:56:28.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T13:56:28.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T13:56:28.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T13:56:28.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T13:56:28.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T13:56:28.889 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:56:28.888+0000 7f4503f25440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:56:28.910 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:56:28.910+0000 7f4503f25440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:56:28.911 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:56:28.911+0000 7f4503f25440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:56:29.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T13:56:29.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T13:56:29.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T13:56:29.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T13:56:29.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T13:56:29.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:56:29.270 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T13:56:29.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T13:56:29.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:56:29.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:56:29.440 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:56:29.440+0000 7f4503f25440 -1 Falling back to public interface 2024-09-18T13:56:29.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:56:30.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:56:30.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:56:30.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T13:56:30.684 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T13:56:30.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:56:30.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:56:31.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:56:31.906 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:56:31.905+0000 7f4503f25440 -1 osd.0 0 log_to_monitors true 2024-09-18T13:56:32.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:56:32.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:56:32.110 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T13:56:32.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T13:56:32.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:56:32.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:56:32.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:56:33.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:56:33.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:56:33.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T13:56:33.616 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T13:56:33.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:56:33.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:56:34.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:56:35.021 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:56:35.021 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:56:35.021 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T13:56:35.021 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T13:56:35.022 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:56:35.022 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:56:35.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:56:36.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:56:36.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:56:36.410 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T13:56:36.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T13:56:36.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:56:36.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:56:36.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:56:37.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:56:37.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:56:37.804 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T13:56:37.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T13:56:37.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:56:37.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:56:38.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:56:39.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:56:39.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:56:39.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T13:56:39.229 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T13:56:39.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:56:39.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:56:39.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:56:40.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:56:40.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:56:40.719 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T13:56:40.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T13:56:40.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:56:40.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:56:41.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:56:42.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:56:42.132 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T13:56:42.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:56:42.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T13:56:42.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:56:42.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:56:42.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:56:43.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:56:43.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:56:43.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T13:56:43.600 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T13:56:43.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:56:43.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:56:44.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:56:45.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:56:45.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:56:45.512 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T13:56:45.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T13:56:45.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:56:45.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:56:46.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:56:47.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:56:47.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:56:47.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T13:56:47.069 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T13:56:47.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:56:47.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:56:47.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:56:48.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:56:48.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:56:48.735 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T13:56:48.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T13:56:48.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:56:48.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:56:49.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:56:50.746 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:56:50.746 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:56:50.747 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T13:56:50.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T13:56:50.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:56:50.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:56:51.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:56:52.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:56:52.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:56:52.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T13:56:52.241 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T13:56:52.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:56:52.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:56:52.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:56:53.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:56:53.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:56:53.782 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T13:56:53.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T13:56:53.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:56:53.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:56:54.230 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/33718,v1:127.0.0.1:6803/33718] [v2:127.0.0.1:6804/33718,v1:127.0.0.1:6805/33718] exists,up 120d8a51-9d44-40cc-b688-4581426ff01d 2024-09-18T13:56:54.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T13:56:54.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T13:56:54.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T13:56:54.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-09-18T13:56:54.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 1 2024-09-18T13:56:54.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-18T13:56:54.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T13:56:54.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T13:56:54.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T13:56:54.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-09-18T13:56:54.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T13:56:54.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T13:56:54.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T13:56:54.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T13:56:54.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-18T13:56:54.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-18T13:56:54.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T13:56:54.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T13:56:54.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-18T13:56:54.235 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T13:56:54.236 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T13:56:54.236 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T13:56:54.236 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T13:56:54.236 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T13:56:54.236 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T13:56:54.236 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T13:56:54.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T13:56:54.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T13:56:54.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T13:56:54.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T13:56:54.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-18T13:56:54.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-18T13:56:54.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T13:56:54.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T13:56:54.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T13:56:54.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T13:56:54.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T13:56:54.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T13:56:54.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-09-18T13:56:54.237 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T13:56:54.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3353f33a-e483-4b8b-976d-f7b4acd36b36 2024-09-18T13:56:54.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 3353f33a-e483-4b8b-976d-f7b4acd36b36' 2024-09-18T13:56:54.237 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 3353f33a-e483-4b8b-976d-f7b4acd36b36 2024-09-18T13:56:54.238 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T13:56:54.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAm3OpmgkAZDxAAnkxBdM8smF/MycU1cPXZhg== 2024-09-18T13:56:54.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAm3OpmgkAZDxAAnkxBdM8smF/MycU1cPXZhg=="}' 2024-09-18T13:56:54.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3353f33a-e483-4b8b-976d-f7b4acd36b36 -i td/divergent-priors/1/new.json 2024-09-18T13:56:54.770 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T13:56:54.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-09-18T13:56:54.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAm3OpmgkAZDxAAnkxBdM8smF/MycU1cPXZhg== --osd-uuid 3353f33a-e483-4b8b-976d-f7b4acd36b36 2024-09-18T13:56:54.814 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:56:54.813+0000 7efe74b1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:56:54.816 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:56:54.815+0000 7efe74b1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:56:54.817 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:56:54.816+0000 7efe74b1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:56:54.817 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:56:54.817+0000 7efe74b1f440 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-18T13:56:54.817 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:56:54.817+0000 7efe74b1f440 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-18T13:56:54.817 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:56:54.817+0000 7efe74b1f440 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-18T13:56:55.027 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:56:55.027+0000 7efe74b1f440 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-18T13:57:00.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-18T13:57:00.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T13:57:00.943 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T13:57:00.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T13:57:00.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T13:57:01.466 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T13:57:01.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T13:57:01.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T13:57:01.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T13:57:01.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T13:57:01.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T13:57:01.497 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:57:01.496+0000 7f7428d2f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:57:01.517 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:57:01.517+0000 7f7428d2f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:57:01.518 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:57:01.518+0000 7f7428d2f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:57:01.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T13:57:01.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T13:57:01.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T13:57:01.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T13:57:01.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T13:57:01.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:57:01.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T13:57:01.867 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T13:57:01.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:57:01.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:57:02.049 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:57:02.049+0000 7f7428d2f440 -1 Falling back to public interface 2024-09-18T13:57:02.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:57:03.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:57:03.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:57:03.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T13:57:03.283 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T13:57:03.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:57:03.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:57:03.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:57:04.597 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:57:04.597+0000 7f7428d2f440 -1 osd.1 0 log_to_monitors true 2024-09-18T13:57:04.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:57:04.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:57:04.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T13:57:04.710 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T13:57:04.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:57:04.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:57:05.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:57:06.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:57:06.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:57:06.284 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T13:57:06.285 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T13:57:06.285 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:57:06.285 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:57:06.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:57:07.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:57:07.730 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T13:57:07.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:57:07.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T13:57:07.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:57:07.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:57:08.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:57:09.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:57:09.149 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T13:57:09.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:57:09.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T13:57:09.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:57:09.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:57:09.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:57:10.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:57:10.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:57:10.558 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T13:57:10.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T13:57:10.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:57:10.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:57:11.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:57:12.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:57:12.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:57:12.033 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T13:57:12.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T13:57:12.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:57:12.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:57:12.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:57:13.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:57:13.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:57:13.563 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T13:57:13.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T13:57:13.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:57:13.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:57:13.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:57:14.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:57:14.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:57:14.977 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T13:57:14.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T13:57:14.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:57:14.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:57:15.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:57:16.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:57:16.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:57:16.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T13:57:16.467 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T13:57:16.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:57:16.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:57:17.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:57:18.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:57:18.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:57:18.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T13:57:18.641 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T13:57:18.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:57:18.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:57:19.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:57:20.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:57:20.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:57:20.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T13:57:20.978 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T13:57:20.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:57:20.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:57:21.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:57:22.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:57:22.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:57:22.821 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T13:57:22.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T13:57:22.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:57:22.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:57:24.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:57:25.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:57:25.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:57:25.017 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T13:57:25.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T13:57:25.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:57:25.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:57:26.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:57:27.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:57:27.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:57:27.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T13:57:27.018 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T13:57:27.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:57:27.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:57:27.448 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/35448,v1:127.0.0.1:6811/35448] [v2:127.0.0.1:6812/35448,v1:127.0.0.1:6813/35448] exists,up 3353f33a-e483-4b8b-976d-f7b4acd36b36 2024-09-18T13:57:27.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T13:57:27.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T13:57:27.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T13:57:27.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-09-18T13:57:27.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 2 2024-09-18T13:57:27.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-18T13:57:27.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T13:57:27.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T13:57:27.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T13:57:27.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-09-18T13:57:27.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T13:57:27.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T13:57:27.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T13:57:27.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T13:57:27.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-18T13:57:27.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-18T13:57:27.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T13:57:27.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T13:57:27.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-18T13:57:27.453 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T13:57:27.453 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T13:57:27.453 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T13:57:27.454 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T13:57:27.454 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T13:57:27.454 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T13:57:27.454 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T13:57:27.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T13:57:27.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T13:57:27.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T13:57:27.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T13:57:27.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-18T13:57:27.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-18T13:57:27.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T13:57:27.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T13:57:27.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T13:57:27.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T13:57:27.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T13:57:27.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T13:57:27.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-09-18T13:57:27.455 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T13:57:27.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d1a5f547-90ee-411b-9aab-d214098efc62 2024-09-18T13:57:27.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 d1a5f547-90ee-411b-9aab-d214098efc62' 2024-09-18T13:57:27.456 INFO:tasks.workunit.client.0.smithi047.stdout:add osd2 d1a5f547-90ee-411b-9aab-d214098efc62 2024-09-18T13:57:27.456 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T13:57:27.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBH3Opmne8mHBAA5Src0jq57S+p0Uv6fOBfSw== 2024-09-18T13:57:27.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBH3Opmne8mHBAA5Src0jq57S+p0Uv6fOBfSw=="}' 2024-09-18T13:57:27.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d1a5f547-90ee-411b-9aab-d214098efc62 -i td/divergent-priors/2/new.json 2024-09-18T13:57:28.065 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T13:57:28.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-09-18T13:57:28.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBH3Opmne8mHBAA5Src0jq57S+p0Uv6fOBfSw== --osd-uuid d1a5f547-90ee-411b-9aab-d214098efc62 2024-09-18T13:57:28.108 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:57:28.108+0000 7fae8ea15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:57:28.123 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:57:28.122+0000 7fae8ea15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:57:28.124 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:57:28.123+0000 7fae8ea15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:57:28.124 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:57:28.124+0000 7fae8ea15440 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-18T13:57:28.124 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:57:28.124+0000 7fae8ea15440 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-18T13:57:28.125 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:57:28.124+0000 7fae8ea15440 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-18T13:57:28.189 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:57:28.189+0000 7fae8ea15440 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-18T13:57:34.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-18T13:57:34.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T13:57:34.083 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd2 key to auth repository 2024-09-18T13:57:34.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T13:57:34.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T13:57:34.615 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T13:57:34.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T13:57:34.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T13:57:34.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T13:57:34.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T13:57:34.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T13:57:34.645 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:57:34.645+0000 7f2bdd04e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:57:34.673 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:57:34.672+0000 7f2bdd04e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:57:34.713 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:57:34.673+0000 7f2bdd04e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:57:34.993 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T13:57:34.993 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T13:57:34.993 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T13:57:34.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T13:57:34.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T13:57:34.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:57:34.994 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T13:57:34.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T13:57:34.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:57:34.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:57:35.197 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:57:35.197+0000 7f2bdd04e440 -1 Falling back to public interface 2024-09-18T13:57:35.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:57:36.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:57:36.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:57:36.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T13:57:36.379 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T13:57:36.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:57:36.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:57:36.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:57:37.675 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:57:37.675+0000 7f2bdd04e440 -1 osd.2 0 log_to_monitors true 2024-09-18T13:57:37.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:57:37.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:57:37.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T13:57:37.766 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T13:57:37.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:57:37.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:57:38.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:57:39.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:57:39.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:57:39.414 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T13:57:39.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T13:57:39.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:57:39.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:57:39.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:57:40.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:57:40.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:57:40.802 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T13:57:40.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T13:57:40.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:57:40.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:57:41.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:57:42.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:57:42.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:57:42.239 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T13:57:42.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T13:57:42.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:57:42.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:57:42.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:57:43.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:57:43.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:57:43.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T13:57:43.873 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T13:57:43.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:57:43.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:57:44.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:57:45.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:57:45.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:57:45.305 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T13:57:45.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T13:57:45.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:57:45.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:57:45.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:57:46.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:57:46.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:57:46.699 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T13:57:46.699 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T13:57:46.699 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:57:46.699 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:57:47.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:57:48.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:57:48.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:57:48.197 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T13:57:48.197 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T13:57:48.197 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:57:48.197 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:57:48.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:57:49.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:57:49.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:57:49.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T13:57:49.694 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T13:57:49.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:57:49.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:57:50.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:57:51.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:57:51.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:57:51.234 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T13:57:51.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T13:57:51.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:57:51.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:57:51.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:57:52.985 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:57:52.985 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:57:52.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T13:57:52.986 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T13:57:52.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:57:52.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:57:54.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:57:55.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:57:55.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:57:55.057 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T13:57:55.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T13:57:55.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:57:55.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:57:56.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:57:57.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:57:57.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:57:57.017 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T13:57:57.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T13:57:57.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:57:57.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:57:57.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:57:58.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:57:58.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:57:58.670 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T13:57:58.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T13:57:58.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:57:58.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:57:59.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:58:00.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:58:00.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:58:00.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T13:58:00.859 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T13:58:00.860 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:58:00.860 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:58:01.246 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/37152,v1:127.0.0.1:6819/37152] [v2:127.0.0.1:6820/37152,v1:127.0.0.1:6821/37152] exists,up d1a5f547-90ee-411b-9aab-d214098efc62 2024-09-18T13:58:01.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T13:58:01.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T13:58:01.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T13:58:01.247 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:72: TEST_divergent: ceph osd set noout 2024-09-18T13:58:01.920 INFO:tasks.workunit.client.0.smithi047.stderr:noout is set 2024-09-18T13:58:01.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:73: TEST_divergent: ceph osd set noin 2024-09-18T13:58:02.774 INFO:tasks.workunit.client.0.smithi047.stderr:noin is set 2024-09-18T13:58:02.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:74: TEST_divergent: ceph osd set nodown 2024-09-18T13:58:03.636 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T13:58:03.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:75: TEST_divergent: create_pool test 1 1 2024-09-18T13:58:03.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-18T13:58:04.387 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test' already exists 2024-09-18T13:58:04.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T13:58:05.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:76: TEST_divergent: ceph osd pool set test size 3 2024-09-18T13:58:06.389 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 1 size to 3 2024-09-18T13:58:06.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:77: TEST_divergent: ceph osd pool set test min_size 2 2024-09-18T13:58:07.260 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 1 min_size to 2 2024-09-18T13:58:07.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:79: TEST_divergent: flush_pg_stats 2024-09-18T13:58:07.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T13:58:07.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T13:58:07.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T13:58:07.656 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T13:58:07.656 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T13:58:07.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T13:58:07.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T13:58:07.657 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T13:58:07.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836499 2024-09-18T13:58:07.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836499 2024-09-18T13:58:07.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836499' 2024-09-18T13:58:07.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T13:58:07.773 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T13:58:07.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672977 2024-09-18T13:58:07.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672977 2024-09-18T13:58:07.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836499 1-42949672977' 2024-09-18T13:58:07.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T13:58:07.889 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T13:58:08.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509445 2024-09-18T13:58:08.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509445 2024-09-18T13:58:08.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836499 1-42949672977 2-64424509445' 2024-09-18T13:58:08.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T13:58:08.006 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836499 2024-09-18T13:58:08.006 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T13:58:08.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T13:58:08.007 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836499 2024-09-18T13:58:08.007 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T13:58:08.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836499 2024-09-18T13:58:08.008 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836499 2024-09-18T13:58:08.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-09-18T13:58:08.009 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T13:58:08.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836499 -lt 21474836499 2024-09-18T13:58:08.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T13:58:08.381 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672977 2024-09-18T13:58:08.381 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T13:58:08.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T13:58:08.382 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672977 2024-09-18T13:58:08.383 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T13:58:08.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672977 2024-09-18T13:58:08.384 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672977 2024-09-18T13:58:08.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-09-18T13:58:08.384 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T13:58:08.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672977 -lt 42949672977 2024-09-18T13:58:08.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T13:58:08.776 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509445 2024-09-18T13:58:08.776 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T13:58:08.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T13:58:08.778 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509445 2024-09-18T13:58:08.778 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T13:58:08.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509445 2024-09-18T13:58:08.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-09-18T13:58:08.779 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509445 2024-09-18T13:58:08.779 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T13:58:09.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509445 -lt 64424509445 2024-09-18T13:58:09.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:80: TEST_divergent: wait_for_clean 2024-09-18T13:58:09.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T13:58:09.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T13:58:09.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T13:58:09.164 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T13:58:09.164 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T13:58:09.165 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T13:58:09.165 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T13:58:09.165 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T13:58:09.165 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T13:58:09.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T13:58:09.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T13:58:09.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T13:58:09.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T13:58:09.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T13:58:09.254 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T13:58:09.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T13:58:09.713 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T13:58:09.713 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T13:58:09.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T13:58:09.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T13:58:09.713 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T13:58:09.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836502 2024-09-18T13:58:09.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836502 2024-09-18T13:58:09.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836502' 2024-09-18T13:58:09.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T13:58:09.830 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T13:58:09.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672979 2024-09-18T13:58:09.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672979 2024-09-18T13:58:09.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836502 1-42949672979' 2024-09-18T13:58:09.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T13:58:09.945 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T13:58:10.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509447 2024-09-18T13:58:10.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509447 2024-09-18T13:58:10.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836502 1-42949672979 2-64424509447' 2024-09-18T13:58:10.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T13:58:10.062 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836502 2024-09-18T13:58:10.062 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T13:58:10.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T13:58:10.063 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836502 2024-09-18T13:58:10.063 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T13:58:10.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836502 2024-09-18T13:58:10.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-09-18T13:58:10.065 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836502 2024-09-18T13:58:10.065 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T13:58:10.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836502 -lt 21474836502 2024-09-18T13:58:10.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T13:58:10.462 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672979 2024-09-18T13:58:10.462 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T13:58:10.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T13:58:10.463 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672979 2024-09-18T13:58:10.463 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T13:58:10.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672979 2024-09-18T13:58:10.465 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672979 2024-09-18T13:58:10.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-09-18T13:58:10.465 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T13:58:10.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672979 -lt 42949672979 2024-09-18T13:58:10.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T13:58:10.844 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509447 2024-09-18T13:58:10.845 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T13:58:10.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T13:58:10.846 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509447 2024-09-18T13:58:10.846 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T13:58:10.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509447 2024-09-18T13:58:10.847 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509447 2024-09-18T13:58:10.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-09-18T13:58:10.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T13:58:11.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509447 -lt 64424509447 2024-09-18T13:58:11.219 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T13:58:11.219 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T13:58:11.219 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T13:58:11.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T13:58:11.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T13:58:11.723 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T13:58:11.723 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T13:58:11.723 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T13:58:11.723 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T13:58:11.723 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T13:58:11.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T13:58:12.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T13:58:12.086 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T13:58:12.086 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T13:58:12.086 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T13:58:12.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T13:58:12.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T13:58:12.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T13:58:12.618 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: ceph pg dump pgs --format=json 2024-09-18T13:58:12.618 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: jq '.pg_stats[0].up_primary' 2024-09-18T13:58:12.971 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T13:58:12.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: local divergent=1 2024-09-18T13:58:12.982 INFO:tasks.workunit.client.0.smithi047.stdout:primary and soon to be divergent is 1 2024-09-18T13:58:12.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:84: TEST_divergent: echo 'primary and soon to be divergent is 1' 2024-09-18T13:58:12.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:85: TEST_divergent: ceph pg dump pgs 2024-09-18T13:58:13.338 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T13:58:13.338 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T13:58:06.125993+0000 0'0 26:12 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T13:58:04.104186+0000 0'0 2024-09-18T13:58:04.104186+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:36:30.459830+0000 0 0 2024-09-18T13:58:13.338 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T13:58:13.338 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T13:58:13.338 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T13:58:13.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:86: TEST_divergent: local non_divergent= 2024-09-18T13:58:13.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-09-18T13:58:13.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 0 = 1 ']' 2024-09-18T13:58:13.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0' 2024-09-18T13:58:13.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-09-18T13:58:13.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 1 = 1 ']' 2024-09-18T13:58:13.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:90: TEST_divergent: continue 2024-09-18T13:58:13.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-09-18T13:58:13.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 2 = 1 ']' 2024-09-18T13:58:13.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0 2' 2024-09-18T13:58:13.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:95: TEST_divergent: echo 'writing initial objects' 2024-09-18T13:58:13.350 INFO:tasks.workunit.client.0.smithi047.stdout:writing initial objects 2024-09-18T13:58:13.351 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: seq 1 100 2024-09-18T13:58:13.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:13.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_1 /etc/fstab 2024-09-18T13:58:13.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:13.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_2 /etc/fstab 2024-09-18T13:58:13.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:13.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_3 /etc/fstab 2024-09-18T13:58:13.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:13.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_4 /etc/fstab 2024-09-18T13:58:13.946 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:13.946 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_5 /etc/fstab 2024-09-18T13:58:14.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:14.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_6 /etc/fstab 2024-09-18T13:58:14.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:14.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_7 /etc/fstab 2024-09-18T13:58:14.390 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:14.390 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_8 /etc/fstab 2024-09-18T13:58:14.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:14.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_9 /etc/fstab 2024-09-18T13:58:14.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:14.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_10 /etc/fstab 2024-09-18T13:58:14.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:14.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_11 /etc/fstab 2024-09-18T13:58:15.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:15.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_12 /etc/fstab 2024-09-18T13:58:15.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:15.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_13 /etc/fstab 2024-09-18T13:58:15.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:15.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_14 /etc/fstab 2024-09-18T13:58:15.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:15.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_15 /etc/fstab 2024-09-18T13:58:15.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:15.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_16 /etc/fstab 2024-09-18T13:58:15.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:15.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_17 /etc/fstab 2024-09-18T13:58:15.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:15.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_18 /etc/fstab 2024-09-18T13:58:16.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:16.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_19 /etc/fstab 2024-09-18T13:58:16.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:16.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_20 /etc/fstab 2024-09-18T13:58:16.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:16.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_21 /etc/fstab 2024-09-18T13:58:16.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:16.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_22 /etc/fstab 2024-09-18T13:58:16.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:16.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_23 /etc/fstab 2024-09-18T13:58:16.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:16.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_24 /etc/fstab 2024-09-18T13:58:17.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:17.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_25 /etc/fstab 2024-09-18T13:58:17.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:17.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_26 /etc/fstab 2024-09-18T13:58:17.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:17.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_27 /etc/fstab 2024-09-18T13:58:17.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:17.432 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_28 /etc/fstab 2024-09-18T13:58:17.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:17.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_29 /etc/fstab 2024-09-18T13:58:17.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:17.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_30 /etc/fstab 2024-09-18T13:58:17.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:17.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_31 /etc/fstab 2024-09-18T13:58:18.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:18.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_32 /etc/fstab 2024-09-18T13:58:18.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:18.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_33 /etc/fstab 2024-09-18T13:58:18.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:18.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_34 /etc/fstab 2024-09-18T13:58:18.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:18.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_35 /etc/fstab 2024-09-18T13:58:18.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:18.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_36 /etc/fstab 2024-09-18T13:58:18.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:18.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_37 /etc/fstab 2024-09-18T13:58:18.948 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:18.949 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_38 /etc/fstab 2024-09-18T13:58:19.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:19.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_39 /etc/fstab 2024-09-18T13:58:19.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:19.242 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_40 /etc/fstab 2024-09-18T13:58:19.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:19.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_41 /etc/fstab 2024-09-18T13:58:19.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:19.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_42 /etc/fstab 2024-09-18T13:58:19.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:19.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_43 /etc/fstab 2024-09-18T13:58:19.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:19.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_44 /etc/fstab 2024-09-18T13:58:19.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:19.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_45 /etc/fstab 2024-09-18T13:58:20.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:20.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_46 /etc/fstab 2024-09-18T13:58:20.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:20.264 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_47 /etc/fstab 2024-09-18T13:58:20.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:20.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_48 /etc/fstab 2024-09-18T13:58:20.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:20.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_49 /etc/fstab 2024-09-18T13:58:20.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:20.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_50 /etc/fstab 2024-09-18T13:58:20.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:20.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_51 /etc/fstab 2024-09-18T13:58:21.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:21.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_52 /etc/fstab 2024-09-18T13:58:21.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:21.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_53 /etc/fstab 2024-09-18T13:58:21.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:21.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_54 /etc/fstab 2024-09-18T13:58:21.470 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:21.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_55 /etc/fstab 2024-09-18T13:58:21.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:21.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_56 /etc/fstab 2024-09-18T13:58:21.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:21.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_57 /etc/fstab 2024-09-18T13:58:21.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:21.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_58 /etc/fstab 2024-09-18T13:58:22.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:22.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_59 /etc/fstab 2024-09-18T13:58:22.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:22.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_60 /etc/fstab 2024-09-18T13:58:22.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:22.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_61 /etc/fstab 2024-09-18T13:58:22.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:22.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_62 /etc/fstab 2024-09-18T13:58:22.660 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:22.660 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_63 /etc/fstab 2024-09-18T13:58:22.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:22.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_64 /etc/fstab 2024-09-18T13:58:22.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:22.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_65 /etc/fstab 2024-09-18T13:58:23.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:23.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_66 /etc/fstab 2024-09-18T13:58:23.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:23.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_67 /etc/fstab 2024-09-18T13:58:23.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:23.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_68 /etc/fstab 2024-09-18T13:58:23.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:23.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_69 /etc/fstab 2024-09-18T13:58:23.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:23.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_70 /etc/fstab 2024-09-18T13:58:23.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:23.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_71 /etc/fstab 2024-09-18T13:58:23.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:23.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_72 /etc/fstab 2024-09-18T13:58:24.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:24.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_73 /etc/fstab 2024-09-18T13:58:24.285 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:24.285 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_74 /etc/fstab 2024-09-18T13:58:24.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:24.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_75 /etc/fstab 2024-09-18T13:58:24.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:24.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_76 /etc/fstab 2024-09-18T13:58:24.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:24.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_77 /etc/fstab 2024-09-18T13:58:24.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:24.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_78 /etc/fstab 2024-09-18T13:58:25.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:25.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_79 /etc/fstab 2024-09-18T13:58:25.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:25.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_80 /etc/fstab 2024-09-18T13:58:25.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:25.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_81 /etc/fstab 2024-09-18T13:58:25.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:25.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_82 /etc/fstab 2024-09-18T13:58:25.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:25.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_83 /etc/fstab 2024-09-18T13:58:25.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:25.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_84 /etc/fstab 2024-09-18T13:58:25.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:25.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_85 /etc/fstab 2024-09-18T13:58:26.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:26.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_86 /etc/fstab 2024-09-18T13:58:26.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:26.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_87 /etc/fstab 2024-09-18T13:58:26.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:26.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_88 /etc/fstab 2024-09-18T13:58:26.514 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:26.514 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_89 /etc/fstab 2024-09-18T13:58:26.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:26.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_90 /etc/fstab 2024-09-18T13:58:26.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:26.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_91 /etc/fstab 2024-09-18T13:58:27.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:27.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_92 /etc/fstab 2024-09-18T13:58:27.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:27.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_93 /etc/fstab 2024-09-18T13:58:27.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:27.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_94 /etc/fstab 2024-09-18T13:58:27.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:27.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_95 /etc/fstab 2024-09-18T13:58:27.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:27.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_96 /etc/fstab 2024-09-18T13:58:27.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:27.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_97 /etc/fstab 2024-09-18T13:58:27.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:27.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_98 /etc/fstab 2024-09-18T13:58:28.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:28.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_99 /etc/fstab 2024-09-18T13:58:28.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-18T13:58:28.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_100 /etc/fstab 2024-09-18T13:58:28.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-18T13:58:28.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: wait_for_clean 2024-09-18T13:58:28.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T13:58:28.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T13:58:28.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T13:58:28.584 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-18T13:58:28.584 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T13:58:28.585 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T13:58:28.585 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T13:58:28.585 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T13:58:28.585 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T13:58:28.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-18T13:58:28.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T13:58:28.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T13:58:28.648 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T13:58:28.648 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T13:58:28.648 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T13:58:29.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T13:58:29.025 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T13:58:29.025 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T13:58:29.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T13:58:29.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T13:58:29.026 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T13:58:29.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836507 2024-09-18T13:58:29.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836507 2024-09-18T13:58:29.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836507' 2024-09-18T13:58:29.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T13:58:29.142 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T13:58:29.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672985 2024-09-18T13:58:29.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672985 2024-09-18T13:58:29.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836507 1-42949672985' 2024-09-18T13:58:29.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T13:58:29.259 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T13:58:29.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509453 2024-09-18T13:58:29.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509453 2024-09-18T13:58:29.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836507 1-42949672985 2-64424509453' 2024-09-18T13:58:29.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T13:58:29.376 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836507 2024-09-18T13:58:29.376 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T13:58:29.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T13:58:29.378 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836507 2024-09-18T13:58:29.378 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T13:58:29.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836507 2024-09-18T13:58:29.379 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836507 2024-09-18T13:58:29.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-09-18T13:58:29.380 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T13:58:29.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836505 -lt 21474836507 2024-09-18T13:58:29.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T13:58:30.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T13:58:30.797 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T13:58:31.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836508 -lt 21474836507 2024-09-18T13:58:31.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T13:58:31.223 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672985 2024-09-18T13:58:31.223 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T13:58:31.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T13:58:31.224 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672985 2024-09-18T13:58:31.225 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T13:58:31.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672985 2024-09-18T13:58:31.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-09-18T13:58:31.226 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672985 2024-09-18T13:58:31.226 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T13:58:31.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672985 -lt 42949672985 2024-09-18T13:58:31.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T13:58:31.641 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509453 2024-09-18T13:58:31.641 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T13:58:31.642 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T13:58:31.643 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509453 2024-09-18T13:58:31.643 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T13:58:31.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509453 2024-09-18T13:58:31.644 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509453 2024-09-18T13:58:31.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-09-18T13:58:31.644 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T13:58:32.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509453 -lt 64424509453 2024-09-18T13:58:32.050 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T13:58:32.050 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T13:58:32.051 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T13:58:32.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T13:58:32.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T13:58:32.554 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T13:58:32.554 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T13:58:32.554 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T13:58:32.555 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T13:58:32.555 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T13:58:32.555 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T13:58:32.938 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T13:58:32.939 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T13:58:32.939 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T13:58:32.939 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T13:58:33.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T13:58:33.437 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T13:58:33.437 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T13:58:33.437 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: get_pg test existing_1 2024-09-18T13:58:33.437 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-18T13:58:33.437 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-09-18T13:58:33.438 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-09-18T13:58:33.438 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T13:58:33.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: local pgid=1.0 2024-09-18T13:58:33.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:107: TEST_divergent: echo 'blackholing osds 0 2' 2024-09-18T13:58:33.880 INFO:tasks.workunit.client.0.smithi047.stdout:blackholing osds 0 2 2024-09-18T13:58:33.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:108: TEST_divergent: ceph pg dump pgs 2024-09-18T13:58:34.231 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T13:58:34.231 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 100 0 0 0 0 48300 0 0 16 84 16 active+clean 2024-09-18T13:58:06.125993+0000 27'100 27:112 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T13:58:04.104186+0000 0'0 2024-09-18T13:58:04.104186+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:36:30.459830+0000 0 0 2024-09-18T13:58:34.231 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T13:58:34.231 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T13:58:34.232 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T13:58:34.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2024-09-18T13:58:34.242 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.0 2024-09-18T13:58:34.243 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T13:58:34.243 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T13:58:34.243 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T13:58:34.243 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T13:58:34.243 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T13:58:34.243 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32609/ceph-osd.0.asok 2024-09-18T13:58:34.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2024-09-18T13:58:34.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.32609/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-18T13:58:34.317 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T13:58:34.318 INFO:tasks.workunit.client.0.smithi047.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-18T13:58:34.318 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T13:58:34.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2024-09-18T13:58:34.325 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.2 2024-09-18T13:58:34.325 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T13:58:34.325 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T13:58:34.326 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T13:58:34.326 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T13:58:34.326 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T13:58:34.326 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32609/ceph-osd.2.asok 2024-09-18T13:58:34.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2024-09-18T13:58:34.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.32609/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-18T13:58:34.400 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T13:58:34.400 INFO:tasks.workunit.client.0.smithi047.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-18T13:58:34.400 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T13:58:34.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:114: TEST_divergent: local case5=100 2024-09-18T13:58:34.407 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: expr 100 - 1 2024-09-18T13:58:34.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: local case3=99 2024-09-18T13:58:34.408 INFO:tasks.workunit.client.0.smithi047.stdout:writing divergent object 2024-09-18T13:58:34.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:117: TEST_divergent: echo 'writing divergent object' 2024-09-18T13:58:34.409 INFO:tasks.workunit.client.0.smithi047.stdout:create missing divergent object 2024-09-18T13:58:34.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:119: TEST_divergent: echo 'create missing divergent object' 2024-09-18T13:58:34.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:120: TEST_divergent: inject_eio rep data test existing_99 td/divergent-priors 0 2024-09-18T13:58:34.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:118: TEST_divergent: rados -p test put existing_100 /etc/fstab 2024-09-18T13:58:34.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T13:58:34.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T13:58:34.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T13:58:34.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T13:58:34.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=test 2024-09-18T13:58:34.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T13:58:34.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=existing_99 2024-09-18T13:58:34.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T13:58:34.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/divergent-priors 2024-09-18T13:58:34.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T13:58:34.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T13:58:34.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T13:58:34.413 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds test existing_99 2024-09-18T13:58:34.413 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test 2024-09-18T13:58:34.413 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=existing_99 2024-09-18T13:58:34.413 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test existing_99 2024-09-18T13:58:34.414 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T13:58:34.827 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-09-18T13:58:34.827 INFO:tasks.workunit.client.0.smithi047.stderr:0 2024-09-18T13:58:34.828 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T13:58:34.828 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2 2024-09-18T13:58:34.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('1' '0' '2') 2024-09-18T13:58:34.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T13:58:34.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-18T13:58:34.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T13:58:34.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T13:58:34.828 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/divergent-priors/1/type 2024-09-18T13:58:34.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T13:58:34.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-18T13:58:34.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T13:58:34.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-18T13:58:34.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T13:58:34.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T13:58:34.831 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T13:58:34.831 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-18T13:58:34.831 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T13:58:34.831 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T13:58:34.831 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T13:58:34.831 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T13:58:34.831 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T13:58:34.831 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32609/ceph-osd.1.asok 2024-09-18T13:58:34.831 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.32609/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-18T13:58:34.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T13:58:34.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T13:58:34.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T13:58:34.914 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-18T13:58:34.914 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T13:58:34.914 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T13:58:34.915 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T13:58:34.915 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T13:58:34.915 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T13:58:34.915 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32609/ceph-osd.1.asok 2024-09-18T13:58:34.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T13:58:34.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.32609/ceph-osd.1.asok injectdataerr test existing_99 2024-09-18T13:58:34.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:122: TEST_divergent: sleep 10 2024-09-18T13:58:34.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:121: TEST_divergent: rados -p test get existing_99 td/divergent-priors/existing 2024-09-18T13:58:44.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:123: TEST_divergent: killall -9 rados 2024-09-18T13:58:45.018 INFO:tasks.workunit.client.0.smithi047.stderr:rados: no process found 2024-09-18T13:58:45.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:126: TEST_divergent: echo 'killing all the osds' 2024-09-18T13:58:45.018 INFO:tasks.workunit.client.0.smithi047.stdout:killing all the osds 2024-09-18T13:58:45.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:127: TEST_divergent: ceph pg dump pgs 2024-09-18T13:58:45.386 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T13:58:45.386 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 100 0 0 0 0 48300 0 0 17 84 17 active+clean 2024-09-18T13:58:35.155289+0000 27'101 27:121 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T13:58:04.104186+0000 0'0 2024-09-18T13:58:04.104186+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:36:30.459830+0000 0 0 2024-09-18T13:58:45.386 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T13:58:45.386 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T13:58:45.386 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T13:58:45.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:128: TEST_divergent: kill_daemons td/divergent-priors KILL osd 2024-09-18T13:58:45.397 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T13:58:45.397 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T13:58:45.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T13:58:45.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T13:58:45.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T13:58:45.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T13:58:45.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-09-18T13:58:45.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.0 2024-09-18T13:58:46.017 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already down. 2024-09-18T13:58:46.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-09-18T13:58:46.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.1 2024-09-18T13:58:46.715 INFO:tasks.workunit.client.0.smithi047.stderr:osd.1 is already down. 2024-09-18T13:58:46.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-09-18T13:58:46.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.2 2024-09-18T13:58:47.265 INFO:tasks.workunit.client.0.smithi047.stderr:osd.2 is already down. 2024-09-18T13:58:47.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2024-09-18T13:58:47.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.0 2024-09-18T13:58:47.818 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already out. 2024-09-18T13:58:47.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2024-09-18T13:58:47.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.2 2024-09-18T13:58:48.383 INFO:tasks.workunit.client.0.smithi047.stderr:osd.2 is already out. 2024-09-18T13:58:48.393 INFO:tasks.workunit.client.0.smithi047.stdout:bringing up non_divergent 0 2 2024-09-18T13:58:48.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:139: TEST_divergent: echo 'bringing up non_divergent 0 2' 2024-09-18T13:58:48.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:140: TEST_divergent: ceph pg dump pgs 2024-09-18T13:58:48.782 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T13:58:48.782 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 100 0 0 0 0 48300 0 0 17 84 17 stale+active+clean 2024-09-18T13:58:35.155289+0000 27'101 27:121 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T13:58:04.104186+0000 0'0 2024-09-18T13:58:04.104186+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:36:30.459830+0000 0 0 2024-09-18T13:58:48.782 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T13:58:48.782 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T13:58:48.782 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T13:58:48.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2024-09-18T13:58:48.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 0 2024-09-18T13:58:48.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-18T13:58:48.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T13:58:48.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T13:58:48.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T13:58:48.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-09-18T13:58:48.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T13:58:48.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T13:58:48.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T13:58:48.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T13:58:48.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-18T13:58:48.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-18T13:58:48.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T13:58:48.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T13:58:48.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-18T13:58:48.796 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T13:58:48.796 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T13:58:48.796 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T13:58:48.797 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T13:58:48.797 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T13:58:48.797 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T13:58:48.797 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T13:58:48.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T13:58:48.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T13:58:48.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-18T13:58:48.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-18T13:58:48.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T13:58:48.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T13:58:48.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T13:58:48.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T13:58:48.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T13:58:48.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T13:58:48.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-09-18T13:58:48.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T13:58:48.798 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T13:58:48.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T13:58:48.799 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-09-18T13:58:48.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T13:58:48.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T13:58:48.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T13:58:48.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T13:58:48.825 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:58:48.825+0000 7f77f883f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:58:48.827 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:58:48.827+0000 7f77f883f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:58:48.829 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:58:48.828+0000 7f77f883f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:58:49.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T13:58:49.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T13:58:49.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T13:58:49.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T13:58:49.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T13:58:49.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:58:49.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T13:58:49.207 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T13:58:49.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:58:49.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:58:49.353 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:58:49.353+0000 7f77f883f440 -1 Falling back to public interface 2024-09-18T13:58:49.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:58:50.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:58:50.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:58:50.625 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T13:58:50.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T13:58:50.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:58:50.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:58:51.022 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:58:52.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:58:52.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:58:52.023 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T13:58:52.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T13:58:52.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:58:52.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:58:52.201 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:58:52.201+0000 7f77f883f440 -1 osd.0 27 log_to_monitors true 2024-09-18T13:58:52.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:58:53.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:58:53.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:58:53.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T13:58:53.449 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T13:58:53.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:58:53.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:58:53.874 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up out weight 0 up_from 34 up_thru 0 down_at 28 last_clean_interval [5,27) [v2:127.0.0.1:6802/41358,v1:127.0.0.1:6803/41358] [v2:127.0.0.1:6804/41358,v1:127.0.0.1:6805/41358] exists,up 120d8a51-9d44-40cc-b688-4581426ff01d 2024-09-18T13:58:53.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T13:58:53.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T13:58:53.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T13:58:53.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2024-09-18T13:58:53.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 2 2024-09-18T13:58:53.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-18T13:58:53.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T13:58:53.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T13:58:53.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T13:58:53.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-09-18T13:58:53.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T13:58:53.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T13:58:53.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T13:58:53.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T13:58:53.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-18T13:58:53.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-18T13:58:53.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T13:58:53.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T13:58:53.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-18T13:58:53.879 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T13:58:53.879 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T13:58:53.879 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T13:58:53.879 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T13:58:53.879 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T13:58:53.879 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T13:58:53.879 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T13:58:53.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T13:58:53.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T13:58:53.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-18T13:58:53.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-18T13:58:53.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T13:58:53.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T13:58:53.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T13:58:53.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T13:58:53.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T13:58:53.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T13:58:53.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-09-18T13:58:53.881 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T13:58:53.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T13:58:53.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T13:58:53.882 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-09-18T13:58:53.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T13:58:53.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T13:58:53.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T13:58:53.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T13:58:53.907 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:58:53.907+0000 7f0c60327440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:58:53.909 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:58:53.909+0000 7f0c60327440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:58:53.910 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:58:53.910+0000 7f0c60327440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:58:54.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T13:58:54.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T13:58:54.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T13:58:54.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T13:58:54.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T13:58:54.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:58:54.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T13:58:54.257 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T13:58:54.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:58:54.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:58:54.437 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:58:54.437+0000 7f0c60327440 -1 Falling back to public interface 2024-09-18T13:58:54.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:58:55.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:58:55.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:58:55.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T13:58:55.671 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T13:58:55.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:58:55.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:58:56.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:58:57.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:58:57.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:58:57.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T13:58:57.055 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T13:58:57.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:58:57.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:58:57.338 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:58:57.338+0000 7f0c60327440 -1 osd.2 27 log_to_monitors true 2024-09-18T13:58:57.437 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T13:58:58.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T13:58:58.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:58:58.439 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T13:58:58.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T13:58:58.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:58:58.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:58:58.827 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [15,27) [v2:127.0.0.1:6810/41936,v1:127.0.0.1:6811/41936] [v2:127.0.0.1:6812/41936,v1:127.0.0.1:6813/41936] exists,up d1a5f547-90ee-411b-9aab-d214098efc62 2024-09-18T13:58:58.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T13:58:58.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T13:58:58.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T13:58:58.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2024-09-18T13:58:58.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.0 2024-09-18T13:58:59.430 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already in. 2024-09-18T13:58:59.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2024-09-18T13:58:59.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.2 2024-09-18T13:59:00.346 INFO:tasks.workunit.client.0.smithi047.stderr:osd.2 is already in. 2024-09-18T13:59:00.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-18T13:59:00.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: wait_for_clean 2024-09-18T13:59:00.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T13:59:00.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T13:59:00.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T13:59:00.359 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-18T13:59:00.359 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T13:59:00.359 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T13:59:00.359 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T13:59:00.359 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T13:59:00.359 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T13:59:00.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-18T13:59:00.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T13:59:00.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T13:59:00.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T13:59:00.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T13:59:00.414 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T13:59:00.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T13:59:00.833 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T13:59:00.834 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T13:59:00.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T13:59:00.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T13:59:00.834 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T13:59:00.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=146028888069 2024-09-18T13:59:00.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 146028888069 2024-09-18T13:59:00.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-146028888069' 2024-09-18T13:59:00.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T13:59:00.951 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T13:59:01.034 INFO:tasks.workunit.client.0.smithi047.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-18T13:59:01.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq= 2024-09-18T13:59:01.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z '' 2024-09-18T13:59:01.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: flush_pg_stats: continue 2024-09-18T13:59:01.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T13:59:01.037 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T13:59:01.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=154618822659 2024-09-18T13:59:01.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 154618822659 2024-09-18T13:59:01.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-146028888069 2-154618822659' 2024-09-18T13:59:01.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T13:59:01.152 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-146028888069 2024-09-18T13:59:01.152 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T13:59:01.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T13:59:01.153 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-146028888069 2024-09-18T13:59:01.153 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T13:59:01.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=146028888069 2024-09-18T13:59:01.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 146028888069' 2024-09-18T13:59:01.155 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 146028888069 2024-09-18T13:59:01.155 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T13:59:01.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 146028888067 -lt 146028888069 2024-09-18T13:59:01.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T13:59:02.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T13:59:02.558 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T13:59:02.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 146028888069 -lt 146028888069 2024-09-18T13:59:02.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T13:59:02.946 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-154618822659 2024-09-18T13:59:02.946 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T13:59:02.946 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T13:59:02.947 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-154618822659 2024-09-18T13:59:02.947 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T13:59:02.948 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=154618822659 2024-09-18T13:59:02.948 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 154618822659 2024-09-18T13:59:02.949 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 154618822659' 2024-09-18T13:59:02.949 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T13:59:03.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 154618822659 -lt 154618822659 2024-09-18T13:59:03.368 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T13:59:03.368 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T13:59:03.368 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T13:59:03.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T13:59:03.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T13:59:03.880 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T13:59:03.880 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T13:59:03.881 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T13:59:03.881 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T13:59:03.881 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T13:59:03.881 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T13:59:04.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T13:59:04.279 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T13:59:04.279 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T13:59:04.279 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T13:59:04.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T13:59:04.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-18T13:59:04.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T13:59:04.780 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-18T13:59:04.780 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T13:59:04.780 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T13:59:04.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T13:59:04.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T13:59:04.881 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T13:59:04.881 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T13:59:04.881 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T13:59:04.881 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T13:59:04.881 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T13:59:04.881 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T13:59:05.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T13:59:05.250 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T13:59:05.250 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T13:59:05.250 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T13:59:05.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T13:59:05.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T13:59:05.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T13:59:05.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T13:59:05.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T13:59:05.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T13:59:05.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T13:59:05.756 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T13:59:05.756 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T13:59:06.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T13:59:06.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T13:59:06.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-09-18T13:59:06.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T13:59:06.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-18T13:59:06.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T13:59:06.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T13:59:06.480 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T13:59:06.480 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T13:59:06.480 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T13:59:06.480 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T13:59:06.480 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T13:59:06.480 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T13:59:06.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T13:59:06.845 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T13:59:06.845 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T13:59:06.845 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T13:59:07.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T13:59:07.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T13:59:07.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T13:59:07.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T13:59:07.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T13:59:07.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T13:59:07.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T13:59:07.346 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T13:59:07.346 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T13:59:07.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T13:59:07.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T13:59:07.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-09-18T13:59:07.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T13:59:07.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-18T13:59:08.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T13:59:08.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T13:59:08.321 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T13:59:08.321 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T13:59:08.321 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T13:59:08.321 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T13:59:08.321 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T13:59:08.322 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T13:59:08.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T13:59:08.691 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T13:59:08.692 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T13:59:08.692 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T13:59:09.197 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T13:59:09.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T13:59:09.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T13:59:09.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T13:59:09.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T13:59:09.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T13:59:09.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T13:59:09.199 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T13:59:09.199 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T13:59:09.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T13:59:09.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T13:59:09.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-09-18T13:59:09.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T13:59:09.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-09-18T13:59:10.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T13:59:10.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T13:59:10.527 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T13:59:10.527 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T13:59:10.527 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T13:59:10.527 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T13:59:10.527 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T13:59:10.527 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T13:59:10.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T13:59:10.899 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T13:59:10.899 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T13:59:10.900 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T13:59:11.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T13:59:11.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T13:59:11.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T13:59:11.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T13:59:11.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T13:59:11.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T13:59:11.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T13:59:11.403 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T13:59:11.403 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T13:59:11.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T13:59:11.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T13:59:11.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-09-18T13:59:11.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T13:59:11.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-09-18T13:59:13.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T13:59:13.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T13:59:13.556 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T13:59:13.556 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T13:59:13.556 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T13:59:13.556 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T13:59:13.556 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T13:59:13.556 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T13:59:13.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T13:59:13.935 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T13:59:13.936 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T13:59:13.936 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T13:59:14.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T13:59:14.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T13:59:14.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T13:59:14.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T13:59:14.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T13:59:14.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T13:59:14.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T13:59:14.444 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T13:59:14.444 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T13:59:15.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T13:59:15.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T13:59:15.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-09-18T13:59:15.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T13:59:15.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-09-18T13:59:18.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T13:59:18.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T13:59:18.210 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T13:59:18.210 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T13:59:18.210 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T13:59:18.210 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T13:59:18.210 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T13:59:18.210 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T13:59:18.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T13:59:18.605 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T13:59:18.605 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T13:59:18.605 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T13:59:19.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T13:59:19.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T13:59:19.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T13:59:19.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T13:59:19.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T13:59:19.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T13:59:19.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T13:59:19.149 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T13:59:19.149 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T13:59:19.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T13:59:19.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T13:59:19.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-09-18T13:59:19.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T13:59:19.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-09-18T13:59:26.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T13:59:26.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T13:59:26.084 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T13:59:26.084 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T13:59:26.084 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T13:59:26.084 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T13:59:26.084 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T13:59:26.084 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T13:59:26.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T13:59:26.462 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T13:59:26.462 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T13:59:26.462 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T13:59:26.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T13:59:26.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T13:59:26.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T13:59:26.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T13:59:26.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T13:59:26.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T13:59:26.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T13:59:26.964 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T13:59:26.964 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T13:59:27.483 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T13:59:27.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T13:59:27.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-09-18T13:59:27.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T13:59:27.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-09-18T13:59:34.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T13:59:34.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T13:59:34.786 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T13:59:34.786 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T13:59:34.786 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T13:59:34.786 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T13:59:34.786 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T13:59:34.786 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T13:59:35.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T13:59:35.165 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T13:59:35.165 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T13:59:35.165 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T13:59:35.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T13:59:35.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T13:59:35.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T13:59:35.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T13:59:35.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T13:59:35.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T13:59:35.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T13:59:35.671 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T13:59:35.671 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T13:59:36.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T13:59:36.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T13:59:36.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-09-18T13:59:36.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-18T13:59:36.728 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T13:59:36.728 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_fingerprint": "eb94b41b-7788-405f-9fcb-aaadd428f183", 2024-09-18T13:59:36.728 INFO:tasks.workunit.client.0.smithi047.stdout: "version": "17.2.7-1645-g7061ea6c", 2024-09-18T13:59:36.728 INFO:tasks.workunit.client.0.smithi047.stdout: "commit": "7061ea6ca50908d104e9f3979dc615e118d9e153", 2024-09-18T13:59:36.728 INFO:tasks.workunit.client.0.smithi047.stdout: "timestamp": "2024-09-18T13:59:36.727405+0000", 2024-09-18T13:59:36.728 INFO:tasks.workunit.client.0.smithi047.stdout: "tag": "", 2024-09-18T13:59:36.729 INFO:tasks.workunit.client.0.smithi047.stdout: "health": { 2024-09-18T13:59:36.729 INFO:tasks.workunit.client.0.smithi047.stdout: "status": "HEALTH_WARN", 2024-09-18T13:59:36.729 INFO:tasks.workunit.client.0.smithi047.stdout: "checks": { 2024-09-18T13:59:36.729 INFO:tasks.workunit.client.0.smithi047.stdout: "OSDMAP_FLAGS": { 2024-09-18T13:59:36.729 INFO:tasks.workunit.client.0.smithi047.stdout: "severity": "HEALTH_WARN", 2024-09-18T13:59:36.729 INFO:tasks.workunit.client.0.smithi047.stdout: "summary": { 2024-09-18T13:59:36.729 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-18T13:59:36.729 INFO:tasks.workunit.client.0.smithi047.stdout: "count": 17 2024-09-18T13:59:36.729 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.729 INFO:tasks.workunit.client.0.smithi047.stdout: "detail": [], 2024-09-18T13:59:36.729 INFO:tasks.workunit.client.0.smithi047.stdout: "muted": false 2024-09-18T13:59:36.729 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.729 INFO:tasks.workunit.client.0.smithi047.stdout: "OSD_DOWN": { 2024-09-18T13:59:36.730 INFO:tasks.workunit.client.0.smithi047.stdout: "severity": "HEALTH_WARN", 2024-09-18T13:59:36.730 INFO:tasks.workunit.client.0.smithi047.stdout: "summary": { 2024-09-18T13:59:36.730 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "1 osds down", 2024-09-18T13:59:36.730 INFO:tasks.workunit.client.0.smithi047.stdout: "count": 1 2024-09-18T13:59:36.730 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.730 INFO:tasks.workunit.client.0.smithi047.stdout: "detail": [ 2024-09-18T13:59:36.730 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.730 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "osd.1 (root=default,host=smithi047) is down" 2024-09-18T13:59:36.730 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.730 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T13:59:36.730 INFO:tasks.workunit.client.0.smithi047.stdout: "muted": false 2024-09-18T13:59:36.730 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.730 INFO:tasks.workunit.client.0.smithi047.stdout: "PG_DEGRADED": { 2024-09-18T13:59:36.730 INFO:tasks.workunit.client.0.smithi047.stdout: "severity": "HEALTH_WARN", 2024-09-18T13:59:36.731 INFO:tasks.workunit.client.0.smithi047.stdout: "summary": { 2024-09-18T13:59:36.731 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2024-09-18T13:59:36.731 INFO:tasks.workunit.client.0.smithi047.stdout: "count": 1 2024-09-18T13:59:36.731 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.731 INFO:tasks.workunit.client.0.smithi047.stdout: "detail": [ 2024-09-18T13:59:36.731 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.731 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-09-18T13:59:36.731 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.731 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T13:59:36.731 INFO:tasks.workunit.client.0.smithi047.stdout: "muted": false 2024-09-18T13:59:36.731 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.731 INFO:tasks.workunit.client.0.smithi047.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-18T13:59:36.731 INFO:tasks.workunit.client.0.smithi047.stdout: "severity": "HEALTH_WARN", 2024-09-18T13:59:36.732 INFO:tasks.workunit.client.0.smithi047.stdout: "summary": { 2024-09-18T13:59:36.732 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-18T13:59:36.732 INFO:tasks.workunit.client.0.smithi047.stdout: "count": 1 2024-09-18T13:59:36.732 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.732 INFO:tasks.workunit.client.0.smithi047.stdout: "detail": [ 2024-09-18T13:59:36.732 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.732 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "application not enabled on pool 'test'" 2024-09-18T13:59:36.732 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.732 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.732 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-18T13:59:36.732 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.732 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T13:59:36.732 INFO:tasks.workunit.client.0.smithi047.stdout: "muted": false 2024-09-18T13:59:36.733 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.733 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.733 INFO:tasks.workunit.client.0.smithi047.stdout: "mutes": [] 2024-09-18T13:59:36.733 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.733 INFO:tasks.workunit.client.0.smithi047.stdout: "monmap_first_committed": 1, 2024-09-18T13:59:36.733 INFO:tasks.workunit.client.0.smithi047.stdout: "monmap_last_committed": 1, 2024-09-18T13:59:36.733 INFO:tasks.workunit.client.0.smithi047.stdout: "monmap": { 2024-09-18T13:59:36.733 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 1, 2024-09-18T13:59:36.733 INFO:tasks.workunit.client.0.smithi047.stdout: "fsid": "d7330bd7-8062-424a-bbd8-8c587903c456", 2024-09-18T13:59:36.733 INFO:tasks.workunit.client.0.smithi047.stdout: "modified": "2024-09-18T13:56:20.923696Z", 2024-09-18T13:59:36.733 INFO:tasks.workunit.client.0.smithi047.stdout: "created": "2024-09-18T13:56:20.923696Z", 2024-09-18T13:59:36.733 INFO:tasks.workunit.client.0.smithi047.stdout: "min_mon_release": 17, 2024-09-18T13:59:36.733 INFO:tasks.workunit.client.0.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-18T13:59:36.734 INFO:tasks.workunit.client.0.smithi047.stdout: "election_strategy": 1, 2024-09-18T13:59:36.734 INFO:tasks.workunit.client.0.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-18T13:59:36.734 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_mode": false, 2024-09-18T13:59:36.734 INFO:tasks.workunit.client.0.smithi047.stdout: "tiebreaker_mon": "", 2024-09-18T13:59:36.734 INFO:tasks.workunit.client.0.smithi047.stdout: "removed_ranks: ": "", 2024-09-18T13:59:36.734 INFO:tasks.workunit.client.0.smithi047.stdout: "features": { 2024-09-18T13:59:36.734 INFO:tasks.workunit.client.0.smithi047.stdout: "persistent": [ 2024-09-18T13:59:36.734 INFO:tasks.workunit.client.0.smithi047.stdout: "kraken", 2024-09-18T13:59:36.734 INFO:tasks.workunit.client.0.smithi047.stdout: "luminous", 2024-09-18T13:59:36.734 INFO:tasks.workunit.client.0.smithi047.stdout: "mimic", 2024-09-18T13:59:36.734 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap-prune", 2024-09-18T13:59:36.734 INFO:tasks.workunit.client.0.smithi047.stdout: "nautilus", 2024-09-18T13:59:36.734 INFO:tasks.workunit.client.0.smithi047.stdout: "octopus", 2024-09-18T13:59:36.734 INFO:tasks.workunit.client.0.smithi047.stdout: "pacific", 2024-09-18T13:59:36.734 INFO:tasks.workunit.client.0.smithi047.stdout: "elector-pinging", 2024-09-18T13:59:36.735 INFO:tasks.workunit.client.0.smithi047.stdout: "quincy" 2024-09-18T13:59:36.735 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T13:59:36.735 INFO:tasks.workunit.client.0.smithi047.stdout: "optional": [] 2024-09-18T13:59:36.735 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.735 INFO:tasks.workunit.client.0.smithi047.stdout: "mons": [ 2024-09-18T13:59:36.735 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.735 INFO:tasks.workunit.client.0.smithi047.stdout: "rank": 0, 2024-09-18T13:59:36.735 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "a", 2024-09-18T13:59:36.735 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addrs": { 2024-09-18T13:59:36.735 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T13:59:36.735 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.735 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T13:59:36.735 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:7115", 2024-09-18T13:59:36.735 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 0 2024-09-18T13:59:36.736 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.736 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T13:59:36.736 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.736 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:7115/0", 2024-09-18T13:59:36.736 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-18T13:59:36.736 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 0, 2024-09-18T13:59:36.736 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 0, 2024-09-18T13:59:36.736 INFO:tasks.workunit.client.0.smithi047.stdout: "crush_location": "{}" 2024-09-18T13:59:36.737 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.737 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T13:59:36.737 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.737 INFO:tasks.workunit.client.0.smithi047.stdout: "quorum": [ 2024-09-18T13:59:36.737 INFO:tasks.workunit.client.0.smithi047.stdout: 0 2024-09-18T13:59:36.737 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T13:59:36.737 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap": { 2024-09-18T13:59:36.737 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 40, 2024-09-18T13:59:36.737 INFO:tasks.workunit.client.0.smithi047.stdout: "fsid": "d7330bd7-8062-424a-bbd8-8c587903c456", 2024-09-18T13:59:36.737 INFO:tasks.workunit.client.0.smithi047.stdout: "created": "2024-09-18T13:56:21.753300+0000", 2024-09-18T13:59:36.737 INFO:tasks.workunit.client.0.smithi047.stdout: "modified": "2024-09-18T13:59:00.173128+0000", 2024-09-18T13:59:36.737 INFO:tasks.workunit.client.0.smithi047.stdout: "last_up_change": "2024-09-18T13:58:57.622562+0000", 2024-09-18T13:59:36.737 INFO:tasks.workunit.client.0.smithi047.stdout: "last_in_change": "2024-09-18T13:58:59.888327+0000", 2024-09-18T13:59:36.737 INFO:tasks.workunit.client.0.smithi047.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-18T13:59:36.738 INFO:tasks.workunit.client.0.smithi047.stdout: "flags_num": 5800384, 2024-09-18T13:59:36.738 INFO:tasks.workunit.client.0.smithi047.stdout: "flags_set": [ 2024-09-18T13:59:36.738 INFO:tasks.workunit.client.0.smithi047.stdout: "nodown", 2024-09-18T13:59:36.738 INFO:tasks.workunit.client.0.smithi047.stdout: "noin", 2024-09-18T13:59:36.738 INFO:tasks.workunit.client.0.smithi047.stdout: "noout", 2024-09-18T13:59:36.738 INFO:tasks.workunit.client.0.smithi047.stdout: "pglog_hardlimit", 2024-09-18T13:59:36.738 INFO:tasks.workunit.client.0.smithi047.stdout: "purged_snapdirs", 2024-09-18T13:59:36.738 INFO:tasks.workunit.client.0.smithi047.stdout: "recovery_deletes", 2024-09-18T13:59:36.738 INFO:tasks.workunit.client.0.smithi047.stdout: "sortbitwise" 2024-09-18T13:59:36.738 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T13:59:36.738 INFO:tasks.workunit.client.0.smithi047.stdout: "crush_version": 9, 2024-09-18T13:59:36.738 INFO:tasks.workunit.client.0.smithi047.stdout: "full_ratio": 0.99000000953674316, 2024-09-18T13:59:36.738 INFO:tasks.workunit.client.0.smithi047.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-18T13:59:36.738 INFO:tasks.workunit.client.0.smithi047.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-18T13:59:36.739 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_snapshot": "", 2024-09-18T13:59:36.739 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_max": 1, 2024-09-18T13:59:36.739 INFO:tasks.workunit.client.0.smithi047.stdout: "max_osd": 3, 2024-09-18T13:59:36.739 INFO:tasks.workunit.client.0.smithi047.stdout: "require_min_compat_client": "luminous", 2024-09-18T13:59:36.739 INFO:tasks.workunit.client.0.smithi047.stdout: "min_compat_client": "jewel", 2024-09-18T13:59:36.739 INFO:tasks.workunit.client.0.smithi047.stdout: "require_osd_release": "quincy", 2024-09-18T13:59:36.739 INFO:tasks.workunit.client.0.smithi047.stdout: "pools": [ 2024-09-18T13:59:36.739 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.739 INFO:tasks.workunit.client.0.smithi047.stdout: "pool": 1, 2024-09-18T13:59:36.739 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_name": "test", 2024-09-18T13:59:36.739 INFO:tasks.workunit.client.0.smithi047.stdout: "create_time": "2024-09-18T13:58:04.054535+0000", 2024-09-18T13:59:36.739 INFO:tasks.workunit.client.0.smithi047.stdout: "flags": 1, 2024-09-18T13:59:36.739 INFO:tasks.workunit.client.0.smithi047.stdout: "flags_names": "hashpspool", 2024-09-18T13:59:36.739 INFO:tasks.workunit.client.0.smithi047.stdout: "type": 1, 2024-09-18T13:59:36.740 INFO:tasks.workunit.client.0.smithi047.stdout: "size": 3, 2024-09-18T13:59:36.740 INFO:tasks.workunit.client.0.smithi047.stdout: "min_size": 2, 2024-09-18T13:59:36.740 INFO:tasks.workunit.client.0.smithi047.stdout: "crush_rule": 0, 2024-09-18T13:59:36.740 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_count": 0, 2024-09-18T13:59:36.740 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_target": 0, 2024-09-18T13:59:36.740 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T13:59:36.740 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T13:59:36.740 INFO:tasks.workunit.client.0.smithi047.stdout: "object_hash": 2, 2024-09-18T13:59:36.740 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_autoscale_mode": "off", 2024-09-18T13:59:36.740 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_num": 1, 2024-09-18T13:59:36.740 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_placement_num": 1, 2024-09-18T13:59:36.740 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_placement_num_target": 1, 2024-09-18T13:59:36.740 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_num_target": 1, 2024-09-18T13:59:36.740 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_num_pending": 1, 2024-09-18T13:59:36.741 INFO:tasks.workunit.client.0.smithi047.stdout: "last_pg_merge_meta": { 2024-09-18T13:59:36.741 INFO:tasks.workunit.client.0.smithi047.stdout: "source_pgid": "0.0", 2024-09-18T13:59:36.741 INFO:tasks.workunit.client.0.smithi047.stdout: "ready_epoch": 0, 2024-09-18T13:59:36.741 INFO:tasks.workunit.client.0.smithi047.stdout: "last_epoch_started": 0, 2024-09-18T13:59:36.741 INFO:tasks.workunit.client.0.smithi047.stdout: "last_epoch_clean": 0, 2024-09-18T13:59:36.741 INFO:tasks.workunit.client.0.smithi047.stdout: "source_version": "0'0", 2024-09-18T13:59:36.741 INFO:tasks.workunit.client.0.smithi047.stdout: "target_version": "0'0" 2024-09-18T13:59:36.741 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.741 INFO:tasks.workunit.client.0.smithi047.stdout: "last_change": "27", 2024-09-18T13:59:36.741 INFO:tasks.workunit.client.0.smithi047.stdout: "last_force_op_resend": "0", 2024-09-18T13:59:36.741 INFO:tasks.workunit.client.0.smithi047.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T13:59:36.741 INFO:tasks.workunit.client.0.smithi047.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T13:59:36.741 INFO:tasks.workunit.client.0.smithi047.stdout: "auid": 0, 2024-09-18T13:59:36.742 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_mode": "selfmanaged", 2024-09-18T13:59:36.742 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_seq": 0, 2024-09-18T13:59:36.742 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_epoch": 0, 2024-09-18T13:59:36.742 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_snaps": [], 2024-09-18T13:59:36.742 INFO:tasks.workunit.client.0.smithi047.stdout: "removed_snaps": "[]", 2024-09-18T13:59:36.742 INFO:tasks.workunit.client.0.smithi047.stdout: "quota_max_bytes": 0, 2024-09-18T13:59:36.742 INFO:tasks.workunit.client.0.smithi047.stdout: "quota_max_objects": 0, 2024-09-18T13:59:36.742 INFO:tasks.workunit.client.0.smithi047.stdout: "tiers": [], 2024-09-18T13:59:36.742 INFO:tasks.workunit.client.0.smithi047.stdout: "tier_of": -1, 2024-09-18T13:59:36.742 INFO:tasks.workunit.client.0.smithi047.stdout: "read_tier": -1, 2024-09-18T13:59:36.742 INFO:tasks.workunit.client.0.smithi047.stdout: "write_tier": -1, 2024-09-18T13:59:36.742 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_mode": "none", 2024-09-18T13:59:36.742 INFO:tasks.workunit.client.0.smithi047.stdout: "target_max_bytes": 0, 2024-09-18T13:59:36.742 INFO:tasks.workunit.client.0.smithi047.stdout: "target_max_objects": 0, 2024-09-18T13:59:36.743 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T13:59:36.743 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T13:59:36.743 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T13:59:36.743 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_min_flush_age": 0, 2024-09-18T13:59:36.743 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_min_evict_age": 0, 2024-09-18T13:59:36.743 INFO:tasks.workunit.client.0.smithi047.stdout: "erasure_code_profile": "", 2024-09-18T13:59:36.743 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_params": { 2024-09-18T13:59:36.743 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "none" 2024-09-18T13:59:36.743 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.743 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_period": 0, 2024-09-18T13:59:36.743 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_count": 0, 2024-09-18T13:59:36.743 INFO:tasks.workunit.client.0.smithi047.stdout: "use_gmt_hitset": true, 2024-09-18T13:59:36.743 INFO:tasks.workunit.client.0.smithi047.stdout: "min_read_recency_for_promote": 0, 2024-09-18T13:59:36.743 INFO:tasks.workunit.client.0.smithi047.stdout: "min_write_recency_for_promote": 0, 2024-09-18T13:59:36.743 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T13:59:36.744 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_search_last_n": 0, 2024-09-18T13:59:36.744 INFO:tasks.workunit.client.0.smithi047.stdout: "grade_table": [], 2024-09-18T13:59:36.744 INFO:tasks.workunit.client.0.smithi047.stdout: "stripe_width": 0, 2024-09-18T13:59:36.744 INFO:tasks.workunit.client.0.smithi047.stdout: "expected_num_objects": 0, 2024-09-18T13:59:36.744 INFO:tasks.workunit.client.0.smithi047.stdout: "fast_read": false, 2024-09-18T13:59:36.744 INFO:tasks.workunit.client.0.smithi047.stdout: "options": {}, 2024-09-18T13:59:36.744 INFO:tasks.workunit.client.0.smithi047.stdout: "application_metadata": {} 2024-09-18T13:59:36.744 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.744 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T13:59:36.744 INFO:tasks.workunit.client.0.smithi047.stdout: "osds": [ 2024-09-18T13:59:36.744 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.744 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 0, 2024-09-18T13:59:36.744 INFO:tasks.workunit.client.0.smithi047.stdout: "uuid": "120d8a51-9d44-40cc-b688-4581426ff01d", 2024-09-18T13:59:36.744 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 1, 2024-09-18T13:59:36.744 INFO:tasks.workunit.client.0.smithi047.stdout: "in": 1, 2024-09-18T13:59:36.745 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 1, 2024-09-18T13:59:36.745 INFO:tasks.workunit.client.0.smithi047.stdout: "primary_affinity": 1, 2024-09-18T13:59:36.745 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_begin": 5, 2024-09-18T13:59:36.745 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_end": 27, 2024-09-18T13:59:36.745 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 34, 2024-09-18T13:59:36.745 INFO:tasks.workunit.client.0.smithi047.stdout: "up_thru": 39, 2024-09-18T13:59:36.745 INFO:tasks.workunit.client.0.smithi047.stdout: "down_at": 28, 2024-09-18T13:59:36.745 INFO:tasks.workunit.client.0.smithi047.stdout: "lost_at": 0, 2024-09-18T13:59:36.745 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addrs": { 2024-09-18T13:59:36.745 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T13:59:36.745 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.745 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T13:59:36.745 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6802", 2024-09-18T13:59:36.745 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 41358 2024-09-18T13:59:36.745 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.746 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.746 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T13:59:36.746 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6803", 2024-09-18T13:59:36.746 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 41358 2024-09-18T13:59:36.746 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.746 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T13:59:36.746 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.746 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addrs": { 2024-09-18T13:59:36.746 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T13:59:36.746 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.746 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T13:59:36.746 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6804", 2024-09-18T13:59:36.746 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 41358 2024-09-18T13:59:36.746 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.746 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.747 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T13:59:36.747 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6805", 2024-09-18T13:59:36.747 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 41358 2024-09-18T13:59:36.747 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.747 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T13:59:36.747 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.747 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addrs": { 2024-09-18T13:59:36.747 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T13:59:36.747 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.747 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T13:59:36.747 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6808", 2024-09-18T13:59:36.747 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 41358 2024-09-18T13:59:36.747 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.747 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.747 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T13:59:36.748 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6809", 2024-09-18T13:59:36.748 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 41358 2024-09-18T13:59:36.748 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.748 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T13:59:36.748 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.748 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addrs": { 2024-09-18T13:59:36.748 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T13:59:36.748 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.748 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T13:59:36.748 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6806", 2024-09-18T13:59:36.748 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 41358 2024-09-18T13:59:36.748 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.749 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.749 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T13:59:36.749 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6807", 2024-09-18T13:59:36.749 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 41358 2024-09-18T13:59:36.749 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.749 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T13:59:36.749 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.749 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addr": "127.0.0.1:6803/41358", 2024-09-18T13:59:36.749 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addr": "127.0.0.1:6805/41358", 2024-09-18T13:59:36.749 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addr": "127.0.0.1:6809/41358", 2024-09-18T13:59:36.749 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addr": "127.0.0.1:6807/41358", 2024-09-18T13:59:36.749 INFO:tasks.workunit.client.0.smithi047.stdout: "state": [ 2024-09-18T13:59:36.749 INFO:tasks.workunit.client.0.smithi047.stdout: "exists", 2024-09-18T13:59:36.749 INFO:tasks.workunit.client.0.smithi047.stdout: "up" 2024-09-18T13:59:36.749 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T13:59:36.750 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.750 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.750 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 1, 2024-09-18T13:59:36.750 INFO:tasks.workunit.client.0.smithi047.stdout: "uuid": "3353f33a-e483-4b8b-976d-f7b4acd36b36", 2024-09-18T13:59:36.750 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 0, 2024-09-18T13:59:36.750 INFO:tasks.workunit.client.0.smithi047.stdout: "in": 1, 2024-09-18T13:59:36.750 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 1, 2024-09-18T13:59:36.750 INFO:tasks.workunit.client.0.smithi047.stdout: "primary_affinity": 1, 2024-09-18T13:59:36.750 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_begin": 0, 2024-09-18T13:59:36.750 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_end": 0, 2024-09-18T13:59:36.750 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 10, 2024-09-18T13:59:36.750 INFO:tasks.workunit.client.0.smithi047.stdout: "up_thru": 22, 2024-09-18T13:59:36.750 INFO:tasks.workunit.client.0.smithi047.stdout: "down_at": 29, 2024-09-18T13:59:36.750 INFO:tasks.workunit.client.0.smithi047.stdout: "lost_at": 0, 2024-09-18T13:59:36.750 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addrs": { 2024-09-18T13:59:36.750 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T13:59:36.751 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.751 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T13:59:36.751 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6810", 2024-09-18T13:59:36.751 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 35448 2024-09-18T13:59:36.751 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.751 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.751 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T13:59:36.751 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6811", 2024-09-18T13:59:36.751 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 35448 2024-09-18T13:59:36.751 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.751 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T13:59:36.751 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.751 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addrs": { 2024-09-18T13:59:36.751 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T13:59:36.751 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.752 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T13:59:36.752 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6812", 2024-09-18T13:59:36.752 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 35448 2024-09-18T13:59:36.752 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.752 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.752 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T13:59:36.752 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6813", 2024-09-18T13:59:36.752 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 35448 2024-09-18T13:59:36.752 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.752 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T13:59:36.752 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.752 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addrs": { 2024-09-18T13:59:36.752 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T13:59:36.752 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.753 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T13:59:36.753 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6816", 2024-09-18T13:59:36.753 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 35448 2024-09-18T13:59:36.753 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.753 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.753 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T13:59:36.753 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6817", 2024-09-18T13:59:36.753 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 35448 2024-09-18T13:59:36.753 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.753 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T13:59:36.753 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.753 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addrs": { 2024-09-18T13:59:36.753 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T13:59:36.753 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.753 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T13:59:36.754 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6814", 2024-09-18T13:59:36.754 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 35448 2024-09-18T13:59:36.754 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.754 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.754 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T13:59:36.754 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6815", 2024-09-18T13:59:36.754 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 35448 2024-09-18T13:59:36.754 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.754 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T13:59:36.754 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.754 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addr": "127.0.0.1:6811/35448", 2024-09-18T13:59:36.754 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addr": "127.0.0.1:6813/35448", 2024-09-18T13:59:36.754 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addr": "127.0.0.1:6817/35448", 2024-09-18T13:59:36.754 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addr": "127.0.0.1:6815/35448", 2024-09-18T13:59:36.754 INFO:tasks.workunit.client.0.smithi047.stdout: "state": [ 2024-09-18T13:59:36.755 INFO:tasks.workunit.client.0.smithi047.stdout: "exists" 2024-09-18T13:59:36.755 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T13:59:36.755 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.755 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.755 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 2, 2024-09-18T13:59:36.755 INFO:tasks.workunit.client.0.smithi047.stdout: "uuid": "d1a5f547-90ee-411b-9aab-d214098efc62", 2024-09-18T13:59:36.755 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 1, 2024-09-18T13:59:36.755 INFO:tasks.workunit.client.0.smithi047.stdout: "in": 1, 2024-09-18T13:59:36.755 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 1, 2024-09-18T13:59:36.755 INFO:tasks.workunit.client.0.smithi047.stdout: "primary_affinity": 1, 2024-09-18T13:59:36.755 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_begin": 15, 2024-09-18T13:59:36.755 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_end": 27, 2024-09-18T13:59:36.755 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 36, 2024-09-18T13:59:36.755 INFO:tasks.workunit.client.0.smithi047.stdout: "up_thru": 0, 2024-09-18T13:59:36.755 INFO:tasks.workunit.client.0.smithi047.stdout: "down_at": 30, 2024-09-18T13:59:36.756 INFO:tasks.workunit.client.0.smithi047.stdout: "lost_at": 0, 2024-09-18T13:59:36.756 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addrs": { 2024-09-18T13:59:36.756 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T13:59:36.756 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.756 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T13:59:36.756 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6810", 2024-09-18T13:59:36.756 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 41936 2024-09-18T13:59:36.756 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.756 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.756 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T13:59:36.756 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6811", 2024-09-18T13:59:36.756 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 41936 2024-09-18T13:59:36.756 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.756 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T13:59:36.756 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.757 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addrs": { 2024-09-18T13:59:36.757 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T13:59:36.757 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.757 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T13:59:36.757 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6812", 2024-09-18T13:59:36.757 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 41936 2024-09-18T13:59:36.757 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.757 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.757 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T13:59:36.757 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6813", 2024-09-18T13:59:36.757 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 41936 2024-09-18T13:59:36.757 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.757 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T13:59:36.757 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.758 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addrs": { 2024-09-18T13:59:36.758 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T13:59:36.758 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.758 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T13:59:36.758 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6816", 2024-09-18T13:59:36.758 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 41936 2024-09-18T13:59:36.758 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.758 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.758 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T13:59:36.758 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6817", 2024-09-18T13:59:36.758 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 41936 2024-09-18T13:59:36.758 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.758 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T13:59:36.758 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.758 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addrs": { 2024-09-18T13:59:36.759 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T13:59:36.759 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.759 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T13:59:36.759 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6814", 2024-09-18T13:59:36.759 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 41936 2024-09-18T13:59:36.759 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.759 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.759 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T13:59:36.759 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6815", 2024-09-18T13:59:36.759 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 41936 2024-09-18T13:59:36.759 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.759 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T13:59:36.759 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.759 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addr": "127.0.0.1:6811/41936", 2024-09-18T13:59:36.759 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addr": "127.0.0.1:6813/41936", 2024-09-18T13:59:36.760 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addr": "127.0.0.1:6817/41936", 2024-09-18T13:59:36.760 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addr": "127.0.0.1:6815/41936", 2024-09-18T13:59:36.760 INFO:tasks.workunit.client.0.smithi047.stdout: "state": [ 2024-09-18T13:59:36.760 INFO:tasks.workunit.client.0.smithi047.stdout: "exists", 2024-09-18T13:59:36.760 INFO:tasks.workunit.client.0.smithi047.stdout: "up" 2024-09-18T13:59:36.760 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T13:59:36.760 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.760 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T13:59:36.760 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_xinfo": [ 2024-09-18T13:59:36.760 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.760 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 0, 2024-09-18T13:59:36.760 INFO:tasks.workunit.client.0.smithi047.stdout: "down_stamp": "2024-09-18T13:58:45.825106+0000", 2024-09-18T13:59:36.760 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_probability": 0, 2024-09-18T13:59:36.760 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_interval": 0, 2024-09-18T13:59:36.761 INFO:tasks.workunit.client.0.smithi047.stdout: "features": 4540138320759226367, 2024-09-18T13:59:36.761 INFO:tasks.workunit.client.0.smithi047.stdout: "old_weight": 0, 2024-09-18T13:59:36.761 INFO:tasks.workunit.client.0.smithi047.stdout: "last_purged_snaps_scrub": "2024-09-18T13:56:53.622020+0000", 2024-09-18T13:59:36.761 INFO:tasks.workunit.client.0.smithi047.stdout: "dead_epoch": 0 2024-09-18T13:59:36.761 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.761 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.761 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 1, 2024-09-18T13:59:36.761 INFO:tasks.workunit.client.0.smithi047.stdout: "down_stamp": "2024-09-18T13:58:46.461117+0000", 2024-09-18T13:59:36.761 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_probability": 0, 2024-09-18T13:59:36.761 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_interval": 0, 2024-09-18T13:59:36.761 INFO:tasks.workunit.client.0.smithi047.stdout: "features": 4540138320759226367, 2024-09-18T13:59:36.761 INFO:tasks.workunit.client.0.smithi047.stdout: "old_weight": 0, 2024-09-18T13:59:36.761 INFO:tasks.workunit.client.0.smithi047.stdout: "last_purged_snaps_scrub": "2024-09-18T13:57:26.257296+0000", 2024-09-18T13:59:36.761 INFO:tasks.workunit.client.0.smithi047.stdout: "dead_epoch": 0 2024-09-18T13:59:36.762 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.762 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.762 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 2, 2024-09-18T13:59:36.762 INFO:tasks.workunit.client.0.smithi047.stdout: "down_stamp": "2024-09-18T13:58:47.076793+0000", 2024-09-18T13:59:36.762 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_probability": 0, 2024-09-18T13:59:36.762 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_interval": 0, 2024-09-18T13:59:36.762 INFO:tasks.workunit.client.0.smithi047.stdout: "features": 4540138320759226367, 2024-09-18T13:59:36.762 INFO:tasks.workunit.client.0.smithi047.stdout: "old_weight": 0, 2024-09-18T13:59:36.762 INFO:tasks.workunit.client.0.smithi047.stdout: "last_purged_snaps_scrub": "2024-09-18T13:58:00.279608+0000", 2024-09-18T13:59:36.762 INFO:tasks.workunit.client.0.smithi047.stdout: "dead_epoch": 0 2024-09-18T13:59:36.762 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.762 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T13:59:36.762 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_upmap": [], 2024-09-18T13:59:36.762 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_upmap_items": [], 2024-09-18T13:59:36.763 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_temp": [], 2024-09-18T13:59:36.763 INFO:tasks.workunit.client.0.smithi047.stdout: "primary_temp": [], 2024-09-18T13:59:36.763 INFO:tasks.workunit.client.0.smithi047.stdout: "blocklist": {}, 2024-09-18T13:59:36.763 INFO:tasks.workunit.client.0.smithi047.stdout: "range_blocklist": {}, 2024-09-18T13:59:36.763 INFO:tasks.workunit.client.0.smithi047.stdout: "erasure_code_profiles": { 2024-09-18T13:59:36.763 INFO:tasks.workunit.client.0.smithi047.stdout: "default": { 2024-09-18T13:59:36.763 INFO:tasks.workunit.client.0.smithi047.stdout: "k": "2", 2024-09-18T13:59:36.763 INFO:tasks.workunit.client.0.smithi047.stdout: "m": "2", 2024-09-18T13:59:36.763 INFO:tasks.workunit.client.0.smithi047.stdout: "plugin": "jerasure", 2024-09-18T13:59:36.763 INFO:tasks.workunit.client.0.smithi047.stdout: "technique": "reed_sol_van" 2024-09-18T13:59:36.763 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.763 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.763 INFO:tasks.workunit.client.0.smithi047.stdout: "removed_snaps_queue": [], 2024-09-18T13:59:36.763 INFO:tasks.workunit.client.0.smithi047.stdout: "new_removed_snaps": [], 2024-09-18T13:59:36.763 INFO:tasks.workunit.client.0.smithi047.stdout: "new_purged_snaps": [], 2024-09-18T13:59:36.764 INFO:tasks.workunit.client.0.smithi047.stdout: "crush_node_flags": {}, 2024-09-18T13:59:36.764 INFO:tasks.workunit.client.0.smithi047.stdout: "device_class_flags": {}, 2024-09-18T13:59:36.764 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_mode": { 2024-09-18T13:59:36.764 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_mode_enabled": false, 2024-09-18T13:59:36.764 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_bucket_count": 0, 2024-09-18T13:59:36.764 INFO:tasks.workunit.client.0.smithi047.stdout: "degraded_stretch_mode": 0, 2024-09-18T13:59:36.764 INFO:tasks.workunit.client.0.smithi047.stdout: "recovering_stretch_mode": 0, 2024-09-18T13:59:36.764 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_mode_bucket": 0 2024-09-18T13:59:36.764 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.764 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.764 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_metadata": [ 2024-09-18T13:59:36.764 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.764 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T13:59:36.764 INFO:tasks.workunit.client.0.smithi047.stdout: "arch": "x86_64", 2024-09-18T13:59:36.764 INFO:tasks.workunit.client.0.smithi047.stdout: "back_addr": "[v2:127.0.0.1:6804/41358,v1:127.0.0.1:6805/41358]", 2024-09-18T13:59:36.765 INFO:tasks.workunit.client.0.smithi047.stdout: "back_iface": "", 2024-09-18T13:59:36.765 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs": "1", 2024-09-18T13:59:36.765 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_db": "0", 2024-09-18T13:59:36.765 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_wal": "0", 2024-09-18T13:59:36.765 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_single_shared_device": "1", 2024-09-18T13:59:36.765 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_allocation_from_file": "0", 2024-09-18T13:59:36.765 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_access_mode": "file", 2024-09-18T13:59:36.765 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_block_size": "4096", 2024-09-18T13:59:36.765 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_devices": "sda", 2024-09-18T13:59:36.765 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-18T13:59:36.765 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-18T13:59:36.765 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-18T13:59:36.765 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_rotational": "1", 2024-09-18T13:59:36.766 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_size": "107374182400", 2024-09-18T13:59:36.766 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_support_discard": "1", 2024-09-18T13:59:36.766 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_type": "hdd", 2024-09-18T13:59:36.766 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_min_alloc_size": "4096", 2024-09-18T13:59:36.766 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_release": "quincy", 2024-09-18T13:59:36.766 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T13:59:36.766 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_short": "17.2.7-1645-g7061ea6c", 2024-09-18T13:59:36.766 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_when_created": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T13:59:36.766 INFO:tasks.workunit.client.0.smithi047.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-18T13:59:36.766 INFO:tasks.workunit.client.0.smithi047.stdout: "created_at": "2024-09-18T13:56:27.674497Z", 2024-09-18T13:59:36.766 INFO:tasks.workunit.client.0.smithi047.stdout: "default_device_class": "hdd", 2024-09-18T13:59:36.766 INFO:tasks.workunit.client.0.smithi047.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4HN80", 2024-09-18T13:59:36.766 INFO:tasks.workunit.client.0.smithi047.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-18T13:59:36.766 INFO:tasks.workunit.client.0.smithi047.stdout: "devices": "sda", 2024-09-18T13:59:36.766 INFO:tasks.workunit.client.0.smithi047.stdout: "distro": "centos", 2024-09-18T13:59:36.767 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_description": "CentOS Stream 9", 2024-09-18T13:59:36.767 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_version": "9", 2024-09-18T13:59:36.767 INFO:tasks.workunit.client.0.smithi047.stdout: "front_addr": "[v2:127.0.0.1:6802/41358,v1:127.0.0.1:6803/41358]", 2024-09-18T13:59:36.767 INFO:tasks.workunit.client.0.smithi047.stdout: "front_iface": "", 2024-09-18T13:59:36.767 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/41358,v1:127.0.0.1:6809/41358]", 2024-09-18T13:59:36.767 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/41358,v1:127.0.0.1:6807/41358]", 2024-09-18T13:59:36.767 INFO:tasks.workunit.client.0.smithi047.stdout: "hostname": "smithi047", 2024-09-18T13:59:36.767 INFO:tasks.workunit.client.0.smithi047.stdout: "journal_rotational": "1", 2024-09-18T13:59:36.767 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-18T13:59:36.767 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-18T13:59:36.767 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_swap_kb": "0", 2024-09-18T13:59:36.767 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_total_kb": "32493492", 2024-09-18T13:59:36.767 INFO:tasks.workunit.client.0.smithi047.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-18T13:59:36.767 INFO:tasks.workunit.client.0.smithi047.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-18T13:59:36.768 INFO:tasks.workunit.client.0.smithi047.stdout: "os": "Linux", 2024-09-18T13:59:36.768 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_data": "td/divergent-priors/0", 2024-09-18T13:59:36.768 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_objectstore": "bluestore", 2024-09-18T13:59:36.768 INFO:tasks.workunit.client.0.smithi047.stdout: "osdspec_affinity": "", 2024-09-18T13:59:36.768 INFO:tasks.workunit.client.0.smithi047.stdout: "rotational": "1" 2024-09-18T13:59:36.768 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.768 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.768 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 1, 2024-09-18T13:59:36.768 INFO:tasks.workunit.client.0.smithi047.stdout: "arch": "x86_64", 2024-09-18T13:59:36.768 INFO:tasks.workunit.client.0.smithi047.stdout: "back_addr": "[v2:127.0.0.1:6812/35448,v1:127.0.0.1:6813/35448]", 2024-09-18T13:59:36.768 INFO:tasks.workunit.client.0.smithi047.stdout: "back_iface": "", 2024-09-18T13:59:36.768 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs": "1", 2024-09-18T13:59:36.768 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_db": "0", 2024-09-18T13:59:36.768 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_wal": "0", 2024-09-18T13:59:36.768 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_single_shared_device": "1", 2024-09-18T13:59:36.769 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_allocation_from_file": "0", 2024-09-18T13:59:36.769 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_access_mode": "file", 2024-09-18T13:59:36.769 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_block_size": "4096", 2024-09-18T13:59:36.769 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_devices": "sda", 2024-09-18T13:59:36.769 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-18T13:59:36.769 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-18T13:59:36.769 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-18T13:59:36.769 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_rotational": "1", 2024-09-18T13:59:36.769 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_size": "107374182400", 2024-09-18T13:59:36.769 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_support_discard": "1", 2024-09-18T13:59:36.769 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_type": "hdd", 2024-09-18T13:59:36.769 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_min_alloc_size": "4096", 2024-09-18T13:59:36.769 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_release": "quincy", 2024-09-18T13:59:36.769 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T13:59:36.770 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_short": "17.2.7-1645-g7061ea6c", 2024-09-18T13:59:36.770 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_when_created": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T13:59:36.770 INFO:tasks.workunit.client.0.smithi047.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-18T13:59:36.770 INFO:tasks.workunit.client.0.smithi047.stdout: "created_at": "2024-09-18T13:57:00.239777Z", 2024-09-18T13:59:36.770 INFO:tasks.workunit.client.0.smithi047.stdout: "default_device_class": "hdd", 2024-09-18T13:59:36.770 INFO:tasks.workunit.client.0.smithi047.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4HN80", 2024-09-18T13:59:36.770 INFO:tasks.workunit.client.0.smithi047.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-18T13:59:36.770 INFO:tasks.workunit.client.0.smithi047.stdout: "devices": "sda", 2024-09-18T13:59:36.770 INFO:tasks.workunit.client.0.smithi047.stdout: "distro": "centos", 2024-09-18T13:59:36.770 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_description": "CentOS Stream 9", 2024-09-18T13:59:36.770 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_version": "9", 2024-09-18T13:59:36.770 INFO:tasks.workunit.client.0.smithi047.stdout: "front_addr": "[v2:127.0.0.1:6810/35448,v1:127.0.0.1:6811/35448]", 2024-09-18T13:59:36.770 INFO:tasks.workunit.client.0.smithi047.stdout: "front_iface": "", 2024-09-18T13:59:36.770 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/35448,v1:127.0.0.1:6817/35448]", 2024-09-18T13:59:36.770 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/35448,v1:127.0.0.1:6815/35448]", 2024-09-18T13:59:36.771 INFO:tasks.workunit.client.0.smithi047.stdout: "hostname": "smithi047", 2024-09-18T13:59:36.771 INFO:tasks.workunit.client.0.smithi047.stdout: "journal_rotational": "1", 2024-09-18T13:59:36.771 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-18T13:59:36.771 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-18T13:59:36.771 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_swap_kb": "0", 2024-09-18T13:59:36.771 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_total_kb": "32493492", 2024-09-18T13:59:36.771 INFO:tasks.workunit.client.0.smithi047.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-18T13:59:36.771 INFO:tasks.workunit.client.0.smithi047.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-18T13:59:36.771 INFO:tasks.workunit.client.0.smithi047.stdout: "os": "Linux", 2024-09-18T13:59:36.771 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_data": "td/divergent-priors/1", 2024-09-18T13:59:36.771 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_objectstore": "bluestore", 2024-09-18T13:59:36.771 INFO:tasks.workunit.client.0.smithi047.stdout: "osdspec_affinity": "", 2024-09-18T13:59:36.771 INFO:tasks.workunit.client.0.smithi047.stdout: "rotational": "1" 2024-09-18T13:59:36.771 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.771 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.772 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T13:59:36.772 INFO:tasks.workunit.client.0.smithi047.stdout: "arch": "x86_64", 2024-09-18T13:59:36.772 INFO:tasks.workunit.client.0.smithi047.stdout: "back_addr": "[v2:127.0.0.1:6812/41936,v1:127.0.0.1:6813/41936]", 2024-09-18T13:59:36.772 INFO:tasks.workunit.client.0.smithi047.stdout: "back_iface": "", 2024-09-18T13:59:36.772 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs": "1", 2024-09-18T13:59:36.772 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_db": "0", 2024-09-18T13:59:36.772 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_wal": "0", 2024-09-18T13:59:36.772 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_single_shared_device": "1", 2024-09-18T13:59:36.772 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_allocation_from_file": "0", 2024-09-18T13:59:36.772 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_access_mode": "file", 2024-09-18T13:59:36.772 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_block_size": "4096", 2024-09-18T13:59:36.772 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_devices": "sda", 2024-09-18T13:59:36.772 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-18T13:59:36.773 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-18T13:59:36.773 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-18T13:59:36.773 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_rotational": "1", 2024-09-18T13:59:36.773 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_size": "107374182400", 2024-09-18T13:59:36.773 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_support_discard": "1", 2024-09-18T13:59:36.773 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_type": "hdd", 2024-09-18T13:59:36.773 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_min_alloc_size": "4096", 2024-09-18T13:59:36.773 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_release": "quincy", 2024-09-18T13:59:36.773 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T13:59:36.773 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_short": "17.2.7-1645-g7061ea6c", 2024-09-18T13:59:36.773 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_when_created": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T13:59:36.773 INFO:tasks.workunit.client.0.smithi047.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-18T13:59:36.773 INFO:tasks.workunit.client.0.smithi047.stdout: "created_at": "2024-09-18T13:57:33.427371Z", 2024-09-18T13:59:36.773 INFO:tasks.workunit.client.0.smithi047.stdout: "default_device_class": "hdd", 2024-09-18T13:59:36.773 INFO:tasks.workunit.client.0.smithi047.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4HN80", 2024-09-18T13:59:36.774 INFO:tasks.workunit.client.0.smithi047.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-18T13:59:36.774 INFO:tasks.workunit.client.0.smithi047.stdout: "devices": "sda", 2024-09-18T13:59:36.774 INFO:tasks.workunit.client.0.smithi047.stdout: "distro": "centos", 2024-09-18T13:59:36.774 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_description": "CentOS Stream 9", 2024-09-18T13:59:36.774 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_version": "9", 2024-09-18T13:59:36.774 INFO:tasks.workunit.client.0.smithi047.stdout: "front_addr": "[v2:127.0.0.1:6810/41936,v1:127.0.0.1:6811/41936]", 2024-09-18T13:59:36.774 INFO:tasks.workunit.client.0.smithi047.stdout: "front_iface": "", 2024-09-18T13:59:36.774 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/41936,v1:127.0.0.1:6817/41936]", 2024-09-18T13:59:36.774 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/41936,v1:127.0.0.1:6815/41936]", 2024-09-18T13:59:36.774 INFO:tasks.workunit.client.0.smithi047.stdout: "hostname": "smithi047", 2024-09-18T13:59:36.774 INFO:tasks.workunit.client.0.smithi047.stdout: "journal_rotational": "1", 2024-09-18T13:59:36.774 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-18T13:59:36.774 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-18T13:59:36.774 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_swap_kb": "0", 2024-09-18T13:59:36.775 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_total_kb": "32493492", 2024-09-18T13:59:36.775 INFO:tasks.workunit.client.0.smithi047.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-18T13:59:36.775 INFO:tasks.workunit.client.0.smithi047.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-18T13:59:36.775 INFO:tasks.workunit.client.0.smithi047.stdout: "os": "Linux", 2024-09-18T13:59:36.775 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_data": "td/divergent-priors/2", 2024-09-18T13:59:36.775 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_objectstore": "bluestore", 2024-09-18T13:59:36.775 INFO:tasks.workunit.client.0.smithi047.stdout: "osdspec_affinity": "", 2024-09-18T13:59:36.775 INFO:tasks.workunit.client.0.smithi047.stdout: "rotational": "1" 2024-09-18T13:59:36.775 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.775 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T13:59:36.775 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap_clean_epochs": { 2024-09-18T13:59:36.775 INFO:tasks.workunit.client.0.smithi047.stdout: "min_last_epoch_clean": 0, 2024-09-18T13:59:36.775 INFO:tasks.workunit.client.0.smithi047.stdout: "last_epoch_clean": { 2024-09-18T13:59:36.775 INFO:tasks.workunit.client.0.smithi047.stdout: "per_pool": [] 2024-09-18T13:59:36.775 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.776 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_epochs": [ 2024-09-18T13:59:36.776 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.776 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T13:59:36.776 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 34 2024-09-18T13:59:36.776 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.776 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.776 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T13:59:36.776 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 36 2024-09-18T13:59:36.776 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.776 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T13:59:36.776 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.776 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap_first_committed": 1, 2024-09-18T13:59:36.776 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap_last_committed": 40, 2024-09-18T13:59:36.776 INFO:tasks.workunit.client.0.smithi047.stdout: "crushmap": { 2024-09-18T13:59:36.776 INFO:tasks.workunit.client.0.smithi047.stdout: "devices": [ 2024-09-18T13:59:36.777 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.777 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T13:59:36.777 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "osd.0", 2024-09-18T13:59:36.777 INFO:tasks.workunit.client.0.smithi047.stdout: "class": "hdd" 2024-09-18T13:59:36.777 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.777 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.777 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 1, 2024-09-18T13:59:36.777 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "osd.1", 2024-09-18T13:59:36.777 INFO:tasks.workunit.client.0.smithi047.stdout: "class": "hdd" 2024-09-18T13:59:36.777 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.777 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.777 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T13:59:36.777 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "osd.2", 2024-09-18T13:59:36.777 INFO:tasks.workunit.client.0.smithi047.stdout: "class": "hdd" 2024-09-18T13:59:36.778 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.778 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T13:59:36.778 INFO:tasks.workunit.client.0.smithi047.stdout: "types": [ 2024-09-18T13:59:36.778 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.778 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 0, 2024-09-18T13:59:36.778 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "osd" 2024-09-18T13:59:36.778 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.778 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.778 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 1, 2024-09-18T13:59:36.778 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "host" 2024-09-18T13:59:36.778 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.778 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.778 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 2, 2024-09-18T13:59:36.778 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "chassis" 2024-09-18T13:59:36.779 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.779 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.779 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 3, 2024-09-18T13:59:36.779 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "rack" 2024-09-18T13:59:36.779 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.779 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.779 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 4, 2024-09-18T13:59:36.779 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "row" 2024-09-18T13:59:36.779 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.779 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.779 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 5, 2024-09-18T13:59:36.779 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "pdu" 2024-09-18T13:59:36.779 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.779 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.779 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 6, 2024-09-18T13:59:36.780 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "pod" 2024-09-18T13:59:36.780 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.780 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.780 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 7, 2024-09-18T13:59:36.780 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "room" 2024-09-18T13:59:36.780 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.780 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.780 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 8, 2024-09-18T13:59:36.780 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "datacenter" 2024-09-18T13:59:36.780 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.780 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.780 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 9, 2024-09-18T13:59:36.780 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "zone" 2024-09-18T13:59:36.780 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.780 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.781 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 10, 2024-09-18T13:59:36.781 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "region" 2024-09-18T13:59:36.781 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.781 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.781 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 11, 2024-09-18T13:59:36.781 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "root" 2024-09-18T13:59:36.781 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.781 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T13:59:36.781 INFO:tasks.workunit.client.0.smithi047.stdout: "buckets": [ 2024-09-18T13:59:36.781 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.781 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -1, 2024-09-18T13:59:36.781 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "default", 2024-09-18T13:59:36.781 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 11, 2024-09-18T13:59:36.781 INFO:tasks.workunit.client.0.smithi047.stdout: "type_name": "root", 2024-09-18T13:59:36.781 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T13:59:36.782 INFO:tasks.workunit.client.0.smithi047.stdout: "alg": "straw2", 2024-09-18T13:59:36.782 INFO:tasks.workunit.client.0.smithi047.stdout: "hash": "rjenkins1", 2024-09-18T13:59:36.782 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T13:59:36.782 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.782 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -3, 2024-09-18T13:59:36.782 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T13:59:36.782 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 0 2024-09-18T13:59:36.782 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.782 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T13:59:36.782 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.782 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.782 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -2, 2024-09-18T13:59:36.782 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "default~hdd", 2024-09-18T13:59:36.782 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 11, 2024-09-18T13:59:36.782 INFO:tasks.workunit.client.0.smithi047.stdout: "type_name": "root", 2024-09-18T13:59:36.782 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T13:59:36.783 INFO:tasks.workunit.client.0.smithi047.stdout: "alg": "straw2", 2024-09-18T13:59:36.783 INFO:tasks.workunit.client.0.smithi047.stdout: "hash": "rjenkins1", 2024-09-18T13:59:36.783 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T13:59:36.783 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.783 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -4, 2024-09-18T13:59:36.783 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T13:59:36.783 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 0 2024-09-18T13:59:36.783 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.783 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T13:59:36.783 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.783 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.783 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -3, 2024-09-18T13:59:36.783 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "smithi047", 2024-09-18T13:59:36.783 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 1, 2024-09-18T13:59:36.784 INFO:tasks.workunit.client.0.smithi047.stdout: "type_name": "host", 2024-09-18T13:59:36.784 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T13:59:36.784 INFO:tasks.workunit.client.0.smithi047.stdout: "alg": "straw2", 2024-09-18T13:59:36.784 INFO:tasks.workunit.client.0.smithi047.stdout: "hash": "rjenkins1", 2024-09-18T13:59:36.784 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T13:59:36.784 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.784 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T13:59:36.784 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T13:59:36.784 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 0 2024-09-18T13:59:36.784 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.784 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.784 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 1, 2024-09-18T13:59:36.784 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T13:59:36.784 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 1 2024-09-18T13:59:36.784 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.785 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.785 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T13:59:36.785 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T13:59:36.785 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 2 2024-09-18T13:59:36.785 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.785 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T13:59:36.785 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.785 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.785 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -4, 2024-09-18T13:59:36.785 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "smithi047~hdd", 2024-09-18T13:59:36.785 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 1, 2024-09-18T13:59:36.785 INFO:tasks.workunit.client.0.smithi047.stdout: "type_name": "host", 2024-09-18T13:59:36.785 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T13:59:36.785 INFO:tasks.workunit.client.0.smithi047.stdout: "alg": "straw2", 2024-09-18T13:59:36.785 INFO:tasks.workunit.client.0.smithi047.stdout: "hash": "rjenkins1", 2024-09-18T13:59:36.786 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T13:59:36.786 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.786 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T13:59:36.786 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T13:59:36.786 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 0 2024-09-18T13:59:36.786 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.786 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.786 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 1, 2024-09-18T13:59:36.786 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T13:59:36.786 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 1 2024-09-18T13:59:36.786 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.786 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.786 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T13:59:36.786 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T13:59:36.786 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 2 2024-09-18T13:59:36.787 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.787 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T13:59:36.787 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.787 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T13:59:36.787 INFO:tasks.workunit.client.0.smithi047.stdout: "rules": [ 2024-09-18T13:59:36.787 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.787 INFO:tasks.workunit.client.0.smithi047.stdout: "rule_id": 0, 2024-09-18T13:59:36.787 INFO:tasks.workunit.client.0.smithi047.stdout: "rule_name": "replicated_rule", 2024-09-18T13:59:36.787 INFO:tasks.workunit.client.0.smithi047.stdout: "type": 1, 2024-09-18T13:59:36.787 INFO:tasks.workunit.client.0.smithi047.stdout: "steps": [ 2024-09-18T13:59:36.787 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.787 INFO:tasks.workunit.client.0.smithi047.stdout: "op": "take", 2024-09-18T13:59:36.787 INFO:tasks.workunit.client.0.smithi047.stdout: "item": -1, 2024-09-18T13:59:36.787 INFO:tasks.workunit.client.0.smithi047.stdout: "item_name": "default" 2024-09-18T13:59:36.787 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.787 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.788 INFO:tasks.workunit.client.0.smithi047.stdout: "op": "choose_firstn", 2024-09-18T13:59:36.788 INFO:tasks.workunit.client.0.smithi047.stdout: "num": 0, 2024-09-18T13:59:36.788 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "osd" 2024-09-18T13:59:36.788 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.788 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.788 INFO:tasks.workunit.client.0.smithi047.stdout: "op": "emit" 2024-09-18T13:59:36.788 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.788 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T13:59:36.788 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.788 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T13:59:36.788 INFO:tasks.workunit.client.0.smithi047.stdout: "tunables": { 2024-09-18T13:59:36.788 INFO:tasks.workunit.client.0.smithi047.stdout: "choose_local_tries": 0, 2024-09-18T13:59:36.788 INFO:tasks.workunit.client.0.smithi047.stdout: "choose_local_fallback_tries": 0, 2024-09-18T13:59:36.788 INFO:tasks.workunit.client.0.smithi047.stdout: "choose_total_tries": 50, 2024-09-18T13:59:36.788 INFO:tasks.workunit.client.0.smithi047.stdout: "chooseleaf_descend_once": 1, 2024-09-18T13:59:36.789 INFO:tasks.workunit.client.0.smithi047.stdout: "chooseleaf_vary_r": 1, 2024-09-18T13:59:36.789 INFO:tasks.workunit.client.0.smithi047.stdout: "chooseleaf_stable": 1, 2024-09-18T13:59:36.789 INFO:tasks.workunit.client.0.smithi047.stdout: "straw_calc_version": 1, 2024-09-18T13:59:36.789 INFO:tasks.workunit.client.0.smithi047.stdout: "allowed_bucket_algs": 54, 2024-09-18T13:59:36.789 INFO:tasks.workunit.client.0.smithi047.stdout: "profile": "jewel", 2024-09-18T13:59:36.789 INFO:tasks.workunit.client.0.smithi047.stdout: "optimal_tunables": 1, 2024-09-18T13:59:36.789 INFO:tasks.workunit.client.0.smithi047.stdout: "legacy_tunables": 0, 2024-09-18T13:59:36.789 INFO:tasks.workunit.client.0.smithi047.stdout: "minimum_required_version": "jewel", 2024-09-18T13:59:36.789 INFO:tasks.workunit.client.0.smithi047.stdout: "require_feature_tunables": 1, 2024-09-18T13:59:36.789 INFO:tasks.workunit.client.0.smithi047.stdout: "require_feature_tunables2": 1, 2024-09-18T13:59:36.790 INFO:tasks.workunit.client.0.smithi047.stdout: "has_v2_rules": 0, 2024-09-18T13:59:36.790 INFO:tasks.workunit.client.0.smithi047.stdout: "require_feature_tunables3": 1, 2024-09-18T13:59:36.790 INFO:tasks.workunit.client.0.smithi047.stdout: "has_v3_rules": 0, 2024-09-18T13:59:36.790 INFO:tasks.workunit.client.0.smithi047.stdout: "has_v4_buckets": 1, 2024-09-18T13:59:36.790 INFO:tasks.workunit.client.0.smithi047.stdout: "require_feature_tunables5": 1, 2024-09-18T13:59:36.790 INFO:tasks.workunit.client.0.smithi047.stdout: "has_v5_rules": 0 2024-09-18T13:59:36.790 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.790 INFO:tasks.workunit.client.0.smithi047.stdout: "choose_args": {} 2024-09-18T13:59:36.790 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.790 INFO:tasks.workunit.client.0.smithi047.stdout: "fsmap": { 2024-09-18T13:59:36.790 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 1, 2024-09-18T13:59:36.790 INFO:tasks.workunit.client.0.smithi047.stdout: "default_fscid": -1, 2024-09-18T13:59:36.790 INFO:tasks.workunit.client.0.smithi047.stdout: "compat": { 2024-09-18T13:59:36.790 INFO:tasks.workunit.client.0.smithi047.stdout: "compat": {}, 2024-09-18T13:59:36.790 INFO:tasks.workunit.client.0.smithi047.stdout: "ro_compat": {}, 2024-09-18T13:59:36.791 INFO:tasks.workunit.client.0.smithi047.stdout: "incompat": { 2024-09-18T13:59:36.791 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_1": "base v0.20", 2024-09-18T13:59:36.791 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_2": "client writeable ranges", 2024-09-18T13:59:36.791 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_3": "default file layouts on dirs", 2024-09-18T13:59:36.791 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_4": "dir inode in separate object", 2024-09-18T13:59:36.791 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_5": "mds uses versioned encoding", 2024-09-18T13:59:36.791 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-18T13:59:36.791 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_8": "no anchor table", 2024-09-18T13:59:36.791 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_9": "file layout v2", 2024-09-18T13:59:36.791 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_10": "snaprealm v2" 2024-09-18T13:59:36.791 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.791 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.791 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_flags": { 2024-09-18T13:59:36.791 INFO:tasks.workunit.client.0.smithi047.stdout: "enable_multiple": true, 2024-09-18T13:59:36.791 INFO:tasks.workunit.client.0.smithi047.stdout: "ever_enabled_multiple": true 2024-09-18T13:59:36.792 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.792 INFO:tasks.workunit.client.0.smithi047.stdout: "standbys": [], 2024-09-18T13:59:36.792 INFO:tasks.workunit.client.0.smithi047.stdout: "filesystems": [] 2024-09-18T13:59:36.792 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.792 INFO:tasks.workunit.client.0.smithi047.stdout: "mdsmap_first_committed": 1, 2024-09-18T13:59:36.792 INFO:tasks.workunit.client.0.smithi047.stdout: "mdsmap_last_committed": 1, 2024-09-18T13:59:36.792 INFO:tasks.workunit.client.0.smithi047.stdout: "auth": { 2024-09-18T13:59:36.792 INFO:tasks.workunit.client.0.smithi047.stdout: "first_committed": 1, 2024-09-18T13:59:36.792 INFO:tasks.workunit.client.0.smithi047.stdout: "last_committed": 5, 2024-09-18T13:59:36.792 INFO:tasks.workunit.client.0.smithi047.stdout: "num_secrets": 3 2024-09-18T13:59:36.792 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.792 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg": 1, 2024-09-18T13:59:36.792 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg_active": 1, 2024-09-18T13:59:36.792 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg_unknown": 0, 2024-09-18T13:59:36.792 INFO:tasks.workunit.client.0.smithi047.stdout: "num_osd": 3, 2024-09-18T13:59:36.793 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_sum": { 2024-09-18T13:59:36.793 INFO:tasks.workunit.client.0.smithi047.stdout: "stat_sum": { 2024-09-18T13:59:36.793 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes": 48300, 2024-09-18T13:59:36.793 INFO:tasks.workunit.client.0.smithi047.stderr:report 2683124902 2024-09-18T13:59:36.793 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects": 100, 2024-09-18T13:59:36.793 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_clones": 0, 2024-09-18T13:59:36.793 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_copies": 300, 2024-09-18T13:59:36.794 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing_on_primary": 0, 2024-09-18T13:59:36.794 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing": 0, 2024-09-18T13:59:36.794 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_degraded": 100, 2024-09-18T13:59:36.794 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_misplaced": 0, 2024-09-18T13:59:36.794 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_unfound": 0, 2024-09-18T13:59:36.794 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_dirty": 100, 2024-09-18T13:59:36.794 INFO:tasks.workunit.client.0.smithi047.stdout: "num_whiteouts": 0, 2024-09-18T13:59:36.794 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read": 0, 2024-09-18T13:59:36.794 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read_kb": 0, 2024-09-18T13:59:36.794 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write": 100, 2024-09-18T13:59:36.794 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write_kb": 100, 2024-09-18T13:59:36.794 INFO:tasks.workunit.client.0.smithi047.stdout: "num_scrub_errors": 0, 2024-09-18T13:59:36.794 INFO:tasks.workunit.client.0.smithi047.stdout: "num_shallow_scrub_errors": 0, 2024-09-18T13:59:36.794 INFO:tasks.workunit.client.0.smithi047.stdout: "num_deep_scrub_errors": 0, 2024-09-18T13:59:36.794 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_recovered": 0, 2024-09-18T13:59:36.795 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_recovered": 0, 2024-09-18T13:59:36.795 INFO:tasks.workunit.client.0.smithi047.stdout: "num_keys_recovered": 0, 2024-09-18T13:59:36.795 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_omap": 0, 2024-09-18T13:59:36.795 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_hit_set_archive": 0, 2024-09-18T13:59:36.795 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_hit_set_archive": 0, 2024-09-18T13:59:36.795 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush": 0, 2024-09-18T13:59:36.795 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_kb": 0, 2024-09-18T13:59:36.795 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict": 0, 2024-09-18T13:59:36.795 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_kb": 0, 2024-09-18T13:59:36.795 INFO:tasks.workunit.client.0.smithi047.stdout: "num_promote": 0, 2024-09-18T13:59:36.795 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_high": 0, 2024-09-18T13:59:36.795 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_low": 0, 2024-09-18T13:59:36.795 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_some": 0, 2024-09-18T13:59:36.795 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_full": 0, 2024-09-18T13:59:36.795 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_pinned": 0, 2024-09-18T13:59:36.796 INFO:tasks.workunit.client.0.smithi047.stdout: "num_legacy_snapsets": 0, 2024-09-18T13:59:36.796 INFO:tasks.workunit.client.0.smithi047.stdout: "num_large_omap_objects": 0, 2024-09-18T13:59:36.796 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_manifest": 0, 2024-09-18T13:59:36.796 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_bytes": 0, 2024-09-18T13:59:36.796 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_keys": 0, 2024-09-18T13:59:36.796 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_repaired": 0 2024-09-18T13:59:36.796 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.796 INFO:tasks.workunit.client.0.smithi047.stdout: "store_stats": { 2024-09-18T13:59:36.796 INFO:tasks.workunit.client.0.smithi047.stdout: "total": 0, 2024-09-18T13:59:36.796 INFO:tasks.workunit.client.0.smithi047.stdout: "available": 0, 2024-09-18T13:59:36.796 INFO:tasks.workunit.client.0.smithi047.stdout: "internally_reserved": 0, 2024-09-18T13:59:36.796 INFO:tasks.workunit.client.0.smithi047.stdout: "allocated": 0, 2024-09-18T13:59:36.796 INFO:tasks.workunit.client.0.smithi047.stdout: "data_stored": 0, 2024-09-18T13:59:36.796 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed": 0, 2024-09-18T13:59:36.797 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_allocated": 0, 2024-09-18T13:59:36.797 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_original": 0, 2024-09-18T13:59:36.797 INFO:tasks.workunit.client.0.smithi047.stdout: "omap_allocated": 0, 2024-09-18T13:59:36.797 INFO:tasks.workunit.client.0.smithi047.stdout: "internal_metadata": 0 2024-09-18T13:59:36.797 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.797 INFO:tasks.workunit.client.0.smithi047.stdout: "log_size": 16, 2024-09-18T13:59:36.797 INFO:tasks.workunit.client.0.smithi047.stdout: "ondisk_log_size": 16, 2024-09-18T13:59:36.797 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 2, 2024-09-18T13:59:36.797 INFO:tasks.workunit.client.0.smithi047.stdout: "acting": 2, 2024-09-18T13:59:36.797 INFO:tasks.workunit.client.0.smithi047.stdout: "num_store_stats": 0 2024-09-18T13:59:36.797 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.797 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_sum": { 2024-09-18T13:59:36.797 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 0, 2024-09-18T13:59:36.797 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 0, 2024-09-18T13:59:36.797 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pgs": 3, 2024-09-18T13:59:36.798 INFO:tasks.workunit.client.0.smithi047.stdout: "num_osds": 3, 2024-09-18T13:59:36.798 INFO:tasks.workunit.client.0.smithi047.stdout: "num_per_pool_osds": 3, 2024-09-18T13:59:36.798 INFO:tasks.workunit.client.0.smithi047.stdout: "num_per_pool_omap_osds": 3, 2024-09-18T13:59:36.798 INFO:tasks.workunit.client.0.smithi047.stdout: "kb": 314572800, 2024-09-18T13:59:36.798 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used": 894608, 2024-09-18T13:59:36.799 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_data": 2080, 2024-09-18T13:59:36.800 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_omap": 10, 2024-09-18T13:59:36.800 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_meta": 892469, 2024-09-18T13:59:36.800 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_avail": 313678192, 2024-09-18T13:59:36.800 INFO:tasks.workunit.client.0.smithi047.stdout: "statfs": { 2024-09-18T13:59:36.800 INFO:tasks.workunit.client.0.smithi047.stdout: "total": 322122547200, 2024-09-18T13:59:36.800 INFO:tasks.workunit.client.0.smithi047.stdout: "available": 321206468608, 2024-09-18T13:59:36.800 INFO:tasks.workunit.client.0.smithi047.stdout: "internally_reserved": 0, 2024-09-18T13:59:36.800 INFO:tasks.workunit.client.0.smithi047.stdout: "allocated": 2129920, 2024-09-18T13:59:36.800 INFO:tasks.workunit.client.0.smithi047.stdout: "data_stored": 408484, 2024-09-18T13:59:36.800 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed": 0, 2024-09-18T13:59:36.800 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_allocated": 0, 2024-09-18T13:59:36.800 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_original": 0, 2024-09-18T13:59:36.801 INFO:tasks.workunit.client.0.smithi047.stdout: "omap_allocated": 10790, 2024-09-18T13:59:36.801 INFO:tasks.workunit.client.0.smithi047.stdout: "internal_metadata": 913888730 2024-09-18T13:59:36.801 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.801 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_peers": [], 2024-09-18T13:59:36.801 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_trim_queue_len": 0, 2024-09-18T13:59:36.801 INFO:tasks.workunit.client.0.smithi047.stdout: "num_snap_trimming": 0, 2024-09-18T13:59:36.801 INFO:tasks.workunit.client.0.smithi047.stdout: "num_shards_repaired": 1, 2024-09-18T13:59:36.801 INFO:tasks.workunit.client.0.smithi047.stdout: "op_queue_age_hist": { 2024-09-18T13:59:36.801 INFO:tasks.workunit.client.0.smithi047.stdout: "histogram": [], 2024-09-18T13:59:36.801 INFO:tasks.workunit.client.0.smithi047.stdout: "upper_bound": 1 2024-09-18T13:59:36.801 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.801 INFO:tasks.workunit.client.0.smithi047.stdout: "perf_stat": { 2024-09-18T13:59:36.801 INFO:tasks.workunit.client.0.smithi047.stdout: "commit_latency_ms": 0, 2024-09-18T13:59:36.802 INFO:tasks.workunit.client.0.smithi047.stdout: "apply_latency_ms": 0, 2024-09-18T13:59:36.802 INFO:tasks.workunit.client.0.smithi047.stdout: "commit_latency_ns": 0, 2024-09-18T13:59:36.802 INFO:tasks.workunit.client.0.smithi047.stdout: "apply_latency_ns": 0 2024-09-18T13:59:36.802 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.802 INFO:tasks.workunit.client.0.smithi047.stdout: "alerts": [], 2024-09-18T13:59:36.802 INFO:tasks.workunit.client.0.smithi047.stdout: "network_ping_times": [] 2024-09-18T13:59:36.802 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.802 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_sum_by_class": { 2024-09-18T13:59:36.802 INFO:tasks.workunit.client.0.smithi047.stdout: "hdd": { 2024-09-18T13:59:36.802 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 0, 2024-09-18T13:59:36.802 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 0, 2024-09-18T13:59:36.802 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pgs": 3, 2024-09-18T13:59:36.802 INFO:tasks.workunit.client.0.smithi047.stdout: "num_osds": 3, 2024-09-18T13:59:36.802 INFO:tasks.workunit.client.0.smithi047.stdout: "num_per_pool_osds": 3, 2024-09-18T13:59:36.803 INFO:tasks.workunit.client.0.smithi047.stdout: "num_per_pool_omap_osds": 3, 2024-09-18T13:59:36.803 INFO:tasks.workunit.client.0.smithi047.stdout: "kb": 314572800, 2024-09-18T13:59:36.803 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used": 894608, 2024-09-18T13:59:36.803 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_data": 2080, 2024-09-18T13:59:36.803 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_omap": 10, 2024-09-18T13:59:36.803 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_meta": 892469, 2024-09-18T13:59:36.803 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_avail": 313678192, 2024-09-18T13:59:36.803 INFO:tasks.workunit.client.0.smithi047.stdout: "statfs": { 2024-09-18T13:59:36.803 INFO:tasks.workunit.client.0.smithi047.stdout: "total": 322122547200, 2024-09-18T13:59:36.803 INFO:tasks.workunit.client.0.smithi047.stdout: "available": 321206468608, 2024-09-18T13:59:36.803 INFO:tasks.workunit.client.0.smithi047.stdout: "internally_reserved": 0, 2024-09-18T13:59:36.803 INFO:tasks.workunit.client.0.smithi047.stdout: "allocated": 2129920, 2024-09-18T13:59:36.803 INFO:tasks.workunit.client.0.smithi047.stdout: "data_stored": 408484, 2024-09-18T13:59:36.803 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed": 0, 2024-09-18T13:59:36.803 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_allocated": 0, 2024-09-18T13:59:36.804 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_original": 0, 2024-09-18T13:59:36.804 INFO:tasks.workunit.client.0.smithi047.stdout: "omap_allocated": 10790, 2024-09-18T13:59:36.804 INFO:tasks.workunit.client.0.smithi047.stdout: "internal_metadata": 913888730 2024-09-18T13:59:36.804 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.804 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_peers": [], 2024-09-18T13:59:36.804 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_trim_queue_len": 0, 2024-09-18T13:59:36.804 INFO:tasks.workunit.client.0.smithi047.stdout: "num_snap_trimming": 0, 2024-09-18T13:59:36.804 INFO:tasks.workunit.client.0.smithi047.stdout: "num_shards_repaired": 1, 2024-09-18T13:59:36.804 INFO:tasks.workunit.client.0.smithi047.stdout: "op_queue_age_hist": { 2024-09-18T13:59:36.804 INFO:tasks.workunit.client.0.smithi047.stdout: "histogram": [], 2024-09-18T13:59:36.804 INFO:tasks.workunit.client.0.smithi047.stdout: "upper_bound": 1 2024-09-18T13:59:36.804 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.804 INFO:tasks.workunit.client.0.smithi047.stdout: "perf_stat": { 2024-09-18T13:59:36.804 INFO:tasks.workunit.client.0.smithi047.stdout: "commit_latency_ms": 0, 2024-09-18T13:59:36.805 INFO:tasks.workunit.client.0.smithi047.stdout: "apply_latency_ms": 0, 2024-09-18T13:59:36.805 INFO:tasks.workunit.client.0.smithi047.stdout: "commit_latency_ns": 0, 2024-09-18T13:59:36.805 INFO:tasks.workunit.client.0.smithi047.stdout: "apply_latency_ns": 0 2024-09-18T13:59:36.805 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.805 INFO:tasks.workunit.client.0.smithi047.stdout: "alerts": [], 2024-09-18T13:59:36.805 INFO:tasks.workunit.client.0.smithi047.stdout: "network_ping_times": [] 2024-09-18T13:59:36.805 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.805 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.805 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_stats": [ 2024-09-18T13:59:36.805 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.805 INFO:tasks.workunit.client.0.smithi047.stdout: "poolid": 1, 2024-09-18T13:59:36.805 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg": 1, 2024-09-18T13:59:36.805 INFO:tasks.workunit.client.0.smithi047.stdout: "stat_sum": { 2024-09-18T13:59:36.805 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes": 48300, 2024-09-18T13:59:36.805 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects": 100, 2024-09-18T13:59:36.806 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_clones": 0, 2024-09-18T13:59:36.806 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_copies": 300, 2024-09-18T13:59:36.806 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing_on_primary": 0, 2024-09-18T13:59:36.806 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing": 0, 2024-09-18T13:59:36.806 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_degraded": 100, 2024-09-18T13:59:36.806 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_misplaced": 0, 2024-09-18T13:59:36.806 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_unfound": 0, 2024-09-18T13:59:36.806 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_dirty": 100, 2024-09-18T13:59:36.806 INFO:tasks.workunit.client.0.smithi047.stdout: "num_whiteouts": 0, 2024-09-18T13:59:36.806 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read": 0, 2024-09-18T13:59:36.806 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read_kb": 0, 2024-09-18T13:59:36.806 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write": 100, 2024-09-18T13:59:36.806 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write_kb": 100, 2024-09-18T13:59:36.806 INFO:tasks.workunit.client.0.smithi047.stdout: "num_scrub_errors": 0, 2024-09-18T13:59:36.806 INFO:tasks.workunit.client.0.smithi047.stdout: "num_shallow_scrub_errors": 0, 2024-09-18T13:59:36.807 INFO:tasks.workunit.client.0.smithi047.stdout: "num_deep_scrub_errors": 0, 2024-09-18T13:59:36.807 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_recovered": 0, 2024-09-18T13:59:36.807 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_recovered": 0, 2024-09-18T13:59:36.807 INFO:tasks.workunit.client.0.smithi047.stdout: "num_keys_recovered": 0, 2024-09-18T13:59:36.807 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_omap": 0, 2024-09-18T13:59:36.807 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_hit_set_archive": 0, 2024-09-18T13:59:36.807 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_hit_set_archive": 0, 2024-09-18T13:59:36.807 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush": 0, 2024-09-18T13:59:36.807 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_kb": 0, 2024-09-18T13:59:36.807 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict": 0, 2024-09-18T13:59:36.807 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_kb": 0, 2024-09-18T13:59:36.807 INFO:tasks.workunit.client.0.smithi047.stdout: "num_promote": 0, 2024-09-18T13:59:36.807 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_high": 0, 2024-09-18T13:59:36.807 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_low": 0, 2024-09-18T13:59:36.808 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_some": 0, 2024-09-18T13:59:36.808 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_full": 0, 2024-09-18T13:59:36.808 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_pinned": 0, 2024-09-18T13:59:36.808 INFO:tasks.workunit.client.0.smithi047.stdout: "num_legacy_snapsets": 0, 2024-09-18T13:59:36.808 INFO:tasks.workunit.client.0.smithi047.stdout: "num_large_omap_objects": 0, 2024-09-18T13:59:36.808 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_manifest": 0, 2024-09-18T13:59:36.808 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_bytes": 0, 2024-09-18T13:59:36.808 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_keys": 0, 2024-09-18T13:59:36.808 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_repaired": 0 2024-09-18T13:59:36.808 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.808 INFO:tasks.workunit.client.0.smithi047.stdout: "store_stats": { 2024-09-18T13:59:36.808 INFO:tasks.workunit.client.0.smithi047.stdout: "total": 0, 2024-09-18T13:59:36.808 INFO:tasks.workunit.client.0.smithi047.stdout: "available": 0, 2024-09-18T13:59:36.808 INFO:tasks.workunit.client.0.smithi047.stdout: "internally_reserved": 0, 2024-09-18T13:59:36.809 INFO:tasks.workunit.client.0.smithi047.stdout: "allocated": 1228800, 2024-09-18T13:59:36.809 INFO:tasks.workunit.client.0.smithi047.stdout: "data_stored": 144900, 2024-09-18T13:59:36.809 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed": 0, 2024-09-18T13:59:36.809 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_allocated": 0, 2024-09-18T13:59:36.809 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_original": 0, 2024-09-18T13:59:36.809 INFO:tasks.workunit.client.0.smithi047.stdout: "omap_allocated": 0, 2024-09-18T13:59:36.809 INFO:tasks.workunit.client.0.smithi047.stdout: "internal_metadata": 0 2024-09-18T13:59:36.809 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.809 INFO:tasks.workunit.client.0.smithi047.stdout: "log_size": 16, 2024-09-18T13:59:36.809 INFO:tasks.workunit.client.0.smithi047.stdout: "ondisk_log_size": 16, 2024-09-18T13:59:36.809 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 2, 2024-09-18T13:59:36.809 INFO:tasks.workunit.client.0.smithi047.stdout: "acting": 2, 2024-09-18T13:59:36.809 INFO:tasks.workunit.client.0.smithi047.stdout: "num_store_stats": 3 2024-09-18T13:59:36.809 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.809 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T13:59:36.810 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_stats": [ 2024-09-18T13:59:36.810 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.810 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 0, 2024-09-18T13:59:36.810 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 146028888076 2024-09-18T13:59:36.810 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.810 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.810 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 1, 2024-09-18T13:59:36.810 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 42949672988 2024-09-18T13:59:36.810 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.810 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.810 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 2, 2024-09-18T13:59:36.810 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 154618822666 2024-09-18T13:59:36.810 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.810 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T13:59:36.810 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg_by_state": [ 2024-09-18T13:59:36.811 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.811 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "active+undersized+degraded", 2024-09-18T13:59:36.811 INFO:tasks.workunit.client.0.smithi047.stdout: "num": 1 2024-09-18T13:59:36.811 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.811 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T13:59:36.811 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg_by_osd": [ 2024-09-18T13:59:36.811 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.811 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 2, 2024-09-18T13:59:36.811 INFO:tasks.workunit.client.0.smithi047.stdout: "num_primary_pg": 0, 2024-09-18T13:59:36.811 INFO:tasks.workunit.client.0.smithi047.stdout: "num_acting_pg": 1, 2024-09-18T13:59:36.811 INFO:tasks.workunit.client.0.smithi047.stdout: "num_up_not_acting_pg": 0 2024-09-18T13:59:36.811 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.811 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.812 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 0, 2024-09-18T13:59:36.812 INFO:tasks.workunit.client.0.smithi047.stdout: "num_primary_pg": 1, 2024-09-18T13:59:36.812 INFO:tasks.workunit.client.0.smithi047.stdout: "num_acting_pg": 1, 2024-09-18T13:59:36.812 INFO:tasks.workunit.client.0.smithi047.stdout: "num_up_not_acting_pg": 0 2024-09-18T13:59:36.812 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.812 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.812 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 1, 2024-09-18T13:59:36.812 INFO:tasks.workunit.client.0.smithi047.stdout: "num_primary_pg": 0, 2024-09-18T13:59:36.812 INFO:tasks.workunit.client.0.smithi047.stdout: "num_acting_pg": 0, 2024-09-18T13:59:36.812 INFO:tasks.workunit.client.0.smithi047.stdout: "num_up_not_acting_pg": 0 2024-09-18T13:59:36.812 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.812 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T13:59:36.812 INFO:tasks.workunit.client.0.smithi047.stdout: "purged_snaps": [ 2024-09-18T13:59:36.812 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T13:59:36.813 INFO:tasks.workunit.client.0.smithi047.stdout: "pool": 1, 2024-09-18T13:59:36.813 INFO:tasks.workunit.client.0.smithi047.stdout: "purged_snaps": {} 2024-09-18T13:59:36.813 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.813 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T13:59:36.813 INFO:tasks.workunit.client.0.smithi047.stdout: "servicemap": { 2024-09-18T13:59:36.813 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 5, 2024-09-18T13:59:36.813 INFO:tasks.workunit.client.0.smithi047.stdout: "modified": "2024-09-18T13:58:50.235024+0000", 2024-09-18T13:59:36.813 INFO:tasks.workunit.client.0.smithi047.stdout: "services": {} 2024-09-18T13:59:36.813 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T13:59:36.813 INFO:tasks.workunit.client.0.smithi047.stdout: "paxos": { 2024-09-18T13:59:36.813 INFO:tasks.workunit.client.0.smithi047.stdout: "first_committed": 1, 2024-09-18T13:59:36.813 INFO:tasks.workunit.client.0.smithi047.stdout: "last_committed": 508, 2024-09-18T13:59:36.813 INFO:tasks.workunit.client.0.smithi047.stdout: "last_pn": 0, 2024-09-18T13:59:36.813 INFO:tasks.workunit.client.0.smithi047.stdout: "accepted_pn": 0 2024-09-18T13:59:36.814 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T13:59:36.814 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T13:59:36.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-18T13:59:36.814 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: expr 2 + 2 2024-09-18T13:59:36.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: objname=existing_4 2024-09-18T13:59:36.815 INFO:tasks.workunit.client.0.smithi047.stdout:writing non-divergent object existing_4 2024-09-18T13:59:36.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:154: TEST_divergent: echo 'writing non-divergent object existing_4' 2024-09-18T13:59:36.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:155: TEST_divergent: ceph pg dump pgs 2024-09-18T13:59:37.135 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T13:59:37.136 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 100 0 100 0 0 48300 0 0 16 84 16 active+undersized+degraded 2024-09-18T13:59:00.541745+0000 27'100 40:126 [0,2] 0 [0,2] 0 0'0 2024-09-18T13:58:04.104186+0000 0'0 2024-09-18T13:58:04.104186+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:37:01.991304+0000 0 0 2024-09-18T13:59:37.136 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T13:59:37.136 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T13:59:37.136 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T13:59:37.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:156: TEST_divergent: rados -p test put existing_4 /etc/resolv.conf 2024-09-18T13:59:37.248 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:159: TEST_divergent: echo 'delay recovery' 2024-09-18T13:59:37.248 INFO:tasks.workunit.client.0.smithi047.stdout:delay recovery 2024-09-18T13:59:37.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:160: TEST_divergent: ceph pg dump pgs 2024-09-18T13:59:37.601 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T13:59:37.602 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 100 0 100 0 0 48300 0 0 16 84 16 active+undersized+degraded 2024-09-18T13:59:00.541745+0000 27'100 40:126 [0,2] 0 [0,2] 0 0'0 2024-09-18T13:58:04.104186+0000 0'0 2024-09-18T13:58:04.104186+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:37:01.991304+0000 0 0 2024-09-18T13:59:37.602 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T13:59:37.602 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T13:59:37.602 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T13:59:37.612 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2024-09-18T13:59:37.613 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.0 2024-09-18T13:59:37.613 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T13:59:37.613 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T13:59:37.613 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T13:59:37.613 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T13:59:37.613 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T13:59:37.613 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32609/ceph-osd.0.asok 2024-09-18T13:59:37.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2024-09-18T13:59:37.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.32609/ceph-osd.0.asok set_recovery_delay 100000 2024-09-18T13:59:37.691 INFO:tasks.workunit.client.0.smithi047.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-18T13:59:37.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2024-09-18T13:59:37.699 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.2 2024-09-18T13:59:37.699 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T13:59:37.699 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T13:59:37.699 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T13:59:37.700 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T13:59:37.700 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T13:59:37.700 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32609/ceph-osd.2.asok 2024-09-18T13:59:37.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2024-09-18T13:59:37.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.32609/ceph-osd.2.asok set_recovery_delay 100000 2024-09-18T13:59:37.778 INFO:tasks.workunit.client.0.smithi047.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-18T13:59:37.785 INFO:tasks.workunit.client.0.smithi047.stdout:revive divergent 1 2024-09-18T13:59:37.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:167: TEST_divergent: echo 'revive divergent 1' 2024-09-18T13:59:37.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:168: TEST_divergent: ceph pg dump pgs 2024-09-18T13:59:38.140 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T13:59:38.140 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 100 0 100 0 0 48300 0 0 16 84 16 active+undersized+degraded 2024-09-18T13:59:00.541745+0000 27'100 40:126 [0,2] 0 [0,2] 0 0'0 2024-09-18T13:58:04.104186+0000 0'0 2024-09-18T13:58:04.104186+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:37:01.991304+0000 0 0 2024-09-18T13:59:38.141 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T13:59:38.141 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T13:59:38.141 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T13:59:38.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:169: TEST_divergent: ceph osd set noup 2024-09-18T13:59:38.866 INFO:tasks.workunit.client.0.smithi047.stderr:noup is set 2024-09-18T13:59:38.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:170: TEST_divergent: activate_osd td/divergent-priors 1 2024-09-18T13:59:38.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-18T13:59:38.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T13:59:38.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T13:59:38.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T13:59:38.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-09-18T13:59:38.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T13:59:38.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T13:59:38.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T13:59:38.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T13:59:38.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-18T13:59:38.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-18T13:59:38.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T13:59:38.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T13:59:38.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-18T13:59:38.881 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T13:59:38.881 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T13:59:38.881 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T13:59:38.881 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T13:59:38.881 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T13:59:38.881 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T13:59:38.881 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T13:59:38.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T13:59:38.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T13:59:38.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-18T13:59:38.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-18T13:59:38.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T13:59:38.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T13:59:38.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T13:59:38.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T13:59:38.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T13:59:38.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T13:59:38.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-09-18T13:59:38.882 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T13:59:38.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T13:59:38.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T13:59:38.883 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-09-18T13:59:38.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T13:59:38.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T13:59:38.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T13:59:38.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T13:59:38.911 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:59:38.910+0000 7fda2f40e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:59:38.913 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:59:38.913+0000 7fda2f40e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:59:38.914 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:59:38.914+0000 7fda2f40e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:59:39.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-09-18T13:59:39.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:171: TEST_divergent: sleep 5 2024-09-18T13:59:39.432 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:59:39.432+0000 7fda2f40e440 -1 Falling back to public interface 2024-09-18T13:59:42.267 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:59:42.266+0000 7fda2f40e440 -1 osd.1 27 log_to_monitors true 2024-09-18T13:59:44.298 INFO:tasks.workunit.client.0.smithi047.stdout:delay recovery divergent 2024-09-18T13:59:44.299 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:173: TEST_divergent: echo 'delay recovery divergent' 2024-09-18T13:59:44.299 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:174: TEST_divergent: ceph pg dump pgs 2024-09-18T13:59:44.665 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T13:59:44.665 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-18T13:59:00.541745+0000 40'101 42:129 [0,2] 0 [0,2] 0 0'0 2024-09-18T13:58:04.104186+0000 0'0 2024-09-18T13:58:04.104186+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:37:01.991304+0000 0 0 2024-09-18T13:59:44.665 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T13:59:44.665 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T13:59:44.665 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T13:59:44.677 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: get_asok_path osd.1 2024-09-18T13:59:44.677 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T13:59:44.677 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T13:59:44.677 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T13:59:44.677 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T13:59:44.677 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T13:59:44.677 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32609/ceph-osd.1.asok 2024-09-18T13:59:44.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: CEPH_ARGS= 2024-09-18T13:59:44.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.32609/ceph-osd.1.asok set_recovery_delay 100000 2024-09-18T13:59:44.755 INFO:tasks.workunit.client.0.smithi047.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-18T13:59:44.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:177: TEST_divergent: ceph osd unset noup 2024-09-18T13:59:45.390 INFO:tasks.workunit.client.0.smithi047.stderr:noup is unset 2024-09-18T13:59:45.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:179: TEST_divergent: wait_for_osd up 0 2024-09-18T13:59:45.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T13:59:45.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T13:59:45.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T13:59:45.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T13:59:45.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:59:45.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T13:59:45.403 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T13:59:45.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:59:45.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T13:59:45.776 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 34 up_thru 39 down_at 28 last_clean_interval [5,27) [v2:127.0.0.1:6802/41358,v1:127.0.0.1:6803/41358] [v2:127.0.0.1:6804/41358,v1:127.0.0.1:6805/41358] exists,up 120d8a51-9d44-40cc-b688-4581426ff01d 2024-09-18T13:59:45.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T13:59:45.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T13:59:45.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T13:59:45.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:180: TEST_divergent: wait_for_osd up 1 2024-09-18T13:59:45.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T13:59:45.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T13:59:45.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T13:59:45.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T13:59:45.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:59:45.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T13:59:45.778 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T13:59:45.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:59:45.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:59:46.227 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 45 up_thru 45 down_at 29 last_clean_interval [10,27) [v2:127.0.0.1:6818/43589,v1:127.0.0.1:6819/43589] [v2:127.0.0.1:6820/43589,v1:127.0.0.1:6821/43589] exists,up 3353f33a-e483-4b8b-976d-f7b4acd36b36 2024-09-18T13:59:46.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T13:59:46.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T13:59:46.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T13:59:46.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:181: TEST_divergent: wait_for_osd up 2 2024-09-18T13:59:46.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T13:59:46.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T13:59:46.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T13:59:46.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T13:59:46.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:59:46.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T13:59:46.229 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T13:59:46.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:59:46.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T13:59:46.633 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 36 up_thru 0 down_at 30 last_clean_interval [15,27) [v2:127.0.0.1:6810/41936,v1:127.0.0.1:6811/41936] [v2:127.0.0.1:6812/41936,v1:127.0.0.1:6813/41936] exists,up d1a5f547-90ee-411b-9aab-d214098efc62 2024-09-18T13:59:46.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T13:59:46.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T13:59:46.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T13:59:46.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:183: TEST_divergent: ceph pg dump pgs 2024-09-18T13:59:46.984 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T13:59:46.984 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-18T13:59:00.541745+0000 40'101 42:129 [0,2] 0 [0,2] 0 0'0 2024-09-18T13:58:04.104186+0000 0'0 2024-09-18T13:58:04.104186+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:37:01.991304+0000 0 0 2024-09-18T13:59:46.985 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T13:59:46.985 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T13:59:46.985 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T13:59:46.995 INFO:tasks.workunit.client.0.smithi047.stdout:wait for peering 2024-09-18T13:59:46.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:184: TEST_divergent: echo 'wait for peering' 2024-09-18T13:59:46.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:185: TEST_divergent: ceph pg dump pgs 2024-09-18T13:59:47.347 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T13:59:47.347 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-18T13:59:00.541745+0000 40'101 42:129 [0,2] 0 [0,2] 0 0'0 2024-09-18T13:58:04.104186+0000 0'0 2024-09-18T13:58:04.104186+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:37:01.991304+0000 0 0 2024-09-18T13:59:47.347 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T13:59:47.348 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T13:59:47.348 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T13:59:47.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:186: TEST_divergent: rados -p test put foo /etc/fstab 2024-09-18T13:59:47.473 INFO:tasks.workunit.client.0.smithi047.stdout:killing divergent 1 2024-09-18T13:59:47.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:188: TEST_divergent: echo 'killing divergent 1' 2024-09-18T13:59:47.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:189: TEST_divergent: ceph pg dump pgs 2024-09-18T13:59:47.829 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T13:59:47.829 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-18T13:59:00.541745+0000 40'101 42:129 [0,2] 0 [0,2] 0 0'0 2024-09-18T13:58:04.104186+0000 0'0 2024-09-18T13:58:04.104186+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:37:01.991304+0000 0 0 2024-09-18T13:59:47.829 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T13:59:47.829 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T13:59:47.829 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T13:59:47.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:190: TEST_divergent: kill_daemons td/divergent-priors KILL osd.1 2024-09-18T13:59:47.840 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T13:59:47.840 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T13:59:47.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T13:59:47.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T13:59:47.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T13:59:47.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T13:59:47.946 INFO:tasks.workunit.client.0.smithi047.stdout:reviving divergent 1 2024-09-18T13:59:47.946 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:192: TEST_divergent: echo 'reviving divergent 1' 2024-09-18T13:59:47.946 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:193: TEST_divergent: ceph pg dump pgs 2024-09-18T13:59:48.305 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T13:59:48.305 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 101 1 1 0 0 48454 0 0 18 84 18 active+recovering+degraded 2024-09-18T13:59:45.969235+0000 46'102 46:138 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T13:58:04.104186+0000 0'0 2024-09-18T13:58:04.104186+0000 0 0 periodic scrub scheduled @ 2024-09-19T18:37:31.002910+0000 0 0 2024-09-18T13:59:48.305 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T13:59:48.305 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T13:59:48.305 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T13:59:48.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:194: TEST_divergent: activate_osd td/divergent-priors 1 2024-09-18T13:59:48.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-18T13:59:48.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T13:59:48.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T13:59:48.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T13:59:48.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-09-18T13:59:48.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T13:59:48.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T13:59:48.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T13:59:48.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T13:59:48.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-18T13:59:48.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-18T13:59:48.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T13:59:48.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T13:59:48.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-18T13:59:48.319 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T13:59:48.319 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T13:59:48.319 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T13:59:48.319 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T13:59:48.319 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T13:59:48.320 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T13:59:48.320 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T13:59:48.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T13:59:48.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T13:59:48.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-18T13:59:48.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-18T13:59:48.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T13:59:48.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T13:59:48.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T13:59:48.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T13:59:48.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T13:59:48.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T13:59:48.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-09-18T13:59:48.321 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T13:59:48.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T13:59:48.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T13:59:48.322 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-09-18T13:59:48.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T13:59:48.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T13:59:48.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T13:59:48.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T13:59:48.349 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:59:48.348+0000 7f6317e34440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:59:48.351 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:59:48.351+0000 7f6317e34440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:59:48.352 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:59:48.352+0000 7f6317e34440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T13:59:48.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T13:59:48.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T13:59:48.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T13:59:48.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T13:59:48.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T13:59:48.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T13:59:48.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T13:59:48.737 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T13:59:48.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T13:59:48.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T13:59:48.873 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:59:48.872+0000 7f6317e34440 -1 Falling back to public interface 2024-09-18T13:59:49.145 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 45 up_thru 45 down_at 29 last_clean_interval [10,27) [v2:127.0.0.1:6818/43589,v1:127.0.0.1:6819/43589] [v2:127.0.0.1:6820/43589,v1:127.0.0.1:6821/43589] exists,up 3353f33a-e483-4b8b-976d-f7b4acd36b36 2024-09-18T13:59:49.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T13:59:49.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T13:59:49.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T13:59:49.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:196: TEST_divergent: sleep 20 2024-09-18T13:59:51.791 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T13:59:51.790+0000 7f6317e34440 -1 osd.1 46 log_to_monitors true 2024-09-18T14:00:09.149 INFO:tasks.workunit.client.0.smithi047.stdout:allowing recovery 2024-09-18T14:00:09.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:198: TEST_divergent: echo 'allowing recovery' 2024-09-18T14:00:09.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:199: TEST_divergent: ceph pg dump pgs 2024-09-18T14:00:09.510 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:00:09.511 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 101 0 0 0 0 48454 0 0 18 84 18 active+clean 2024-09-18T13:59:53.291016+0000 46'102 50:152 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T13:58:04.104186+0000 0'0 2024-09-18T13:58:04.104186+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:22:26.903605+0000 0 0 2024-09-18T14:00:09.511 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:00:09.511 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:00:09.511 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:00:09.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-09-18T14:00:09.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-18T14:00:09.628 INFO:tasks.workunit.client.0.smithi047.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-18T14:00:09.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-09-18T14:00:09.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-18T14:00:09.745 INFO:tasks.workunit.client.0.smithi047.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-18T14:00:09.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-09-18T14:00:09.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-18T14:00:09.861 INFO:tasks.workunit.client.0.smithi047.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-18T14:00:09.870 INFO:tasks.workunit.client.0.smithi047.stdout:reading divergent objects 2024-09-18T14:00:09.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:206: TEST_divergent: echo 'reading divergent objects' 2024-09-18T14:00:09.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:207: TEST_divergent: ceph pg dump pgs 2024-09-18T14:00:10.223 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:00:10.224 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 101 0 0 0 0 48454 0 0 18 84 18 active+clean 2024-09-18T13:59:53.291016+0000 46'102 50:152 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T13:58:04.104186+0000 0'0 2024-09-18T13:58:04.104186+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:22:26.903605+0000 0 0 2024-09-18T14:00:10.224 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:00:10.224 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:00:10.224 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:00:10.234 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: expr 2 + 2 2024-09-18T14:00:10.235 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: seq 1 4 2024-09-18T14:00:10.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-18T14:00:10.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_1 td/divergent-priors/existing 2024-09-18T14:00:10.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-18T14:00:10.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_2 td/divergent-priors/existing 2024-09-18T14:00:10.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-18T14:00:10.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_3 td/divergent-priors/existing 2024-09-18T14:00:10.389 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-18T14:00:10.389 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_4 td/divergent-priors/existing 2024-09-18T14:00:10.420 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:212: TEST_divergent: rm -f td/divergent-priors/existing 2024-09-18T14:00:10.421 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: find td/divergent-priors -name '*osd*log' 2024-09-18T14:00:10.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-18T14:00:10.427 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T13:59:45.775+0000 7fda123c7640 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 27'101 (27'100) modify 1:c156033e:::existing_100:head by client.4612.0:1 2024-09-18T13:58:34.438229+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-18T14:00:10.427 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T13:59:45.775+0000 7fda123c7640 20 _merge_object_divergent_entries: keeping 27'101 (27'100) modify 1:c156033e:::existing_100:head by client.4612.0:1 2024-09-18T13:58:34.438229+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-18T14:00:10.427 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T13:59:45.775+0000 7fda123c7640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-18T14:00:10.428 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T13:59:45.775+0000 7fda123c7640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 27'100 first_divergent_update: 27'101 last_divergent_update: 27'101 2024-09-18T14:00:10.428 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T13:59:45.775+0000 7fda123c7640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2024-09-18T14:00:10.428 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T13:59:45.775+0000 7fda123c7640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2024-09-18T14:00:10.428 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T13:59:45.775+0000 7fda123c7640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 27'101 2024-09-18T14:00:10.428 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T13:59:45.775+0000 7fda123c7640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 27'101 (27'100) modify 1:c156033e:::existing_100:head by client.4612.0:1 2024-09-18T13:58:34.438229+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-18T14:00:10.428 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T13:59:45.775+0000 7fda123c7640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2024-09-18T14:00:10.429 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: find td/divergent-priors -name '*osd*log' 2024-09-18T14:00:10.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: grep -q '_merge_object_divergent_entries.*cannot roll back, removing and adding to missing' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-18T14:00:10.432 INFO:tasks.workunit.client.0.smithi047.stdout:success 2024-09-18T14:00:10.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:221: TEST_divergent: echo success 2024-09-18T14:00:10.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:223: TEST_divergent: delete_pool test 2024-09-18T14:00:10.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-18T14:00:10.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-18T14:00:11.083 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test' does not exist 2024-09-18T14:00:11.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:224: TEST_divergent: kill_daemons td/divergent-priors 2024-09-18T14:00:11.094 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:00:11.094 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:00:11.095 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:00:11.095 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:00:11.095 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:00:11.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:00:11.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-18T14:00:11.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-18T14:00:11.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:00:11.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-18T14:00:11.405 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:00:11.405 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:00:11.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:00:11.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:00:11.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:00:11.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:00:11.410 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:00:11.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:00:11.411 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:00:11.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:00:11.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:00:11.412 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:00:11.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:00:11.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:00:11.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:00:11.413 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:00:11.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:00:11.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:00:11.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-18T14:00:11.435 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:00:11.435 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:00:11.435 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:00:11.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.32609 2024-09-18T14:00:11.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:00:11.437 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:00:11.437 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-18T14:00:11.437 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-18T14:00:11.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-09-18T14:00:11.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-09-18T14:00:11.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-18T14:00:11.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:00:11.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-18T14:00:11.438 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:00:11.438 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:00:11.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:00:11.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:00:11.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:00:11.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:00:11.439 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:00:11.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:00:11.440 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:00:11.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:00:11.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:00:11.441 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:00:11.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:00:11.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:00:11.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:00:11.442 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:00:11.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:00:11.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:00:11.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-18T14:00:11.445 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:00:11.445 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:00:11.445 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:00:11.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.32609 2024-09-18T14:00:11.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:00:11.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:00:11.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-09-18T14:00:11.447 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T14:00:11.447 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:00:11.447 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:00:11.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.32609 2024-09-18T14:00:11.449 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T14:00:11.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T14:00:11.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T14:00:11.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-18T14:00:11.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_2 td/divergent-priors 2024-09-18T14:00:11.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:421: TEST_divergent_2: local dir=td/divergent-priors 2024-09-18T14:00:11.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:424: TEST_divergent_2: local dummyfile=/etc/fstab 2024-09-18T14:00:11.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:425: TEST_divergent_2: local dummyfile2=/etc/resolv.conf 2024-09-18T14:00:11.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:427: TEST_divergent_2: local num_osds=3 2024-09-18T14:00:11.450 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: expr 3 - 1 2024-09-18T14:00:11.450 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: seq 0 2 2024-09-18T14:00:11.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: local 'osds=0 2024-09-18T14:00:11.451 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T14:00:11.451 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T14:00:11.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:429: TEST_divergent_2: run_mon td/divergent-priors a 2024-09-18T14:00:11.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-09-18T14:00:11.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T14:00:11.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T14:00:11.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T14:00:11.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-09-18T14:00:11.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-18T14:00:11.800 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T14:00:11.800 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:00:11.800 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:00:11.800 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:00:11.800 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:00:11.800 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:00:11.800 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:00:11.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T14:00:12.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T14:00:12.087 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T14:00:12.087 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:00:12.087 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:00:12.087 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T14:00:12.087 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T14:00:12.087 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:00:12.087 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:00:12.087 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:00:12.087 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:00:12.088 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:00:12.088 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:00:12.088 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32609/ceph-mon.a.asok 2024-09-18T14:00:12.088 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:00:12.088 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.32609/ceph-mon.a.asok config get fsid 2024-09-18T14:00:12.164 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T14:00:12.164 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:00:12.164 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:00:12.165 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T14:00:12.165 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T14:00:12.165 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:00:12.165 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:00:12.165 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:00:12.165 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:00:12.166 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:00:12.166 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:00:12.166 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32609/ceph-mon.a.asok 2024-09-18T14:00:12.166 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:00:12.166 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.32609/ceph-mon.a.asok config get mon_host 2024-09-18T14:00:12.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:430: TEST_divergent_2: run_mgr td/divergent-priors x 2024-09-18T14:00:12.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-09-18T14:00:12.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T14:00:12.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T14:00:12.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T14:00:12.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-09-18T14:00:12.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T14:00:12.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T14:00:12.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:00:12.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:00:12.724 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:00:12.724 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:00:12.724 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:00:12.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:00:12.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:00:12.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:00:12.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-09-18T14:00:12.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 0 2024-09-18T14:00:12.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-18T14:00:12.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:00:12.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T14:00:12.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:00:12.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-09-18T14:00:12.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T14:00:12.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:00:12.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:00:12.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:00:12.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-18T14:00:12.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-18T14:00:12.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:00:12.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:00:12.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-18T14:00:12.757 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:00:12.757 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:00:12.757 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:00:12.757 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:00:12.758 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:00:12.758 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:00:12.758 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:00:12.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:00:12.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:00:12.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:00:12.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:00:12.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-18T14:00:12.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-18T14:00:12.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:00:12.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:00:12.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:00:12.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:00:12.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:00:12.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:00:12.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-09-18T14:00:12.759 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:00:12.760 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=df87ad03-46d9-487e-8f7a-aab30f82a1bd 2024-09-18T14:00:12.760 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 df87ad03-46d9-487e-8f7a-aab30f82a1bd 2024-09-18T14:00:12.760 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 df87ad03-46d9-487e-8f7a-aab30f82a1bd' 2024-09-18T14:00:12.760 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:00:12.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDs3Opm/xRWLhAA0n0dcR4+NmKtbYvfc5NyTQ== 2024-09-18T14:00:12.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDs3Opm/xRWLhAA0n0dcR4+NmKtbYvfc5NyTQ=="}' 2024-09-18T14:00:12.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new df87ad03-46d9-487e-8f7a-aab30f82a1bd -i td/divergent-priors/0/new.json 2024-09-18T14:00:13.066 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:00:13.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-09-18T14:00:13.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDs3Opm/xRWLhAA0n0dcR4+NmKtbYvfc5NyTQ== --osd-uuid df87ad03-46d9-487e-8f7a-aab30f82a1bd 2024-09-18T14:00:13.109 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:00:13.108+0000 7fe19b437440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:00:13.110 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:00:13.110+0000 7fe19b437440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:00:13.111 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:00:13.111+0000 7fe19b437440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:00:13.112 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:00:13.111+0000 7fe19b437440 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-18T14:00:13.112 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:00:13.111+0000 7fe19b437440 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-18T14:00:13.112 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:00:13.111+0000 7fe19b437440 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-18T14:00:13.151 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:00:13.150+0000 7fe19b437440 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-18T14:00:18.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-18T14:00:18.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:00:18.352 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T14:00:18.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T14:00:18.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:00:18.875 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T14:00:18.875 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T14:00:18.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:00:18.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:00:18.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:00:18.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:00:18.906 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:00:18.906+0000 7f986be08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:00:18.908 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:00:18.908+0000 7f986be08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:00:18.909 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:00:18.909+0000 7f986be08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:00:19.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T14:00:19.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:00:19.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:00:19.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:00:19.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:00:19.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:00:19.272 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:00:19.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:00:19.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:00:19.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:00:19.428 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:00:19.428+0000 7f986be08440 -1 Falling back to public interface 2024-09-18T14:00:19.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:00:20.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:00:20.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:00:20.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:00:20.682 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:00:20.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:00:20.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:00:21.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:00:21.779 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:00:21.778+0000 7f986be08440 -1 osd.0 0 log_to_monitors true 2024-09-18T14:00:22.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:00:22.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:00:22.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:00:22.108 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:00:22.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:00:22.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:00:22.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:00:23.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:00:23.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:00:23.659 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:00:23.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:00:23.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:00:23.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:00:24.052 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:00:25.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:00:25.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:00:25.054 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:00:25.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:00:25.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:00:25.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:00:25.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:00:26.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:00:26.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:00:26.518 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:00:26.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:00:26.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:00:26.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:00:26.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:00:27.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:00:27.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:00:27.941 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:00:27.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:00:27.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:00:27.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:00:28.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:00:29.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:00:29.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:00:29.360 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:00:29.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:00:29.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:00:29.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:00:29.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:00:30.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:00:30.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:00:30.779 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:00:30.780 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:00:30.780 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:00:30.780 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:00:31.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:00:32.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:00:32.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:00:32.166 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:00:32.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:00:32.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:00:32.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:00:32.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:00:33.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:00:33.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:00:33.723 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:00:33.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:00:33.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:00:33.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:00:34.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:00:35.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:00:35.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:00:35.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:00:35.349 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:00:35.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:00:35.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:00:36.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:00:37.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:00:37.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:00:37.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:00:37.280 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:00:37.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:00:37.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:00:38.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:00:39.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:00:39.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:00:39.125 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:00:39.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:00:39.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:00:39.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:00:39.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:00:40.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:00:40.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:00:40.517 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:00:40.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:00:40.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:00:40.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:00:41.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:00:42.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:00:42.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:00:42.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:00:42.721 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:00:42.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:00:42.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:00:43.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:00:44.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:00:44.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:00:44.468 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:00:44.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:00:44.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:00:44.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:00:44.885 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/46167,v1:127.0.0.1:6803/46167] [v2:127.0.0.1:6804/46167,v1:127.0.0.1:6805/46167] exists,up df87ad03-46d9-487e-8f7a-aab30f82a1bd 2024-09-18T14:00:44.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:00:44.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:00:44.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:00:44.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-09-18T14:00:44.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 1 2024-09-18T14:00:44.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-18T14:00:44.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:00:44.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T14:00:44.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:00:44.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-09-18T14:00:44.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T14:00:44.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:00:44.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:00:44.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:00:44.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-18T14:00:44.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-18T14:00:44.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:00:44.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:00:44.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-18T14:00:44.890 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:00:44.890 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:00:44.890 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:00:44.890 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:00:44.890 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:00:44.890 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:00:44.890 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:00:44.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:00:44.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:00:44.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:00:44.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:00:44.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-18T14:00:44.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-18T14:00:44.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:00:44.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:00:44.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:00:44.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:00:44.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:00:44.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:00:44.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-09-18T14:00:44.891 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:00:44.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b8177855-da06-4ebd-9d12-538bef0d43b8 2024-09-18T14:00:44.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b8177855-da06-4ebd-9d12-538bef0d43b8' 2024-09-18T14:00:44.892 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 b8177855-da06-4ebd-9d12-538bef0d43b8 2024-09-18T14:00:44.892 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:00:44.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAM3epmpi0iNhAAUoeiE0FKMHH3LCFoMD0rFQ== 2024-09-18T14:00:44.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAM3epmpi0iNhAAUoeiE0FKMHH3LCFoMD0rFQ=="}' 2024-09-18T14:00:44.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b8177855-da06-4ebd-9d12-538bef0d43b8 -i td/divergent-priors/1/new.json 2024-09-18T14:00:45.502 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:00:45.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-09-18T14:00:45.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAM3epmpi0iNhAAUoeiE0FKMHH3LCFoMD0rFQ== --osd-uuid b8177855-da06-4ebd-9d12-538bef0d43b8 2024-09-18T14:00:45.545 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:00:45.545+0000 7f0f5200c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:00:45.547 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:00:45.547+0000 7f0f5200c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:00:45.549 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:00:45.548+0000 7f0f5200c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:00:45.549 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:00:45.548+0000 7f0f5200c440 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-18T14:00:45.549 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:00:45.548+0000 7f0f5200c440 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-18T14:00:45.549 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:00:45.548+0000 7f0f5200c440 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-18T14:00:45.626 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:00:45.625+0000 7f0f5200c440 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-18T14:00:51.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-18T14:00:51.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:00:51.206 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T14:00:51.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T14:00:51.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:00:51.731 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T14:00:51.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T14:00:51.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:00:51.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:00:51.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:00:51.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:00:51.762 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:00:51.762+0000 7fab58d1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:00:51.773 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:00:51.773+0000 7fab58d1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:00:51.774 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:00:51.774+0000 7fab58d1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:00:52.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T14:00:52.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:00:52.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:00:52.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:00:52.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:00:52.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:00:52.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:00:52.131 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:00:52.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:00:52.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:00:52.305 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:00:52.305+0000 7fab58d1d440 -1 Falling back to public interface 2024-09-18T14:00:52.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:00:53.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:00:53.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:00:53.547 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:00:53.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:00:53.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:00:53.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:00:53.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:00:54.812 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:00:54.812+0000 7fab58d1d440 -1 osd.1 0 log_to_monitors true 2024-09-18T14:00:54.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:00:54.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:00:54.929 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:00:54.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:00:54.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:00:54.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:00:55.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:00:56.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:00:56.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:00:56.545 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:00:56.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:00:56.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:00:56.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:00:57.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:00:58.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:00:58.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:00:58.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:00:58.122 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:00:58.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:00:58.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:00:58.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:00:59.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:00:59.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:00:59.697 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:00:59.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:00:59.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:00:59.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:01:00.095 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:01:01.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:01:01.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:01:01.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:01:01.090 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:01:01.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:01:01.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:01:01.648 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:01:02.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:01:02.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:01:02.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:01:02.650 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:01:02.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:01:02.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:01:03.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:01:04.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:01:04.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:01:04.124 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:01:04.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:01:04.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:01:04.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:01:04.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:01:05.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:01:05.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:01:05.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:01:05.713 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:01:05.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:01:05.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:01:06.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:01:07.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:01:07.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:01:07.109 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:01:07.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:01:07.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:01:07.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:01:08.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:01:09.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:01:09.316 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:01:09.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:01:09.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:01:09.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:01:09.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:01:10.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:01:11.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:01:11.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:01:11.044 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:01:11.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:01:11.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:01:11.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:01:11.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:01:12.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:01:12.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:01:12.554 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:01:12.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:01:12.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:01:12.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:01:13.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:01:14.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:01:14.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:01:14.025 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:01:14.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:01:14.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:01:14.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:01:14.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:01:15.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:01:15.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:01:15.787 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:01:15.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:01:15.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:01:15.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:01:16.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:01:17.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:01:17.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:01:17.569 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:01:17.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:01:17.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:01:17.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:01:17.957 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/47896,v1:127.0.0.1:6811/47896] [v2:127.0.0.1:6812/47896,v1:127.0.0.1:6813/47896] exists,up b8177855-da06-4ebd-9d12-538bef0d43b8 2024-09-18T14:01:17.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:01:17.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:01:17.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:01:17.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-09-18T14:01:17.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 2 2024-09-18T14:01:17.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-18T14:01:17.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:01:17.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T14:01:17.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:01:17.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-09-18T14:01:17.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T14:01:17.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:01:17.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:01:17.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:01:17.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-18T14:01:17.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-18T14:01:17.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:01:17.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:01:17.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-18T14:01:17.963 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:01:17.963 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:01:17.963 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:01:17.963 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:01:17.963 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:01:17.963 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:01:17.963 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:01:17.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:01:17.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:01:17.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:01:17.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:01:17.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-18T14:01:17.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-18T14:01:17.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:01:17.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:01:17.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:01:17.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:01:17.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:01:17.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:01:17.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-09-18T14:01:17.964 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:01:17.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dec93258-345a-406b-85ef-8316978cb8b7 2024-09-18T14:01:17.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 dec93258-345a-406b-85ef-8316978cb8b7' 2024-09-18T14:01:17.965 INFO:tasks.workunit.client.0.smithi047.stdout:add osd2 dec93258-345a-406b-85ef-8316978cb8b7 2024-09-18T14:01:17.965 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:01:17.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAt3epmFgt7OhAAPlhYjrkClZp9p2OjaXzjKg== 2024-09-18T14:01:17.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAt3epmFgt7OhAAPlhYjrkClZp9p2OjaXzjKg=="}' 2024-09-18T14:01:17.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dec93258-345a-406b-85ef-8316978cb8b7 -i td/divergent-priors/2/new.json 2024-09-18T14:01:18.522 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:01:18.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-09-18T14:01:18.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAt3epmFgt7OhAAPlhYjrkClZp9p2OjaXzjKg== --osd-uuid dec93258-345a-406b-85ef-8316978cb8b7 2024-09-18T14:01:18.566 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:01:18.566+0000 7fddaea14440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:01:18.581 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:01:18.581+0000 7fddaea14440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:01:18.582 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:01:18.582+0000 7fddaea14440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:01:18.583 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:01:18.582+0000 7fddaea14440 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-18T14:01:18.583 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:01:18.582+0000 7fddaea14440 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-18T14:01:18.583 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:01:18.582+0000 7fddaea14440 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-18T14:01:18.665 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:01:18.665+0000 7fddaea14440 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-18T14:01:24.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-18T14:01:24.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:01:24.497 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd2 key to auth repository 2024-09-18T14:01:24.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T14:01:24.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:01:25.024 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T14:01:25.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T14:01:25.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:01:25.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:01:25.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:01:25.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:01:25.055 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:01:25.054+0000 7ff58711d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:01:25.066 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:01:25.066+0000 7ff58711d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:01:25.067 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:01:25.067+0000 7ff58711d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:01:25.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T14:01:25.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:01:25.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T14:01:25.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:01:25.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:01:25.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:01:25.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:01:25.397 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:01:25.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:01:25.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:01:25.582 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:01:25.582+0000 7ff58711d440 -1 Falling back to public interface 2024-09-18T14:01:25.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:01:26.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:01:26.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:01:26.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:01:26.797 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:01:26.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:01:26.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:01:27.247 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:01:28.045 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:01:28.044+0000 7ff58711d440 -1 osd.2 0 log_to_monitors true 2024-09-18T14:01:28.248 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:01:28.249 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:01:28.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:01:28.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:01:28.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:01:28.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:01:28.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:01:29.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:01:29.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:01:29.781 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:01:29.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:01:29.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:01:29.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:01:30.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:01:31.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:01:31.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:01:31.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:01:31.170 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:01:31.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:01:31.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:01:31.826 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:01:32.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:01:32.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:01:32.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:01:32.828 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:01:32.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:01:32.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:01:33.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:01:34.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:01:34.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:01:34.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:01:34.278 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:01:34.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:01:34.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:01:34.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:01:35.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:01:35.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:01:35.814 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:01:35.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:01:35.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:01:35.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:01:36.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:01:37.197 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:01:37.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:01:37.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:01:37.198 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:01:37.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:01:37.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:01:37.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:01:38.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:01:38.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:01:38.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:01:38.796 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:01:38.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:01:38.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:01:39.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:01:40.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:01:40.209 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:01:40.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:01:40.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:01:40.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:01:40.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:01:40.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:01:41.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:01:41.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:01:41.723 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:01:41.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:01:41.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:01:41.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:01:42.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:01:43.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:01:43.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:01:43.382 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:01:43.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:01:43.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:01:43.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:01:44.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:01:45.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:01:45.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:01:45.165 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:01:45.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:01:45.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:01:45.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:01:46.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:01:47.285 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:01:47.285 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:01:47.285 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:01:47.285 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:01:47.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:01:47.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:01:47.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:01:48.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:01:48.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:01:48.912 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:01:48.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:01:48.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:01:48.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:01:49.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:01:50.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:01:50.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:01:50.654 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:01:50.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:01:50.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:01:50.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:01:51.066 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/49641,v1:127.0.0.1:6819/49641] [v2:127.0.0.1:6820/49641,v1:127.0.0.1:6821/49641] exists,up dec93258-345a-406b-85ef-8316978cb8b7 2024-09-18T14:01:51.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:01:51.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:01:51.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:01:51.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:436: TEST_divergent_2: ceph osd set noout 2024-09-18T14:01:51.730 INFO:tasks.workunit.client.0.smithi047.stderr:noout is set 2024-09-18T14:01:51.741 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:437: TEST_divergent_2: ceph osd set noin 2024-09-18T14:01:52.534 INFO:tasks.workunit.client.0.smithi047.stderr:noin is set 2024-09-18T14:01:52.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:438: TEST_divergent_2: ceph osd set nodown 2024-09-18T14:01:53.433 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T14:01:53.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:439: TEST_divergent_2: create_pool test 1 1 2024-09-18T14:01:53.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-18T14:01:54.135 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test' already exists 2024-09-18T14:01:54.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T14:01:55.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:440: TEST_divergent_2: ceph osd pool set test size 3 2024-09-18T14:01:55.839 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 1 size to 3 2024-09-18T14:01:55.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:441: TEST_divergent_2: ceph osd pool set test min_size 2 2024-09-18T14:01:56.681 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 1 min_size to 2 2024-09-18T14:01:56.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:443: TEST_divergent_2: flush_pg_stats 2024-09-18T14:01:56.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:01:56.692 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:01:57.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:01:57.137 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T14:01:57.137 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T14:01:57.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:01:57.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:01:57.137 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:01:57.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836500 2024-09-18T14:01:57.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836500 2024-09-18T14:01:57.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500' 2024-09-18T14:01:57.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:01:57.253 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:01:57.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672972 2024-09-18T14:01:57.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672972 2024-09-18T14:01:57.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500 1-42949672972' 2024-09-18T14:01:57.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:01:57.369 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:01:57.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509446 2024-09-18T14:01:57.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509446 2024-09-18T14:01:57.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500 1-42949672972 2-64424509446' 2024-09-18T14:01:57.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:01:57.486 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836500 2024-09-18T14:01:57.486 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:01:57.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:01:57.487 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836500 2024-09-18T14:01:57.487 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:01:57.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836500 2024-09-18T14:01:57.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-09-18T14:01:57.488 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836500 2024-09-18T14:01:57.489 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:01:57.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836498 -lt 21474836500 2024-09-18T14:01:57.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:01:58.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:01:58.867 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:01:59.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836500 -lt 21474836500 2024-09-18T14:01:59.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:01:59.284 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672972 2024-09-18T14:01:59.284 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:01:59.285 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:01:59.286 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672972 2024-09-18T14:01:59.286 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:01:59.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672972 2024-09-18T14:01:59.287 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672972 2024-09-18T14:01:59.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-09-18T14:01:59.287 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:01:59.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672973 -lt 42949672972 2024-09-18T14:01:59.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:01:59.727 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509446 2024-09-18T14:01:59.727 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:01:59.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:01:59.728 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509446 2024-09-18T14:01:59.729 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:01:59.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509446 2024-09-18T14:01:59.730 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509446 2024-09-18T14:01:59.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-09-18T14:01:59.730 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:02:00.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509446 -lt 64424509446 2024-09-18T14:02:00.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:444: TEST_divergent_2: wait_for_clean 2024-09-18T14:02:00.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:02:00.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:02:00.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:02:00.147 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:02:00.147 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:02:00.147 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:02:00.147 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:02:00.147 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:02:00.147 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:02:00.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T14:02:00.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:02:00.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:02:00.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:02:00.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:02:00.237 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:02:00.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:02:00.620 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T14:02:00.620 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T14:02:00.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:02:00.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:02:00.620 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:02:00.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836503 2024-09-18T14:02:00.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836503 2024-09-18T14:02:00.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836503' 2024-09-18T14:02:00.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:02:00.736 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:02:00.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672975 2024-09-18T14:02:00.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672975 2024-09-18T14:02:00.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836503 1-42949672975' 2024-09-18T14:02:00.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:02:00.865 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:02:00.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509448 2024-09-18T14:02:00.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509448 2024-09-18T14:02:00.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836503 1-42949672975 2-64424509448' 2024-09-18T14:02:00.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:02:00.998 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836503 2024-09-18T14:02:00.998 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:02:00.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:02:00.999 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836503 2024-09-18T14:02:00.999 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:02:01.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836503 2024-09-18T14:02:01.000 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836503 2024-09-18T14:02:01.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-09-18T14:02:01.001 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:02:01.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836501 -lt 21474836503 2024-09-18T14:02:01.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:02:02.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:02:02.401 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:02:02.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836503 -lt 21474836503 2024-09-18T14:02:02.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:02:02.811 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672975 2024-09-18T14:02:02.811 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:02:02.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:02:02.812 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672975 2024-09-18T14:02:02.812 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:02:02.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672975 2024-09-18T14:02:02.813 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672975 2024-09-18T14:02:02.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-09-18T14:02:02.814 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:02:03.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672975 -lt 42949672975 2024-09-18T14:02:03.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:02:03.229 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509448 2024-09-18T14:02:03.229 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:02:03.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:02:03.230 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509448 2024-09-18T14:02:03.230 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:02:03.231 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509448 2024-09-18T14:02:03.231 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-09-18T14:02:03.232 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509448 2024-09-18T14:02:03.232 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:02:03.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509449 -lt 64424509448 2024-09-18T14:02:03.646 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:02:03.646 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:02:03.646 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:02:04.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T14:02:04.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:02:04.223 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:02:04.223 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:02:04.223 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:02:04.223 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:02:04.224 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:02:04.224 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:02:04.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T14:02:04.591 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:02:04.591 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:02:04.591 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:02:05.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T14:02:05.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:02:05.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:02:05.098 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: ceph pg dump pgs --format=json 2024-09-18T14:02:05.098 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: jq '.pg_stats[0].up_primary' 2024-09-18T14:02:05.484 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:02:05.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: local divergent=1 2024-09-18T14:02:05.495 INFO:tasks.workunit.client.0.smithi047.stdout:primary and soon to be divergent is 1 2024-09-18T14:02:05.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:448: TEST_divergent_2: echo 'primary and soon to be divergent is 1' 2024-09-18T14:02:05.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:449: TEST_divergent_2: ceph pg dump pgs 2024-09-18T14:02:05.845 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:02:05.846 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:01:54.589287+0000 0'0 28:13 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:01:53.783298+0000 0'0 2024-09-18T14:01:53.783298+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:14:37.645718+0000 0 0 2024-09-18T14:02:05.846 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:02:05.846 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:02:05.846 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:02:05.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:450: TEST_divergent_2: local non_divergent= 2024-09-18T14:02:05.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-09-18T14:02:05.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 0 = 1 ']' 2024-09-18T14:02:05.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0' 2024-09-18T14:02:05.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-09-18T14:02:05.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 1 = 1 ']' 2024-09-18T14:02:05.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:454: TEST_divergent_2: continue 2024-09-18T14:02:05.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-09-18T14:02:05.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 2 = 1 ']' 2024-09-18T14:02:05.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0 2' 2024-09-18T14:02:05.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:459: TEST_divergent_2: echo 'writing initial objects' 2024-09-18T14:02:05.858 INFO:tasks.workunit.client.0.smithi047.stdout:writing initial objects 2024-09-18T14:02:05.858 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: seq 1 100 2024-09-18T14:02:05.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:05.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_1 /etc/fstab 2024-09-18T14:02:06.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:06.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_2 /etc/fstab 2024-09-18T14:02:06.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:06.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_3 /etc/fstab 2024-09-18T14:02:06.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:06.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_4 /etc/fstab 2024-09-18T14:02:06.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:06.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_5 /etc/fstab 2024-09-18T14:02:06.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:06.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_6 /etc/fstab 2024-09-18T14:02:06.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:06.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_7 /etc/fstab 2024-09-18T14:02:06.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:06.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_8 /etc/fstab 2024-09-18T14:02:07.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:07.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_9 /etc/fstab 2024-09-18T14:02:07.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:07.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_10 /etc/fstab 2024-09-18T14:02:07.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:07.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_11 /etc/fstab 2024-09-18T14:02:07.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:07.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_12 /etc/fstab 2024-09-18T14:02:07.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:07.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_13 /etc/fstab 2024-09-18T14:02:07.760 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:07.761 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_14 /etc/fstab 2024-09-18T14:02:07.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:07.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_15 /etc/fstab 2024-09-18T14:02:08.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:08.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_16 /etc/fstab 2024-09-18T14:02:08.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:08.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_17 /etc/fstab 2024-09-18T14:02:08.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:08.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_18 /etc/fstab 2024-09-18T14:02:08.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:08.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_19 /etc/fstab 2024-09-18T14:02:08.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:08.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_20 /etc/fstab 2024-09-18T14:02:08.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:08.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_21 /etc/fstab 2024-09-18T14:02:08.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:08.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_22 /etc/fstab 2024-09-18T14:02:09.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:09.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_23 /etc/fstab 2024-09-18T14:02:09.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:09.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_24 /etc/fstab 2024-09-18T14:02:09.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:09.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_25 /etc/fstab 2024-09-18T14:02:09.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:09.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_26 /etc/fstab 2024-09-18T14:02:09.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:09.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_27 /etc/fstab 2024-09-18T14:02:09.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:09.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_28 /etc/fstab 2024-09-18T14:02:09.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:09.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_29 /etc/fstab 2024-09-18T14:02:10.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:10.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_30 /etc/fstab 2024-09-18T14:02:10.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:10.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_31 /etc/fstab 2024-09-18T14:02:10.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:10.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_32 /etc/fstab 2024-09-18T14:02:10.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:10.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_33 /etc/fstab 2024-09-18T14:02:10.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:10.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_34 /etc/fstab 2024-09-18T14:02:10.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:10.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_35 /etc/fstab 2024-09-18T14:02:11.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:11.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_36 /etc/fstab 2024-09-18T14:02:11.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:11.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_37 /etc/fstab 2024-09-18T14:02:11.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:11.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_38 /etc/fstab 2024-09-18T14:02:11.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:11.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_39 /etc/fstab 2024-09-18T14:02:11.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:11.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_40 /etc/fstab 2024-09-18T14:02:11.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:11.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_41 /etc/fstab 2024-09-18T14:02:11.907 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:11.907 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_42 /etc/fstab 2024-09-18T14:02:12.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:12.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_43 /etc/fstab 2024-09-18T14:02:12.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:12.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_44 /etc/fstab 2024-09-18T14:02:12.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:12.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_45 /etc/fstab 2024-09-18T14:02:12.527 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:12.527 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_46 /etc/fstab 2024-09-18T14:02:12.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:12.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_47 /etc/fstab 2024-09-18T14:02:12.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:12.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_48 /etc/fstab 2024-09-18T14:02:12.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:12.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_49 /etc/fstab 2024-09-18T14:02:13.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:13.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_50 /etc/fstab 2024-09-18T14:02:13.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:13.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_51 /etc/fstab 2024-09-18T14:02:13.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:13.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_52 /etc/fstab 2024-09-18T14:02:13.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:13.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_53 /etc/fstab 2024-09-18T14:02:13.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:13.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_54 /etc/fstab 2024-09-18T14:02:13.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:13.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_55 /etc/fstab 2024-09-18T14:02:13.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:13.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_56 /etc/fstab 2024-09-18T14:02:14.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:14.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_57 /etc/fstab 2024-09-18T14:02:14.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:14.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_58 /etc/fstab 2024-09-18T14:02:14.389 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:14.389 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_59 /etc/fstab 2024-09-18T14:02:14.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:14.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_60 /etc/fstab 2024-09-18T14:02:14.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:14.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_61 /etc/fstab 2024-09-18T14:02:14.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:14.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_62 /etc/fstab 2024-09-18T14:02:14.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:14.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_63 /etc/fstab 2024-09-18T14:02:15.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:15.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_64 /etc/fstab 2024-09-18T14:02:15.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:15.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_65 /etc/fstab 2024-09-18T14:02:15.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:15.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_66 /etc/fstab 2024-09-18T14:02:15.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:15.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_67 /etc/fstab 2024-09-18T14:02:15.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:15.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_68 /etc/fstab 2024-09-18T14:02:15.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:15.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_69 /etc/fstab 2024-09-18T14:02:15.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:15.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_70 /etc/fstab 2024-09-18T14:02:16.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:16.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_71 /etc/fstab 2024-09-18T14:02:16.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:16.242 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_72 /etc/fstab 2024-09-18T14:02:16.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:16.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_73 /etc/fstab 2024-09-18T14:02:16.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:16.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_74 /etc/fstab 2024-09-18T14:02:16.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:16.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_75 /etc/fstab 2024-09-18T14:02:16.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:16.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_76 /etc/fstab 2024-09-18T14:02:16.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:16.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_77 /etc/fstab 2024-09-18T14:02:17.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:17.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_78 /etc/fstab 2024-09-18T14:02:17.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:17.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_79 /etc/fstab 2024-09-18T14:02:17.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:17.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_80 /etc/fstab 2024-09-18T14:02:17.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:17.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_81 /etc/fstab 2024-09-18T14:02:17.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:17.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_82 /etc/fstab 2024-09-18T14:02:17.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:17.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_83 /etc/fstab 2024-09-18T14:02:17.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:17.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_84 /etc/fstab 2024-09-18T14:02:18.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:18.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_85 /etc/fstab 2024-09-18T14:02:18.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:18.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_86 /etc/fstab 2024-09-18T14:02:18.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:18.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_87 /etc/fstab 2024-09-18T14:02:18.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:18.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_88 /etc/fstab 2024-09-18T14:02:18.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:18.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_89 /etc/fstab 2024-09-18T14:02:18.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:18.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_90 /etc/fstab 2024-09-18T14:02:18.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:18.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_91 /etc/fstab 2024-09-18T14:02:19.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:19.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_92 /etc/fstab 2024-09-18T14:02:19.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:19.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_93 /etc/fstab 2024-09-18T14:02:19.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:19.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_94 /etc/fstab 2024-09-18T14:02:19.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:19.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_95 /etc/fstab 2024-09-18T14:02:19.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:19.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_96 /etc/fstab 2024-09-18T14:02:19.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:19.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_97 /etc/fstab 2024-09-18T14:02:19.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:19.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_98 /etc/fstab 2024-09-18T14:02:20.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:20.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_99 /etc/fstab 2024-09-18T14:02:20.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-18T14:02:20.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_100 /etc/fstab 2024-09-18T14:02:20.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-18T14:02:20.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: wait_for_clean 2024-09-18T14:02:20.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:02:20.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:02:20.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:02:20.426 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-18T14:02:20.427 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:02:20.427 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:02:20.427 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:02:20.427 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:02:20.427 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:02:20.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-18T14:02:20.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:02:20.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:02:20.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:02:20.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:02:20.486 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:02:20.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:02:20.903 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T14:02:20.903 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T14:02:20.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:02:20.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:02:20.903 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:02:21.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836509 2024-09-18T14:02:21.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836509 2024-09-18T14:02:21.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509' 2024-09-18T14:02:21.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:02:21.019 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:02:21.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672981 2024-09-18T14:02:21.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672981 2024-09-18T14:02:21.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509 1-42949672981' 2024-09-18T14:02:21.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:02:21.135 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:02:21.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509454 2024-09-18T14:02:21.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509454 2024-09-18T14:02:21.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509 1-42949672981 2-64424509454' 2024-09-18T14:02:21.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:02:21.250 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836509 2024-09-18T14:02:21.250 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:02:21.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:02:21.251 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836509 2024-09-18T14:02:21.251 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:02:21.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836509 2024-09-18T14:02:21.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-09-18T14:02:21.253 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836509 2024-09-18T14:02:21.253 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:02:21.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836507 -lt 21474836509 2024-09-18T14:02:21.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:02:22.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:02:22.629 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:02:23.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836509 -lt 21474836509 2024-09-18T14:02:23.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:02:23.065 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672981 2024-09-18T14:02:23.065 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:02:23.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:02:23.067 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672981 2024-09-18T14:02:23.067 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:02:23.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672981 2024-09-18T14:02:23.068 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672981 2024-09-18T14:02:23.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-09-18T14:02:23.068 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:02:23.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672981 -lt 42949672981 2024-09-18T14:02:23.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:02:23.525 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509454 2024-09-18T14:02:23.525 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:02:23.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:02:23.526 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509454 2024-09-18T14:02:23.526 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:02:23.527 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509454 2024-09-18T14:02:23.527 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509454 2024-09-18T14:02:23.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-09-18T14:02:23.528 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:02:23.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509455 -lt 64424509454 2024-09-18T14:02:23.945 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:02:23.945 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:02:23.945 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:02:24.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T14:02:24.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:02:24.450 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:02:24.450 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:02:24.450 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:02:24.451 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:02:24.451 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:02:24.451 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:02:24.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T14:02:24.829 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:02:24.829 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:02:24.829 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:02:25.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T14:02:25.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:02:25.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:02:25.336 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: get_pg test existing_1 2024-09-18T14:02:25.336 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-18T14:02:25.336 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-09-18T14:02:25.336 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-09-18T14:02:25.336 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T14:02:25.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: local pgid=1.0 2024-09-18T14:02:25.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:471: TEST_divergent_2: echo 'blackholing osds 0 2' 2024-09-18T14:02:25.735 INFO:tasks.workunit.client.0.smithi047.stdout:blackholing osds 0 2 2024-09-18T14:02:25.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:472: TEST_divergent_2: ceph pg dump pgs 2024-09-18T14:02:26.094 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:02:26.094 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 100 0 0 0 0 48300 0 0 16 84 16 active+clean 2024-09-18T14:01:54.589287+0000 29'100 29:113 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:01:53.783298+0000 0'0 2024-09-18T14:01:53.783298+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:14:37.645718+0000 0 0 2024-09-18T14:02:26.095 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:02:26.095 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:02:26.095 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:02:26.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2024-09-18T14:02:26.106 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.0 2024-09-18T14:02:26.106 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:02:26.106 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:02:26.106 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:02:26.106 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:02:26.106 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:02:26.106 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32609/ceph-osd.0.asok 2024-09-18T14:02:26.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2024-09-18T14:02:26.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.32609/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-18T14:02:26.181 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T14:02:26.181 INFO:tasks.workunit.client.0.smithi047.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-18T14:02:26.181 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T14:02:26.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2024-09-18T14:02:26.189 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.2 2024-09-18T14:02:26.189 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:02:26.189 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:02:26.189 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:02:26.190 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:02:26.190 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:02:26.190 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32609/ceph-osd.2.asok 2024-09-18T14:02:26.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2024-09-18T14:02:26.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.32609/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-18T14:02:26.263 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T14:02:26.263 INFO:tasks.workunit.client.0.smithi047.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-18T14:02:26.264 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T14:02:26.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:479: TEST_divergent_2: echo 'create new divergent objects' 2024-09-18T14:02:26.270 INFO:tasks.workunit.client.0.smithi047.stdout:create new divergent objects 2024-09-18T14:02:26.270 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: seq 1 2 2024-09-18T14:02:26.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-18T14:02:26.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-18T14:02:26.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_1 2024-09-18T14:02:26.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:485: TEST_divergent_2: echo 'writing divergent objects' 2024-09-18T14:02:26.272 INFO:tasks.workunit.client.0.smithi047.stdout:writing divergent objects 2024-09-18T14:02:26.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_2 2024-09-18T14:02:26.273 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: seq 1 2 2024-09-18T14:02:26.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-18T14:02:26.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-18T14:02:26.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_1 /etc/resolv.conf 2024-09-18T14:02:26.274 INFO:tasks.workunit.client.0.smithi047.stdout:remove divergent objects 2024-09-18T14:02:26.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:491: TEST_divergent_2: echo 'remove divergent objects' 2024-09-18T14:02:26.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_2 /etc/resolv.conf 2024-09-18T14:02:26.274 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: seq 1 2 2024-09-18T14:02:26.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-18T14:02:26.275 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 1 + 2 2024-09-18T14:02:26.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=3 2024-09-18T14:02:26.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-18T14:02:26.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_3 2024-09-18T14:02:26.277 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 2 + 2 2024-09-18T14:02:26.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=4 2024-09-18T14:02:26.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:497: TEST_divergent_2: sleep 10 2024-09-18T14:02:26.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_4 2024-09-18T14:02:36.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:498: TEST_divergent_2: killall -9 rados 2024-09-18T14:02:36.287 INFO:tasks.workunit.client.0.smithi047.stderr:rados: no process found 2024-09-18T14:02:36.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:501: TEST_divergent_2: echo 'killing all the osds' 2024-09-18T14:02:36.287 INFO:tasks.workunit.client.0.smithi047.stdout:killing all the osds 2024-09-18T14:02:36.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:502: TEST_divergent_2: ceph pg dump pgs 2024-09-18T14:02:36.650 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:02:36.650 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 100 0 0 0 0 46676 0 0 16 90 16 active+clean 2024-09-18T14:01:54.589287+0000 29'106 29:119 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:01:53.783298+0000 0'0 2024-09-18T14:01:53.783298+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:14:37.645718+0000 0 0 2024-09-18T14:02:36.650 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:02:36.650 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:02:36.650 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:02:36.660 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:503: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd 2024-09-18T14:02:36.661 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:02:36.661 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:02:36.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:02:36.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:02:36.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:02:36.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:02:36.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-09-18T14:02:36.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.0 2024-09-18T14:02:37.286 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already down. 2024-09-18T14:02:37.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-09-18T14:02:37.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.1 2024-09-18T14:02:37.848 INFO:tasks.workunit.client.0.smithi047.stderr:osd.1 is already down. 2024-09-18T14:02:37.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-09-18T14:02:37.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.2 2024-09-18T14:02:38.414 INFO:tasks.workunit.client.0.smithi047.stderr:osd.2 is already down. 2024-09-18T14:02:38.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2024-09-18T14:02:38.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.0 2024-09-18T14:02:38.912 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already out. 2024-09-18T14:02:38.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2024-09-18T14:02:38.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.2 2024-09-18T14:02:39.419 INFO:tasks.workunit.client.0.smithi047.stderr:osd.2 is already out. 2024-09-18T14:02:39.430 INFO:tasks.workunit.client.0.smithi047.stdout:bringing up non_divergent 0 2 2024-09-18T14:02:39.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:514: TEST_divergent_2: echo 'bringing up non_divergent 0 2' 2024-09-18T14:02:39.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:515: TEST_divergent_2: ceph pg dump pgs 2024-09-18T14:02:39.780 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:02:39.780 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 100 0 0 0 0 46676 0 0 16 90 16 stale+active+clean 2024-09-18T14:01:54.589287+0000 29'106 29:119 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:01:53.783298+0000 0'0 2024-09-18T14:01:53.783298+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:14:37.645718+0000 0 0 2024-09-18T14:02:39.780 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:02:39.780 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:02:39.780 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:02:39.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2024-09-18T14:02:39.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 0 2024-09-18T14:02:39.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-18T14:02:39.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:02:39.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T14:02:39.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:02:39.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-09-18T14:02:39.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T14:02:39.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:02:39.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:02:39.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:02:39.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-18T14:02:39.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-18T14:02:39.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:02:39.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:02:39.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-18T14:02:39.794 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:02:39.794 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:02:39.794 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:02:39.794 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:02:39.794 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:02:39.794 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:02:39.794 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:02:39.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:02:39.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:02:39.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-18T14:02:39.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-18T14:02:39.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:02:39.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:02:39.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:02:39.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:02:39.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:02:39.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:02:39.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-09-18T14:02:39.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T14:02:39.797 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T14:02:39.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:02:39.798 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-09-18T14:02:39.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T14:02:39.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:02:39.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:02:39.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:02:39.823 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:02:39.823+0000 7fbb84408440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:02:39.825 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:02:39.825+0000 7fbb84408440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:02:39.826 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:02:39.826+0000 7fbb84408440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:02:40.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T14:02:40.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:02:40.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:02:40.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:02:40.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:02:40.202 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:02:40.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:02:40.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:02:40.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:02:40.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:02:40.340 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:02:40.340+0000 7fbb84408440 -1 Falling back to public interface 2024-09-18T14:02:40.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:02:41.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:02:41.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:02:41.720 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:02:41.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:02:41.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:02:41.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:02:42.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:02:43.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:02:43.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:02:43.171 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:02:43.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:02:43.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:02:43.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:02:43.247 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:02:43.247+0000 7fbb84408440 -1 osd.0 29 log_to_monitors true 2024-09-18T14:02:43.587 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/53956,v1:127.0.0.1:6803/53956] [v2:127.0.0.1:6804/53956,v1:127.0.0.1:6805/53956] exists,up df87ad03-46d9-487e-8f7a-aab30f82a1bd 2024-09-18T14:02:43.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:02:43.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:02:43.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:02:43.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2024-09-18T14:02:43.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 2 2024-09-18T14:02:43.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-18T14:02:43.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:02:43.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T14:02:43.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:02:43.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-09-18T14:02:43.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T14:02:43.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:02:43.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:02:43.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:02:43.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-18T14:02:43.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-18T14:02:43.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:02:43.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:02:43.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-18T14:02:43.592 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:02:43.592 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:02:43.592 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:02:43.592 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:02:43.592 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:02:43.592 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:02:43.592 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:02:43.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:02:43.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:02:43.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-18T14:02:43.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-18T14:02:43.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:02:43.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:02:43.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:02:43.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:02:43.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:02:43.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:02:43.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-09-18T14:02:43.593 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T14:02:43.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T14:02:43.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:02:43.594 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-09-18T14:02:43.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T14:02:43.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:02:43.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:02:43.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:02:43.621 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:02:43.620+0000 7f61b041c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:02:43.692 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:02:43.692+0000 7f61b041c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:02:43.693 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:02:43.693+0000 7f61b041c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:02:44.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T14:02:44.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:02:44.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T14:02:44.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:02:44.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:02:44.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:02:44.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:02:44.029 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:02:44.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:02:44.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:02:44.211 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:02:44.211+0000 7f61b041c440 -1 Falling back to public interface 2024-09-18T14:02:44.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:02:45.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:02:45.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:02:45.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:02:45.401 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:02:45.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:02:45.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:02:45.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:02:46.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:02:46.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:02:46.825 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:02:46.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:02:46.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:02:46.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:02:47.175 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:02:47.175+0000 7f61b041c440 -1 osd.2 29 log_to_monitors true 2024-09-18T14:02:47.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:02:48.355 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:02:48.355 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:02:48.355 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:02:48.355 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:02:48.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:02:48.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:02:48.798 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/54526,v1:127.0.0.1:6811/54526] [v2:127.0.0.1:6812/54526,v1:127.0.0.1:6813/54526] exists,up dec93258-345a-406b-85ef-8316978cb8b7 2024-09-18T14:02:48.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:02:48.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:02:48.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:02:48.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2024-09-18T14:02:48.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.0 2024-09-18T14:02:49.364 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already in. 2024-09-18T14:02:49.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2024-09-18T14:02:49.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.2 2024-09-18T14:02:49.991 INFO:tasks.workunit.client.0.smithi047.stderr:osd.2 is already in. 2024-09-18T14:02:50.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-18T14:02:50.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: wait_for_clean 2024-09-18T14:02:50.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:02:50.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:02:50.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:02:50.002 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-18T14:02:50.003 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:02:50.003 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:02:50.003 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:02:50.003 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:02:50.003 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:02:50.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-18T14:02:50.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:02:50.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:02:50.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:02:50.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:02:50.059 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:02:50.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:02:50.480 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T14:02:50.480 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T14:02:50.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:02:50.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:02:50.480 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:02:50.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=154618822660 2024-09-18T14:02:50.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 154618822660 2024-09-18T14:02:50.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-154618822660' 2024-09-18T14:02:50.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:02:50.621 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:02:50.706 INFO:tasks.workunit.client.0.smithi047.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-18T14:02:50.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq= 2024-09-18T14:02:50.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z '' 2024-09-18T14:02:50.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: flush_pg_stats: continue 2024-09-18T14:02:50.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:02:50.708 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:02:50.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=163208757251 2024-09-18T14:02:50.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 163208757251 2024-09-18T14:02:50.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-154618822660 2-163208757251' 2024-09-18T14:02:50.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:02:50.824 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-154618822660 2024-09-18T14:02:50.825 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:02:50.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:02:50.826 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-154618822660 2024-09-18T14:02:50.826 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:02:50.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=154618822660 2024-09-18T14:02:50.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 154618822660' 2024-09-18T14:02:50.827 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 154618822660 2024-09-18T14:02:50.827 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:02:51.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 154618822658 -lt 154618822660 2024-09-18T14:02:51.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:02:52.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:02:52.241 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:02:52.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 154618822660 -lt 154618822660 2024-09-18T14:02:52.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:02:52.626 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-163208757251 2024-09-18T14:02:52.626 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:02:52.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:02:52.628 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-163208757251 2024-09-18T14:02:52.628 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:02:52.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=163208757251 2024-09-18T14:02:52.629 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 163208757251 2024-09-18T14:02:52.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 163208757251' 2024-09-18T14:02:52.629 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:02:53.022 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 163208757251 -lt 163208757251 2024-09-18T14:02:53.022 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:02:53.022 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:02:53.022 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:02:53.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T14:02:53.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:02:53.526 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:02:53.526 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:02:53.526 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:02:53.526 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:02:53.526 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:02:53.526 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:02:53.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:02:53.940 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:02:53.940 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:02:53.940 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:02:54.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:02:54.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-18T14:02:54.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T14:02:54.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-18T14:02:54.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:02:54.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:02:54.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:02:54.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:02:54.547 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:02:54.547 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:02:54.548 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:02:54.548 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:02:54.548 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:02:54.548 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:02:54.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:02:54.917 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:02:54.918 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:02:54.918 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:02:55.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:02:55.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:02:55.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:02:55.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:02:55.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:02:55.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:02:55.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:02:55.415 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:02:55.415 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:02:55.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:02:55.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:02:55.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-09-18T14:02:55.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:02:55.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-18T14:02:56.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:02:56.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:02:56.129 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:02:56.129 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:02:56.129 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:02:56.129 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:02:56.129 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:02:56.130 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:02:56.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:02:56.505 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:02:56.505 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:02:56.505 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:02:57.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:02:57.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:02:57.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:02:57.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:02:57.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:02:57.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:02:57.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:02:57.003 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:02:57.003 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:02:57.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:02:57.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:02:57.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-09-18T14:02:57.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:02:57.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-18T14:02:57.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:02:57.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:02:57.900 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:02:57.900 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:02:57.900 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:02:57.900 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:02:57.901 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:02:57.901 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:02:58.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:02:58.281 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:02:58.281 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:02:58.281 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:02:58.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:02:58.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:02:58.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:02:58.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:02:58.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:02:58.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:02:58.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:02:58.783 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:02:58.783 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:02:59.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:02:59.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:02:59.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-09-18T14:02:59.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:02:59.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-09-18T14:03:00.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:03:00.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:03:00.141 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:03:00.141 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:03:00.141 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:03:00.141 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:03:00.141 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:03:00.141 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:03:00.514 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:03:00.515 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:03:00.515 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:03:00.515 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:03:01.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:03:01.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:03:01.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:03:01.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:03:01.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:03:01.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:03:01.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:03:01.037 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:03:01.037 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:03:01.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:03:01.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:03:01.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-09-18T14:03:01.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:03:01.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-09-18T14:03:03.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:03:03.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:03:03.139 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:03:03.139 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:03:03.139 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:03:03.139 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:03:03.139 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:03:03.139 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:03:03.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:03:03.513 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:03:03.513 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:03:03.513 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:03:04.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:03:04.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:03:04.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:03:04.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:03:04.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:03:04.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:03:04.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:03:04.040 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:03:04.040 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:03:04.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:03:04.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:03:04.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-09-18T14:03:04.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:03:04.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-09-18T14:03:07.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:03:07.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:03:07.744 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:03:07.744 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:03:07.744 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:03:07.744 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:03:07.745 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:03:07.745 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:03:08.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:03:08.120 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:03:08.120 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:03:08.120 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:03:08.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:03:08.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:03:08.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:03:08.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:03:08.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:03:08.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:03:08.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:03:08.626 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:03:08.626 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:03:09.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:03:09.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:03:09.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-09-18T14:03:09.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:03:09.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-09-18T14:03:15.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:03:15.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:03:15.566 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:03:15.566 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:03:15.566 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:03:15.566 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:03:15.566 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:03:15.566 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:03:15.947 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:03:15.947 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:03:15.947 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:03:15.948 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:03:16.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:03:16.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:03:16.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:03:16.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:03:16.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:03:16.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:03:16.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:03:16.457 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:03:16.457 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:03:16.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:03:16.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:03:16.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-09-18T14:03:16.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:03:16.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-09-18T14:03:24.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:03:24.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:03:24.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:03:24.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:03:24.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:03:24.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:03:24.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:03:24.273 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:03:24.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:03:24.659 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:03:24.659 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:03:24.659 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:03:25.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:03:25.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:03:25.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:03:25.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:03:25.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:03:25.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:03:25.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:03:25.159 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:03:25.159 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:03:25.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:03:25.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:03:25.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-09-18T14:03:25.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-18T14:03:26.190 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T14:03:26.190 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_fingerprint": "20585fb2-188b-4e34-98c2-e62d5079ffd5", 2024-09-18T14:03:26.190 INFO:tasks.workunit.client.0.smithi047.stdout: "version": "17.2.7-1645-g7061ea6c", 2024-09-18T14:03:26.190 INFO:tasks.workunit.client.0.smithi047.stdout: "commit": "7061ea6ca50908d104e9f3979dc615e118d9e153", 2024-09-18T14:03:26.190 INFO:tasks.workunit.client.0.smithi047.stdout: "timestamp": "2024-09-18T14:03:26.189122+0000", 2024-09-18T14:03:26.190 INFO:tasks.workunit.client.0.smithi047.stdout: "tag": "", 2024-09-18T14:03:26.190 INFO:tasks.workunit.client.0.smithi047.stdout: "health": { 2024-09-18T14:03:26.190 INFO:tasks.workunit.client.0.smithi047.stdout: "status": "HEALTH_WARN", 2024-09-18T14:03:26.190 INFO:tasks.workunit.client.0.smithi047.stdout: "checks": { 2024-09-18T14:03:26.190 INFO:tasks.workunit.client.0.smithi047.stdout: "OSDMAP_FLAGS": { 2024-09-18T14:03:26.191 INFO:tasks.workunit.client.0.smithi047.stdout: "severity": "HEALTH_WARN", 2024-09-18T14:03:26.191 INFO:tasks.workunit.client.0.smithi047.stdout: "summary": { 2024-09-18T14:03:26.191 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-18T14:03:26.191 INFO:tasks.workunit.client.0.smithi047.stdout: "count": 17 2024-09-18T14:03:26.191 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.191 INFO:tasks.workunit.client.0.smithi047.stdout: "detail": [], 2024-09-18T14:03:26.191 INFO:tasks.workunit.client.0.smithi047.stdout: "muted": false 2024-09-18T14:03:26.191 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.191 INFO:tasks.workunit.client.0.smithi047.stdout: "OSD_DOWN": { 2024-09-18T14:03:26.191 INFO:tasks.workunit.client.0.smithi047.stdout: "severity": "HEALTH_WARN", 2024-09-18T14:03:26.191 INFO:tasks.workunit.client.0.smithi047.stdout: "summary": { 2024-09-18T14:03:26.192 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "1 osds down", 2024-09-18T14:03:26.192 INFO:tasks.workunit.client.0.smithi047.stdout: "count": 1 2024-09-18T14:03:26.192 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.192 INFO:tasks.workunit.client.0.smithi047.stdout: "detail": [ 2024-09-18T14:03:26.192 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.192 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "osd.1 (root=default,host=smithi047) is down" 2024-09-18T14:03:26.192 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.192 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:03:26.192 INFO:tasks.workunit.client.0.smithi047.stdout: "muted": false 2024-09-18T14:03:26.192 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.192 INFO:tasks.workunit.client.0.smithi047.stdout: "PG_DEGRADED": { 2024-09-18T14:03:26.192 INFO:tasks.workunit.client.0.smithi047.stdout: "severity": "HEALTH_WARN", 2024-09-18T14:03:26.193 INFO:tasks.workunit.client.0.smithi047.stdout: "summary": { 2024-09-18T14:03:26.193 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2024-09-18T14:03:26.193 INFO:tasks.workunit.client.0.smithi047.stdout: "count": 1 2024-09-18T14:03:26.193 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.193 INFO:tasks.workunit.client.0.smithi047.stdout: "detail": [ 2024-09-18T14:03:26.193 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.193 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-09-18T14:03:26.193 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.193 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:03:26.193 INFO:tasks.workunit.client.0.smithi047.stdout: "muted": false 2024-09-18T14:03:26.193 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.193 INFO:tasks.workunit.client.0.smithi047.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-18T14:03:26.193 INFO:tasks.workunit.client.0.smithi047.stdout: "severity": "HEALTH_WARN", 2024-09-18T14:03:26.194 INFO:tasks.workunit.client.0.smithi047.stdout: "summary": { 2024-09-18T14:03:26.194 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-18T14:03:26.194 INFO:tasks.workunit.client.0.smithi047.stdout: "count": 1 2024-09-18T14:03:26.194 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.194 INFO:tasks.workunit.client.0.smithi047.stdout: "detail": [ 2024-09-18T14:03:26.194 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.194 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "application not enabled on pool 'test'" 2024-09-18T14:03:26.194 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.194 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.194 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-18T14:03:26.194 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.194 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:03:26.194 INFO:tasks.workunit.client.0.smithi047.stdout: "muted": false 2024-09-18T14:03:26.195 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.195 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.195 INFO:tasks.workunit.client.0.smithi047.stdout: "mutes": [] 2024-09-18T14:03:26.195 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.195 INFO:tasks.workunit.client.0.smithi047.stdout: "monmap_first_committed": 1, 2024-09-18T14:03:26.195 INFO:tasks.workunit.client.0.smithi047.stdout: "monmap_last_committed": 1, 2024-09-18T14:03:26.195 INFO:tasks.workunit.client.0.smithi047.stdout: "monmap": { 2024-09-18T14:03:26.195 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 1, 2024-09-18T14:03:26.195 INFO:tasks.workunit.client.0.smithi047.stdout: "fsid": "d7330bd7-8062-424a-bbd8-8c587903c456", 2024-09-18T14:03:26.196 INFO:tasks.workunit.client.0.smithi047.stdout: "modified": "2024-09-18T14:00:11.473527Z", 2024-09-18T14:03:26.196 INFO:tasks.workunit.client.0.smithi047.stdout: "created": "2024-09-18T14:00:11.473527Z", 2024-09-18T14:03:26.196 INFO:tasks.workunit.client.0.smithi047.stdout: "min_mon_release": 17, 2024-09-18T14:03:26.196 INFO:tasks.workunit.client.0.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-18T14:03:26.196 INFO:tasks.workunit.client.0.smithi047.stdout: "election_strategy": 1, 2024-09-18T14:03:26.196 INFO:tasks.workunit.client.0.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-18T14:03:26.196 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_mode": false, 2024-09-18T14:03:26.196 INFO:tasks.workunit.client.0.smithi047.stdout: "tiebreaker_mon": "", 2024-09-18T14:03:26.197 INFO:tasks.workunit.client.0.smithi047.stdout: "removed_ranks: ": "", 2024-09-18T14:03:26.197 INFO:tasks.workunit.client.0.smithi047.stdout: "features": { 2024-09-18T14:03:26.197 INFO:tasks.workunit.client.0.smithi047.stdout: "persistent": [ 2024-09-18T14:03:26.197 INFO:tasks.workunit.client.0.smithi047.stdout: "kraken", 2024-09-18T14:03:26.197 INFO:tasks.workunit.client.0.smithi047.stdout: "luminous", 2024-09-18T14:03:26.197 INFO:tasks.workunit.client.0.smithi047.stdout: "mimic", 2024-09-18T14:03:26.197 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap-prune", 2024-09-18T14:03:26.197 INFO:tasks.workunit.client.0.smithi047.stdout: "nautilus", 2024-09-18T14:03:26.197 INFO:tasks.workunit.client.0.smithi047.stdout: "octopus", 2024-09-18T14:03:26.198 INFO:tasks.workunit.client.0.smithi047.stdout: "pacific", 2024-09-18T14:03:26.198 INFO:tasks.workunit.client.0.smithi047.stdout: "elector-pinging", 2024-09-18T14:03:26.198 INFO:tasks.workunit.client.0.smithi047.stdout: "quincy" 2024-09-18T14:03:26.198 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:03:26.198 INFO:tasks.workunit.client.0.smithi047.stdout: "optional": [] 2024-09-18T14:03:26.198 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.198 INFO:tasks.workunit.client.0.smithi047.stdout: "mons": [ 2024-09-18T14:03:26.198 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.198 INFO:tasks.workunit.client.0.smithi047.stdout: "rank": 0, 2024-09-18T14:03:26.198 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "a", 2024-09-18T14:03:26.199 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addrs": { 2024-09-18T14:03:26.199 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:03:26.199 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.199 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:03:26.199 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:7115", 2024-09-18T14:03:26.199 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 0 2024-09-18T14:03:26.199 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.199 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:03:26.199 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.199 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:7115/0", 2024-09-18T14:03:26.199 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-18T14:03:26.199 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 0, 2024-09-18T14:03:26.199 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 0, 2024-09-18T14:03:26.200 INFO:tasks.workunit.client.0.smithi047.stdout: "crush_location": "{}" 2024-09-18T14:03:26.200 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.200 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:03:26.200 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.200 INFO:tasks.workunit.client.0.smithi047.stdout: "quorum": [ 2024-09-18T14:03:26.200 INFO:tasks.workunit.client.0.smithi047.stdout: 0 2024-09-18T14:03:26.200 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:03:26.200 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap": { 2024-09-18T14:03:26.200 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 41, 2024-09-18T14:03:26.200 INFO:tasks.workunit.client.0.smithi047.stdout: "fsid": "d7330bd7-8062-424a-bbd8-8c587903c456", 2024-09-18T14:03:26.200 INFO:tasks.workunit.client.0.smithi047.stdout: "created": "2024-09-18T14:00:12.273086+0000", 2024-09-18T14:03:26.200 INFO:tasks.workunit.client.0.smithi047.stdout: "modified": "2024-09-18T14:02:50.040549+0000", 2024-09-18T14:03:26.200 INFO:tasks.workunit.client.0.smithi047.stdout: "last_up_change": "2024-09-18T14:02:47.419439+0000", 2024-09-18T14:03:26.200 INFO:tasks.workunit.client.0.smithi047.stdout: "last_in_change": "2024-09-18T14:02:49.728019+0000", 2024-09-18T14:03:26.201 INFO:tasks.workunit.client.0.smithi047.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-18T14:03:26.201 INFO:tasks.workunit.client.0.smithi047.stdout: "flags_num": 5800384, 2024-09-18T14:03:26.201 INFO:tasks.workunit.client.0.smithi047.stdout: "flags_set": [ 2024-09-18T14:03:26.201 INFO:tasks.workunit.client.0.smithi047.stdout: "nodown", 2024-09-18T14:03:26.201 INFO:tasks.workunit.client.0.smithi047.stdout: "noin", 2024-09-18T14:03:26.201 INFO:tasks.workunit.client.0.smithi047.stdout: "noout", 2024-09-18T14:03:26.201 INFO:tasks.workunit.client.0.smithi047.stdout: "pglog_hardlimit", 2024-09-18T14:03:26.201 INFO:tasks.workunit.client.0.smithi047.stdout: "purged_snapdirs", 2024-09-18T14:03:26.201 INFO:tasks.workunit.client.0.smithi047.stdout: "recovery_deletes", 2024-09-18T14:03:26.201 INFO:tasks.workunit.client.0.smithi047.stdout: "sortbitwise" 2024-09-18T14:03:26.201 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:03:26.201 INFO:tasks.workunit.client.0.smithi047.stdout: "crush_version": 9, 2024-09-18T14:03:26.202 INFO:tasks.workunit.client.0.smithi047.stdout: "full_ratio": 0.99000000953674316, 2024-09-18T14:03:26.202 INFO:tasks.workunit.client.0.smithi047.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-18T14:03:26.202 INFO:tasks.workunit.client.0.smithi047.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-18T14:03:26.202 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_snapshot": "", 2024-09-18T14:03:26.202 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_max": 1, 2024-09-18T14:03:26.202 INFO:tasks.workunit.client.0.smithi047.stdout: "max_osd": 3, 2024-09-18T14:03:26.202 INFO:tasks.workunit.client.0.smithi047.stdout: "require_min_compat_client": "luminous", 2024-09-18T14:03:26.202 INFO:tasks.workunit.client.0.smithi047.stdout: "min_compat_client": "jewel", 2024-09-18T14:03:26.202 INFO:tasks.workunit.client.0.smithi047.stdout: "require_osd_release": "quincy", 2024-09-18T14:03:26.202 INFO:tasks.workunit.client.0.smithi047.stdout: "pools": [ 2024-09-18T14:03:26.202 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.202 INFO:tasks.workunit.client.0.smithi047.stdout: "pool": 1, 2024-09-18T14:03:26.202 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_name": "test", 2024-09-18T14:03:26.202 INFO:tasks.workunit.client.0.smithi047.stdout: "create_time": "2024-09-18T14:01:53.733083+0000", 2024-09-18T14:03:26.203 INFO:tasks.workunit.client.0.smithi047.stdout: "flags": 1, 2024-09-18T14:03:26.203 INFO:tasks.workunit.client.0.smithi047.stdout: "flags_names": "hashpspool", 2024-09-18T14:03:26.203 INFO:tasks.workunit.client.0.smithi047.stdout: "type": 1, 2024-09-18T14:03:26.203 INFO:tasks.workunit.client.0.smithi047.stdout: "size": 3, 2024-09-18T14:03:26.203 INFO:tasks.workunit.client.0.smithi047.stdout: "min_size": 2, 2024-09-18T14:03:26.203 INFO:tasks.workunit.client.0.smithi047.stdout: "crush_rule": 0, 2024-09-18T14:03:26.203 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_count": 0, 2024-09-18T14:03:26.203 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_target": 0, 2024-09-18T14:03:26.203 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T14:03:26.203 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T14:03:26.203 INFO:tasks.workunit.client.0.smithi047.stdout: "object_hash": 2, 2024-09-18T14:03:26.203 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_autoscale_mode": "off", 2024-09-18T14:03:26.204 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_num": 1, 2024-09-18T14:03:26.204 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_placement_num": 1, 2024-09-18T14:03:26.204 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_placement_num_target": 1, 2024-09-18T14:03:26.204 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_num_target": 1, 2024-09-18T14:03:26.204 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_num_pending": 1, 2024-09-18T14:03:26.204 INFO:tasks.workunit.client.0.smithi047.stdout: "last_pg_merge_meta": { 2024-09-18T14:03:26.205 INFO:tasks.workunit.client.0.smithi047.stdout: "source_pgid": "0.0", 2024-09-18T14:03:26.205 INFO:tasks.workunit.client.0.smithi047.stdout: "ready_epoch": 0, 2024-09-18T14:03:26.205 INFO:tasks.workunit.client.0.smithi047.stdout: "last_epoch_started": 0, 2024-09-18T14:03:26.205 INFO:tasks.workunit.client.0.smithi047.stdout: "last_epoch_clean": 0, 2024-09-18T14:03:26.205 INFO:tasks.workunit.client.0.smithi047.stdout: "source_version": "0'0", 2024-09-18T14:03:26.206 INFO:tasks.workunit.client.0.smithi047.stdout: "target_version": "0'0" 2024-09-18T14:03:26.206 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.206 INFO:tasks.workunit.client.0.smithi047.stdout: "last_change": "29", 2024-09-18T14:03:26.206 INFO:tasks.workunit.client.0.smithi047.stdout: "last_force_op_resend": "0", 2024-09-18T14:03:26.206 INFO:tasks.workunit.client.0.smithi047.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T14:03:26.206 INFO:tasks.workunit.client.0.smithi047.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T14:03:26.206 INFO:tasks.workunit.client.0.smithi047.stdout: "auid": 0, 2024-09-18T14:03:26.206 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_mode": "selfmanaged", 2024-09-18T14:03:26.206 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_seq": 0, 2024-09-18T14:03:26.206 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_epoch": 0, 2024-09-18T14:03:26.206 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_snaps": [], 2024-09-18T14:03:26.206 INFO:tasks.workunit.client.0.smithi047.stdout: "removed_snaps": "[]", 2024-09-18T14:03:26.206 INFO:tasks.workunit.client.0.smithi047.stdout: "quota_max_bytes": 0, 2024-09-18T14:03:26.206 INFO:tasks.workunit.client.0.smithi047.stdout: "quota_max_objects": 0, 2024-09-18T14:03:26.206 INFO:tasks.workunit.client.0.smithi047.stdout: "tiers": [], 2024-09-18T14:03:26.207 INFO:tasks.workunit.client.0.smithi047.stdout: "tier_of": -1, 2024-09-18T14:03:26.207 INFO:tasks.workunit.client.0.smithi047.stdout: "read_tier": -1, 2024-09-18T14:03:26.207 INFO:tasks.workunit.client.0.smithi047.stdout: "write_tier": -1, 2024-09-18T14:03:26.207 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_mode": "none", 2024-09-18T14:03:26.207 INFO:tasks.workunit.client.0.smithi047.stdout: "target_max_bytes": 0, 2024-09-18T14:03:26.207 INFO:tasks.workunit.client.0.smithi047.stdout: "target_max_objects": 0, 2024-09-18T14:03:26.207 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T14:03:26.207 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T14:03:26.207 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T14:03:26.207 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_min_flush_age": 0, 2024-09-18T14:03:26.207 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_min_evict_age": 0, 2024-09-18T14:03:26.207 INFO:tasks.workunit.client.0.smithi047.stdout: "erasure_code_profile": "", 2024-09-18T14:03:26.207 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_params": { 2024-09-18T14:03:26.207 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "none" 2024-09-18T14:03:26.208 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.208 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_period": 0, 2024-09-18T14:03:26.208 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_count": 0, 2024-09-18T14:03:26.208 INFO:tasks.workunit.client.0.smithi047.stdout: "use_gmt_hitset": true, 2024-09-18T14:03:26.208 INFO:tasks.workunit.client.0.smithi047.stdout: "min_read_recency_for_promote": 0, 2024-09-18T14:03:26.208 INFO:tasks.workunit.client.0.smithi047.stdout: "min_write_recency_for_promote": 0, 2024-09-18T14:03:26.208 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T14:03:26.208 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_search_last_n": 0, 2024-09-18T14:03:26.208 INFO:tasks.workunit.client.0.smithi047.stdout: "grade_table": [], 2024-09-18T14:03:26.208 INFO:tasks.workunit.client.0.smithi047.stdout: "stripe_width": 0, 2024-09-18T14:03:26.208 INFO:tasks.workunit.client.0.smithi047.stdout: "expected_num_objects": 0, 2024-09-18T14:03:26.209 INFO:tasks.workunit.client.0.smithi047.stdout: "fast_read": false, 2024-09-18T14:03:26.209 INFO:tasks.workunit.client.0.smithi047.stdout: "options": {}, 2024-09-18T14:03:26.209 INFO:tasks.workunit.client.0.smithi047.stdout: "application_metadata": {} 2024-09-18T14:03:26.209 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.209 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:03:26.209 INFO:tasks.workunit.client.0.smithi047.stdout: "osds": [ 2024-09-18T14:03:26.209 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.209 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 0, 2024-09-18T14:03:26.209 INFO:tasks.workunit.client.0.smithi047.stdout: "uuid": "df87ad03-46d9-487e-8f7a-aab30f82a1bd", 2024-09-18T14:03:26.210 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 1, 2024-09-18T14:03:26.210 INFO:tasks.workunit.client.0.smithi047.stdout: "in": 1, 2024-09-18T14:03:26.210 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 1, 2024-09-18T14:03:26.210 INFO:tasks.workunit.client.0.smithi047.stdout: "primary_affinity": 1, 2024-09-18T14:03:26.210 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_begin": 5, 2024-09-18T14:03:26.210 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_end": 29, 2024-09-18T14:03:26.210 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 36, 2024-09-18T14:03:26.210 INFO:tasks.workunit.client.0.smithi047.stdout: "up_thru": 40, 2024-09-18T14:03:26.210 INFO:tasks.workunit.client.0.smithi047.stdout: "down_at": 30, 2024-09-18T14:03:26.210 INFO:tasks.workunit.client.0.smithi047.stdout: "lost_at": 0, 2024-09-18T14:03:26.210 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addrs": { 2024-09-18T14:03:26.210 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:03:26.210 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.210 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:03:26.211 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6802", 2024-09-18T14:03:26.211 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 53956 2024-09-18T14:03:26.211 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.211 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.211 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:03:26.211 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6803", 2024-09-18T14:03:26.211 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 53956 2024-09-18T14:03:26.211 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.211 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:03:26.211 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.211 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addrs": { 2024-09-18T14:03:26.211 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:03:26.211 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.211 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:03:26.212 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6804", 2024-09-18T14:03:26.212 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 53956 2024-09-18T14:03:26.212 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.212 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.212 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:03:26.212 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6805", 2024-09-18T14:03:26.212 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 53956 2024-09-18T14:03:26.212 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.212 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:03:26.212 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.212 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addrs": { 2024-09-18T14:03:26.212 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:03:26.212 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.212 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:03:26.213 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6808", 2024-09-18T14:03:26.213 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 53956 2024-09-18T14:03:26.213 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.213 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.213 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:03:26.213 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6809", 2024-09-18T14:03:26.213 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 53956 2024-09-18T14:03:26.213 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.213 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:03:26.213 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.213 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addrs": { 2024-09-18T14:03:26.213 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:03:26.213 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.213 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:03:26.213 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6806", 2024-09-18T14:03:26.214 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 53956 2024-09-18T14:03:26.214 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.214 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.214 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:03:26.214 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6807", 2024-09-18T14:03:26.214 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 53956 2024-09-18T14:03:26.214 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.214 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:03:26.214 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.214 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addr": "127.0.0.1:6803/53956", 2024-09-18T14:03:26.214 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addr": "127.0.0.1:6805/53956", 2024-09-18T14:03:26.214 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addr": "127.0.0.1:6809/53956", 2024-09-18T14:03:26.214 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addr": "127.0.0.1:6807/53956", 2024-09-18T14:03:26.214 INFO:tasks.workunit.client.0.smithi047.stdout: "state": [ 2024-09-18T14:03:26.215 INFO:tasks.workunit.client.0.smithi047.stdout: "exists", 2024-09-18T14:03:26.215 INFO:tasks.workunit.client.0.smithi047.stdout: "up" 2024-09-18T14:03:26.215 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:03:26.215 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.215 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.215 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 1, 2024-09-18T14:03:26.215 INFO:tasks.workunit.client.0.smithi047.stdout: "uuid": "b8177855-da06-4ebd-9d12-538bef0d43b8", 2024-09-18T14:03:26.215 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 0, 2024-09-18T14:03:26.215 INFO:tasks.workunit.client.0.smithi047.stdout: "in": 1, 2024-09-18T14:03:26.215 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 1, 2024-09-18T14:03:26.215 INFO:tasks.workunit.client.0.smithi047.stdout: "primary_affinity": 1, 2024-09-18T14:03:26.215 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_begin": 0, 2024-09-18T14:03:26.215 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_end": 0, 2024-09-18T14:03:26.215 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 10, 2024-09-18T14:03:26.215 INFO:tasks.workunit.client.0.smithi047.stdout: "up_thru": 23, 2024-09-18T14:03:26.216 INFO:tasks.workunit.client.0.smithi047.stdout: "down_at": 31, 2024-09-18T14:03:26.216 INFO:tasks.workunit.client.0.smithi047.stdout: "lost_at": 0, 2024-09-18T14:03:26.216 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addrs": { 2024-09-18T14:03:26.216 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:03:26.216 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.216 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:03:26.216 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6810", 2024-09-18T14:03:26.216 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 47896 2024-09-18T14:03:26.216 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.216 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.216 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:03:26.216 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6811", 2024-09-18T14:03:26.216 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 47896 2024-09-18T14:03:26.216 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.217 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:03:26.217 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.217 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addrs": { 2024-09-18T14:03:26.217 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:03:26.217 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.217 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:03:26.217 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6812", 2024-09-18T14:03:26.217 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 47896 2024-09-18T14:03:26.217 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.217 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.217 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:03:26.217 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6813", 2024-09-18T14:03:26.217 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 47896 2024-09-18T14:03:26.217 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.217 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:03:26.218 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.218 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addrs": { 2024-09-18T14:03:26.218 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:03:26.218 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.218 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:03:26.218 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6816", 2024-09-18T14:03:26.218 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 47896 2024-09-18T14:03:26.218 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.218 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.218 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:03:26.218 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6817", 2024-09-18T14:03:26.218 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 47896 2024-09-18T14:03:26.218 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.218 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:03:26.218 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.219 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addrs": { 2024-09-18T14:03:26.219 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:03:26.219 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.219 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:03:26.219 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6814", 2024-09-18T14:03:26.219 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 47896 2024-09-18T14:03:26.219 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.219 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.219 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:03:26.219 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6815", 2024-09-18T14:03:26.219 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 47896 2024-09-18T14:03:26.219 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.219 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:03:26.219 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.220 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addr": "127.0.0.1:6811/47896", 2024-09-18T14:03:26.220 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addr": "127.0.0.1:6813/47896", 2024-09-18T14:03:26.220 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addr": "127.0.0.1:6817/47896", 2024-09-18T14:03:26.220 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addr": "127.0.0.1:6815/47896", 2024-09-18T14:03:26.220 INFO:tasks.workunit.client.0.smithi047.stdout: "state": [ 2024-09-18T14:03:26.220 INFO:tasks.workunit.client.0.smithi047.stdout: "exists" 2024-09-18T14:03:26.220 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:03:26.220 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.220 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.220 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 2, 2024-09-18T14:03:26.220 INFO:tasks.workunit.client.0.smithi047.stdout: "uuid": "dec93258-345a-406b-85ef-8316978cb8b7", 2024-09-18T14:03:26.220 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 1, 2024-09-18T14:03:26.220 INFO:tasks.workunit.client.0.smithi047.stdout: "in": 1, 2024-09-18T14:03:26.220 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 1, 2024-09-18T14:03:26.221 INFO:tasks.workunit.client.0.smithi047.stdout: "primary_affinity": 1, 2024-09-18T14:03:26.221 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_begin": 15, 2024-09-18T14:03:26.221 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_end": 29, 2024-09-18T14:03:26.221 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 38, 2024-09-18T14:03:26.221 INFO:tasks.workunit.client.0.smithi047.stdout: "up_thru": 0, 2024-09-18T14:03:26.221 INFO:tasks.workunit.client.0.smithi047.stdout: "down_at": 32, 2024-09-18T14:03:26.221 INFO:tasks.workunit.client.0.smithi047.stdout: "lost_at": 0, 2024-09-18T14:03:26.221 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addrs": { 2024-09-18T14:03:26.221 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:03:26.221 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.221 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:03:26.221 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6810", 2024-09-18T14:03:26.221 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 54526 2024-09-18T14:03:26.221 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.221 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.222 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:03:26.222 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6811", 2024-09-18T14:03:26.222 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 54526 2024-09-18T14:03:26.222 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.222 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:03:26.222 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.222 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addrs": { 2024-09-18T14:03:26.222 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:03:26.222 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.222 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:03:26.222 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6812", 2024-09-18T14:03:26.222 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 54526 2024-09-18T14:03:26.222 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.222 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.222 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:03:26.223 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6813", 2024-09-18T14:03:26.223 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 54526 2024-09-18T14:03:26.223 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.223 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:03:26.223 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.223 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addrs": { 2024-09-18T14:03:26.223 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:03:26.223 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.223 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:03:26.223 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6816", 2024-09-18T14:03:26.223 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 54526 2024-09-18T14:03:26.223 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.223 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.223 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:03:26.223 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6817", 2024-09-18T14:03:26.224 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 54526 2024-09-18T14:03:26.224 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.224 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:03:26.224 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.224 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addrs": { 2024-09-18T14:03:26.224 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:03:26.224 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.224 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:03:26.224 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6814", 2024-09-18T14:03:26.224 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 54526 2024-09-18T14:03:26.224 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.224 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.224 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:03:26.224 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6815", 2024-09-18T14:03:26.224 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 54526 2024-09-18T14:03:26.225 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.225 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:03:26.225 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.225 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addr": "127.0.0.1:6811/54526", 2024-09-18T14:03:26.225 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addr": "127.0.0.1:6813/54526", 2024-09-18T14:03:26.225 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addr": "127.0.0.1:6817/54526", 2024-09-18T14:03:26.225 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addr": "127.0.0.1:6815/54526", 2024-09-18T14:03:26.225 INFO:tasks.workunit.client.0.smithi047.stdout: "state": [ 2024-09-18T14:03:26.225 INFO:tasks.workunit.client.0.smithi047.stdout: "exists", 2024-09-18T14:03:26.225 INFO:tasks.workunit.client.0.smithi047.stdout: "up" 2024-09-18T14:03:26.225 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:03:26.225 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.225 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:03:26.225 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_xinfo": [ 2024-09-18T14:03:26.226 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.226 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 0, 2024-09-18T14:03:26.226 INFO:tasks.workunit.client.0.smithi047.stdout: "down_stamp": "2024-09-18T14:02:37.095760+0000", 2024-09-18T14:03:26.226 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_probability": 0, 2024-09-18T14:03:26.226 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_interval": 0, 2024-09-18T14:03:26.226 INFO:tasks.workunit.client.0.smithi047.stdout: "features": 4540138320759226367, 2024-09-18T14:03:26.226 INFO:tasks.workunit.client.0.smithi047.stdout: "old_weight": 0, 2024-09-18T14:03:26.226 INFO:tasks.workunit.client.0.smithi047.stdout: "last_purged_snaps_scrub": "2024-09-18T14:00:43.377185+0000", 2024-09-18T14:03:26.226 INFO:tasks.workunit.client.0.smithi047.stdout: "dead_epoch": 0 2024-09-18T14:03:26.226 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.226 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.226 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 1, 2024-09-18T14:03:26.226 INFO:tasks.workunit.client.0.smithi047.stdout: "down_stamp": "2024-09-18T14:02:37.655555+0000", 2024-09-18T14:03:26.226 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_probability": 0, 2024-09-18T14:03:26.226 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_interval": 0, 2024-09-18T14:03:26.227 INFO:tasks.workunit.client.0.smithi047.stdout: "features": 4540138320759226367, 2024-09-18T14:03:26.227 INFO:tasks.workunit.client.0.smithi047.stdout: "old_weight": 0, 2024-09-18T14:03:26.227 INFO:tasks.workunit.client.0.smithi047.stdout: "last_purged_snaps_scrub": "2024-09-18T14:01:17.318061+0000", 2024-09-18T14:03:26.227 INFO:tasks.workunit.client.0.smithi047.stdout: "dead_epoch": 0 2024-09-18T14:03:26.227 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.227 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.227 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 2, 2024-09-18T14:03:26.227 INFO:tasks.workunit.client.0.smithi047.stdout: "down_stamp": "2024-09-18T14:02:38.212290+0000", 2024-09-18T14:03:26.227 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_probability": 0, 2024-09-18T14:03:26.228 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_interval": 0, 2024-09-18T14:03:26.228 INFO:tasks.workunit.client.0.smithi047.stdout: "features": 4540138320759226367, 2024-09-18T14:03:26.228 INFO:tasks.workunit.client.0.smithi047.stdout: "old_weight": 0, 2024-09-18T14:03:26.228 INFO:tasks.workunit.client.0.smithi047.stdout: "last_purged_snaps_scrub": "2024-09-18T14:01:50.075324+0000", 2024-09-18T14:03:26.228 INFO:tasks.workunit.client.0.smithi047.stdout: "dead_epoch": 0 2024-09-18T14:03:26.228 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.228 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:03:26.228 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_upmap": [], 2024-09-18T14:03:26.228 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_upmap_items": [], 2024-09-18T14:03:26.228 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_temp": [], 2024-09-18T14:03:26.228 INFO:tasks.workunit.client.0.smithi047.stdout: "primary_temp": [], 2024-09-18T14:03:26.228 INFO:tasks.workunit.client.0.smithi047.stdout: "blocklist": {}, 2024-09-18T14:03:26.228 INFO:tasks.workunit.client.0.smithi047.stdout: "range_blocklist": {}, 2024-09-18T14:03:26.228 INFO:tasks.workunit.client.0.smithi047.stdout: "erasure_code_profiles": { 2024-09-18T14:03:26.228 INFO:tasks.workunit.client.0.smithi047.stdout: "default": { 2024-09-18T14:03:26.229 INFO:tasks.workunit.client.0.smithi047.stdout: "k": "2", 2024-09-18T14:03:26.229 INFO:tasks.workunit.client.0.smithi047.stdout: "m": "2", 2024-09-18T14:03:26.229 INFO:tasks.workunit.client.0.smithi047.stdout: "plugin": "jerasure", 2024-09-18T14:03:26.229 INFO:tasks.workunit.client.0.smithi047.stdout: "technique": "reed_sol_van" 2024-09-18T14:03:26.229 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.229 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.229 INFO:tasks.workunit.client.0.smithi047.stdout: "removed_snaps_queue": [], 2024-09-18T14:03:26.229 INFO:tasks.workunit.client.0.smithi047.stdout: "new_removed_snaps": [], 2024-09-18T14:03:26.229 INFO:tasks.workunit.client.0.smithi047.stdout: "new_purged_snaps": [], 2024-09-18T14:03:26.229 INFO:tasks.workunit.client.0.smithi047.stdout: "crush_node_flags": {}, 2024-09-18T14:03:26.229 INFO:tasks.workunit.client.0.smithi047.stdout: "device_class_flags": {}, 2024-09-18T14:03:26.229 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_mode": { 2024-09-18T14:03:26.229 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_mode_enabled": false, 2024-09-18T14:03:26.230 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_bucket_count": 0, 2024-09-18T14:03:26.231 INFO:tasks.workunit.client.0.smithi047.stdout: "degraded_stretch_mode": 0, 2024-09-18T14:03:26.231 INFO:tasks.workunit.client.0.smithi047.stdout: "recovering_stretch_mode": 0, 2024-09-18T14:03:26.231 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_mode_bucket": 0 2024-09-18T14:03:26.231 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.231 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.231 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_metadata": [ 2024-09-18T14:03:26.231 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.231 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T14:03:26.231 INFO:tasks.workunit.client.0.smithi047.stdout: "arch": "x86_64", 2024-09-18T14:03:26.231 INFO:tasks.workunit.client.0.smithi047.stdout: "back_addr": "[v2:127.0.0.1:6804/53956,v1:127.0.0.1:6805/53956]", 2024-09-18T14:03:26.231 INFO:tasks.workunit.client.0.smithi047.stdout: "back_iface": "", 2024-09-18T14:03:26.231 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs": "1", 2024-09-18T14:03:26.232 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_db": "0", 2024-09-18T14:03:26.232 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_wal": "0", 2024-09-18T14:03:26.232 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_single_shared_device": "1", 2024-09-18T14:03:26.232 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_allocation_from_file": "0", 2024-09-18T14:03:26.232 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_access_mode": "file", 2024-09-18T14:03:26.232 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_block_size": "4096", 2024-09-18T14:03:26.232 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_devices": "sda", 2024-09-18T14:03:26.232 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-18T14:03:26.232 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-18T14:03:26.232 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-18T14:03:26.233 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_rotational": "1", 2024-09-18T14:03:26.233 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_size": "107374182400", 2024-09-18T14:03:26.233 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_support_discard": "1", 2024-09-18T14:03:26.233 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_type": "hdd", 2024-09-18T14:03:26.233 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_min_alloc_size": "4096", 2024-09-18T14:03:26.233 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_release": "quincy", 2024-09-18T14:03:26.233 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:03:26.233 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_short": "17.2.7-1645-g7061ea6c", 2024-09-18T14:03:26.233 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_when_created": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:03:26.233 INFO:tasks.workunit.client.0.smithi047.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-18T14:03:26.233 INFO:tasks.workunit.client.0.smithi047.stdout: "created_at": "2024-09-18T14:00:17.705689Z", 2024-09-18T14:03:26.233 INFO:tasks.workunit.client.0.smithi047.stdout: "default_device_class": "hdd", 2024-09-18T14:03:26.233 INFO:tasks.workunit.client.0.smithi047.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4HN80", 2024-09-18T14:03:26.234 INFO:tasks.workunit.client.0.smithi047.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-18T14:03:26.234 INFO:tasks.workunit.client.0.smithi047.stdout: "devices": "sda", 2024-09-18T14:03:26.234 INFO:tasks.workunit.client.0.smithi047.stdout: "distro": "centos", 2024-09-18T14:03:26.234 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_description": "CentOS Stream 9", 2024-09-18T14:03:26.234 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_version": "9", 2024-09-18T14:03:26.234 INFO:tasks.workunit.client.0.smithi047.stdout: "front_addr": "[v2:127.0.0.1:6802/53956,v1:127.0.0.1:6803/53956]", 2024-09-18T14:03:26.234 INFO:tasks.workunit.client.0.smithi047.stdout: "front_iface": "", 2024-09-18T14:03:26.234 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/53956,v1:127.0.0.1:6809/53956]", 2024-09-18T14:03:26.234 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/53956,v1:127.0.0.1:6807/53956]", 2024-09-18T14:03:26.235 INFO:tasks.workunit.client.0.smithi047.stdout: "hostname": "smithi047", 2024-09-18T14:03:26.235 INFO:tasks.workunit.client.0.smithi047.stdout: "journal_rotational": "1", 2024-09-18T14:03:26.235 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-18T14:03:26.235 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-18T14:03:26.235 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_swap_kb": "0", 2024-09-18T14:03:26.235 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_total_kb": "32493492", 2024-09-18T14:03:26.235 INFO:tasks.workunit.client.0.smithi047.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-18T14:03:26.235 INFO:tasks.workunit.client.0.smithi047.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-18T14:03:26.235 INFO:tasks.workunit.client.0.smithi047.stdout: "os": "Linux", 2024-09-18T14:03:26.235 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_data": "td/divergent-priors/0", 2024-09-18T14:03:26.235 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_objectstore": "bluestore", 2024-09-18T14:03:26.235 INFO:tasks.workunit.client.0.smithi047.stdout: "osdspec_affinity": "", 2024-09-18T14:03:26.236 INFO:tasks.workunit.client.0.smithi047.stdout: "rotational": "1" 2024-09-18T14:03:26.236 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.236 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.236 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 1, 2024-09-18T14:03:26.236 INFO:tasks.workunit.client.0.smithi047.stdout: "arch": "x86_64", 2024-09-18T14:03:26.236 INFO:tasks.workunit.client.0.smithi047.stdout: "back_addr": "[v2:127.0.0.1:6812/47896,v1:127.0.0.1:6813/47896]", 2024-09-18T14:03:26.236 INFO:tasks.workunit.client.0.smithi047.stdout: "back_iface": "", 2024-09-18T14:03:26.236 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs": "1", 2024-09-18T14:03:26.236 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_db": "0", 2024-09-18T14:03:26.237 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_wal": "0", 2024-09-18T14:03:26.237 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_single_shared_device": "1", 2024-09-18T14:03:26.237 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_allocation_from_file": "0", 2024-09-18T14:03:26.237 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_access_mode": "file", 2024-09-18T14:03:26.237 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_block_size": "4096", 2024-09-18T14:03:26.237 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_devices": "sda", 2024-09-18T14:03:26.237 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-18T14:03:26.237 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-18T14:03:26.237 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-18T14:03:26.237 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_rotational": "1", 2024-09-18T14:03:26.237 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_size": "107374182400", 2024-09-18T14:03:26.237 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_support_discard": "1", 2024-09-18T14:03:26.238 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_type": "hdd", 2024-09-18T14:03:26.238 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_min_alloc_size": "4096", 2024-09-18T14:03:26.238 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_release": "quincy", 2024-09-18T14:03:26.241 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:03:26.241 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_short": "17.2.7-1645-g7061ea6c", 2024-09-18T14:03:26.241 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_when_created": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:03:26.241 INFO:tasks.workunit.client.0.smithi047.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-18T14:03:26.242 INFO:tasks.workunit.client.0.smithi047.stdout: "created_at": "2024-09-18T14:00:50.562608Z", 2024-09-18T14:03:26.242 INFO:tasks.workunit.client.0.smithi047.stdout: "default_device_class": "hdd", 2024-09-18T14:03:26.242 INFO:tasks.workunit.client.0.smithi047.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4HN80", 2024-09-18T14:03:26.242 INFO:tasks.workunit.client.0.smithi047.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-18T14:03:26.242 INFO:tasks.workunit.client.0.smithi047.stdout: "devices": "sda", 2024-09-18T14:03:26.242 INFO:tasks.workunit.client.0.smithi047.stdout: "distro": "centos", 2024-09-18T14:03:26.242 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_description": "CentOS Stream 9", 2024-09-18T14:03:26.242 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_version": "9", 2024-09-18T14:03:26.242 INFO:tasks.workunit.client.0.smithi047.stdout: "front_addr": "[v2:127.0.0.1:6810/47896,v1:127.0.0.1:6811/47896]", 2024-09-18T14:03:26.242 INFO:tasks.workunit.client.0.smithi047.stdout: "front_iface": "", 2024-09-18T14:03:26.242 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/47896,v1:127.0.0.1:6817/47896]", 2024-09-18T14:03:26.243 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/47896,v1:127.0.0.1:6815/47896]", 2024-09-18T14:03:26.243 INFO:tasks.workunit.client.0.smithi047.stdout: "hostname": "smithi047", 2024-09-18T14:03:26.243 INFO:tasks.workunit.client.0.smithi047.stdout: "journal_rotational": "1", 2024-09-18T14:03:26.243 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-18T14:03:26.243 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-18T14:03:26.243 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_swap_kb": "0", 2024-09-18T14:03:26.243 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_total_kb": "32493492", 2024-09-18T14:03:26.243 INFO:tasks.workunit.client.0.smithi047.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-18T14:03:26.243 INFO:tasks.workunit.client.0.smithi047.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-18T14:03:26.243 INFO:tasks.workunit.client.0.smithi047.stdout: "os": "Linux", 2024-09-18T14:03:26.243 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_data": "td/divergent-priors/1", 2024-09-18T14:03:26.244 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_objectstore": "bluestore", 2024-09-18T14:03:26.244 INFO:tasks.workunit.client.0.smithi047.stdout: "osdspec_affinity": "", 2024-09-18T14:03:26.244 INFO:tasks.workunit.client.0.smithi047.stdout: "rotational": "1" 2024-09-18T14:03:26.244 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.244 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.244 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T14:03:26.244 INFO:tasks.workunit.client.0.smithi047.stdout: "arch": "x86_64", 2024-09-18T14:03:26.244 INFO:tasks.workunit.client.0.smithi047.stdout: "back_addr": "[v2:127.0.0.1:6812/54526,v1:127.0.0.1:6813/54526]", 2024-09-18T14:03:26.244 INFO:tasks.workunit.client.0.smithi047.stdout: "back_iface": "", 2024-09-18T14:03:26.244 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs": "1", 2024-09-18T14:03:26.245 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_db": "0", 2024-09-18T14:03:26.245 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_wal": "0", 2024-09-18T14:03:26.245 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_single_shared_device": "1", 2024-09-18T14:03:26.245 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_allocation_from_file": "0", 2024-09-18T14:03:26.245 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_access_mode": "file", 2024-09-18T14:03:26.245 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_block_size": "4096", 2024-09-18T14:03:26.245 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_devices": "sda", 2024-09-18T14:03:26.245 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-18T14:03:26.245 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-18T14:03:26.245 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-18T14:03:26.245 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_rotational": "1", 2024-09-18T14:03:26.245 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_size": "107374182400", 2024-09-18T14:03:26.245 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_support_discard": "1", 2024-09-18T14:03:26.245 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_type": "hdd", 2024-09-18T14:03:26.246 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_min_alloc_size": "4096", 2024-09-18T14:03:26.246 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_release": "quincy", 2024-09-18T14:03:26.246 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:03:26.246 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_short": "17.2.7-1645-g7061ea6c", 2024-09-18T14:03:26.246 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_when_created": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:03:26.246 INFO:tasks.workunit.client.0.smithi047.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-18T14:03:26.246 INFO:tasks.workunit.client.0.smithi047.stdout: "created_at": "2024-09-18T14:01:23.854123Z", 2024-09-18T14:03:26.246 INFO:tasks.workunit.client.0.smithi047.stdout: "default_device_class": "hdd", 2024-09-18T14:03:26.246 INFO:tasks.workunit.client.0.smithi047.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4HN80", 2024-09-18T14:03:26.246 INFO:tasks.workunit.client.0.smithi047.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-18T14:03:26.247 INFO:tasks.workunit.client.0.smithi047.stdout: "devices": "sda", 2024-09-18T14:03:26.247 INFO:tasks.workunit.client.0.smithi047.stdout: "distro": "centos", 2024-09-18T14:03:26.247 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_description": "CentOS Stream 9", 2024-09-18T14:03:26.247 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_version": "9", 2024-09-18T14:03:26.247 INFO:tasks.workunit.client.0.smithi047.stdout: "front_addr": "[v2:127.0.0.1:6810/54526,v1:127.0.0.1:6811/54526]", 2024-09-18T14:03:26.247 INFO:tasks.workunit.client.0.smithi047.stdout: "front_iface": "", 2024-09-18T14:03:26.247 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/54526,v1:127.0.0.1:6817/54526]", 2024-09-18T14:03:26.247 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/54526,v1:127.0.0.1:6815/54526]", 2024-09-18T14:03:26.247 INFO:tasks.workunit.client.0.smithi047.stdout: "hostname": "smithi047", 2024-09-18T14:03:26.248 INFO:tasks.workunit.client.0.smithi047.stdout: "journal_rotational": "1", 2024-09-18T14:03:26.248 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-18T14:03:26.248 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-18T14:03:26.248 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_swap_kb": "0", 2024-09-18T14:03:26.248 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_total_kb": "32493492", 2024-09-18T14:03:26.248 INFO:tasks.workunit.client.0.smithi047.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-18T14:03:26.248 INFO:tasks.workunit.client.0.smithi047.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-18T14:03:26.248 INFO:tasks.workunit.client.0.smithi047.stdout: "os": "Linux", 2024-09-18T14:03:26.248 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_data": "td/divergent-priors/2", 2024-09-18T14:03:26.248 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_objectstore": "bluestore", 2024-09-18T14:03:26.248 INFO:tasks.workunit.client.0.smithi047.stdout: "osdspec_affinity": "", 2024-09-18T14:03:26.248 INFO:tasks.workunit.client.0.smithi047.stdout: "rotational": "1" 2024-09-18T14:03:26.248 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.249 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:03:26.249 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap_clean_epochs": { 2024-09-18T14:03:26.249 INFO:tasks.workunit.client.0.smithi047.stdout: "min_last_epoch_clean": 0, 2024-09-18T14:03:26.249 INFO:tasks.workunit.client.0.smithi047.stdout: "last_epoch_clean": { 2024-09-18T14:03:26.249 INFO:tasks.workunit.client.0.smithi047.stdout: "per_pool": [] 2024-09-18T14:03:26.249 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.249 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_epochs": [ 2024-09-18T14:03:26.249 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.249 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T14:03:26.249 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 36 2024-09-18T14:03:26.249 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.249 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.249 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T14:03:26.250 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 38 2024-09-18T14:03:26.250 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.250 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:03:26.250 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.250 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap_first_committed": 1, 2024-09-18T14:03:26.250 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap_last_committed": 41, 2024-09-18T14:03:26.250 INFO:tasks.workunit.client.0.smithi047.stdout: "crushmap": { 2024-09-18T14:03:26.250 INFO:tasks.workunit.client.0.smithi047.stdout: "devices": [ 2024-09-18T14:03:26.250 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.250 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T14:03:26.250 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "osd.0", 2024-09-18T14:03:26.250 INFO:tasks.workunit.client.0.smithi047.stdout: "class": "hdd" 2024-09-18T14:03:26.251 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.251 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.251 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 1, 2024-09-18T14:03:26.251 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "osd.1", 2024-09-18T14:03:26.251 INFO:tasks.workunit.client.0.smithi047.stdout: "class": "hdd" 2024-09-18T14:03:26.251 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.251 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.251 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T14:03:26.251 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "osd.2", 2024-09-18T14:03:26.251 INFO:tasks.workunit.client.0.smithi047.stdout: "class": "hdd" 2024-09-18T14:03:26.251 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.251 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:03:26.251 INFO:tasks.workunit.client.0.smithi047.stdout: "types": [ 2024-09-18T14:03:26.252 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.252 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 0, 2024-09-18T14:03:26.252 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "osd" 2024-09-18T14:03:26.252 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.252 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.252 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 1, 2024-09-18T14:03:26.252 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "host" 2024-09-18T14:03:26.252 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.252 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.252 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 2, 2024-09-18T14:03:26.252 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "chassis" 2024-09-18T14:03:26.252 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.252 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.252 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 3, 2024-09-18T14:03:26.253 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "rack" 2024-09-18T14:03:26.253 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.253 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.253 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 4, 2024-09-18T14:03:26.253 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "row" 2024-09-18T14:03:26.253 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.253 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.253 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 5, 2024-09-18T14:03:26.253 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "pdu" 2024-09-18T14:03:26.253 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.254 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.254 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 6, 2024-09-18T14:03:26.254 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "pod" 2024-09-18T14:03:26.254 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.254 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.254 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 7, 2024-09-18T14:03:26.254 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "room" 2024-09-18T14:03:26.254 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.254 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.254 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 8, 2024-09-18T14:03:26.254 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "datacenter" 2024-09-18T14:03:26.254 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.254 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.254 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 9, 2024-09-18T14:03:26.254 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "zone" 2024-09-18T14:03:26.255 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.255 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.255 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 10, 2024-09-18T14:03:26.255 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "region" 2024-09-18T14:03:26.255 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.255 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.255 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 11, 2024-09-18T14:03:26.255 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "root" 2024-09-18T14:03:26.255 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.255 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:03:26.256 INFO:tasks.workunit.client.0.smithi047.stdout: "buckets": [ 2024-09-18T14:03:26.256 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.256 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -1, 2024-09-18T14:03:26.256 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "default", 2024-09-18T14:03:26.256 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 11, 2024-09-18T14:03:26.256 INFO:tasks.workunit.client.0.smithi047.stdout: "type_name": "root", 2024-09-18T14:03:26.256 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:03:26.257 INFO:tasks.workunit.client.0.smithi047.stdout: "alg": "straw2", 2024-09-18T14:03:26.257 INFO:tasks.workunit.client.0.smithi047.stdout: "hash": "rjenkins1", 2024-09-18T14:03:26.257 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T14:03:26.257 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.257 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -3, 2024-09-18T14:03:26.257 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:03:26.257 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 0 2024-09-18T14:03:26.257 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.257 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:03:26.257 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.257 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.257 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -2, 2024-09-18T14:03:26.257 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "default~hdd", 2024-09-18T14:03:26.257 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 11, 2024-09-18T14:03:26.258 INFO:tasks.workunit.client.0.smithi047.stdout: "type_name": "root", 2024-09-18T14:03:26.258 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:03:26.258 INFO:tasks.workunit.client.0.smithi047.stdout: "alg": "straw2", 2024-09-18T14:03:26.258 INFO:tasks.workunit.client.0.smithi047.stdout: "hash": "rjenkins1", 2024-09-18T14:03:26.258 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T14:03:26.258 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.258 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -4, 2024-09-18T14:03:26.258 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:03:26.258 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 0 2024-09-18T14:03:26.258 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.258 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:03:26.258 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.258 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.258 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -3, 2024-09-18T14:03:26.258 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "smithi047", 2024-09-18T14:03:26.259 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 1, 2024-09-18T14:03:26.259 INFO:tasks.workunit.client.0.smithi047.stdout: "type_name": "host", 2024-09-18T14:03:26.259 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:03:26.259 INFO:tasks.workunit.client.0.smithi047.stdout: "alg": "straw2", 2024-09-18T14:03:26.259 INFO:tasks.workunit.client.0.smithi047.stdout: "hash": "rjenkins1", 2024-09-18T14:03:26.259 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T14:03:26.259 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.259 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T14:03:26.259 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:03:26.259 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 0 2024-09-18T14:03:26.259 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.259 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.259 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 1, 2024-09-18T14:03:26.259 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:03:26.259 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 1 2024-09-18T14:03:26.260 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.260 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.260 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T14:03:26.260 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:03:26.260 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 2 2024-09-18T14:03:26.260 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.260 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:03:26.260 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.260 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.260 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -4, 2024-09-18T14:03:26.260 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "smithi047~hdd", 2024-09-18T14:03:26.260 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 1, 2024-09-18T14:03:26.260 INFO:tasks.workunit.client.0.smithi047.stdout: "type_name": "host", 2024-09-18T14:03:26.261 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:03:26.261 INFO:tasks.workunit.client.0.smithi047.stdout: "alg": "straw2", 2024-09-18T14:03:26.261 INFO:tasks.workunit.client.0.smithi047.stdout: "hash": "rjenkins1", 2024-09-18T14:03:26.261 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T14:03:26.261 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.261 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T14:03:26.261 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:03:26.261 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 0 2024-09-18T14:03:26.261 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.261 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.261 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 1, 2024-09-18T14:03:26.261 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:03:26.262 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 1 2024-09-18T14:03:26.262 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.262 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.262 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T14:03:26.262 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:03:26.262 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 2 2024-09-18T14:03:26.262 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.262 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:03:26.262 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.262 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:03:26.262 INFO:tasks.workunit.client.0.smithi047.stdout: "rules": [ 2024-09-18T14:03:26.262 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.262 INFO:tasks.workunit.client.0.smithi047.stdout: "rule_id": 0, 2024-09-18T14:03:26.262 INFO:tasks.workunit.client.0.smithi047.stdout: "rule_name": "replicated_rule", 2024-09-18T14:03:26.263 INFO:tasks.workunit.client.0.smithi047.stdout: "type": 1, 2024-09-18T14:03:26.263 INFO:tasks.workunit.client.0.smithi047.stdout: "steps": [ 2024-09-18T14:03:26.263 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.263 INFO:tasks.workunit.client.0.smithi047.stdout: "op": "take", 2024-09-18T14:03:26.263 INFO:tasks.workunit.client.0.smithi047.stdout: "item": -1, 2024-09-18T14:03:26.263 INFO:tasks.workunit.client.0.smithi047.stdout: "item_name": "default" 2024-09-18T14:03:26.263 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.263 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.263 INFO:tasks.workunit.client.0.smithi047.stdout: "op": "choose_firstn", 2024-09-18T14:03:26.263 INFO:tasks.workunit.client.0.smithi047.stdout: "num": 0, 2024-09-18T14:03:26.263 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "osd" 2024-09-18T14:03:26.263 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.263 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.263 INFO:tasks.workunit.client.0.smithi047.stdout: "op": "emit" 2024-09-18T14:03:26.263 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.264 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:03:26.264 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.264 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:03:26.264 INFO:tasks.workunit.client.0.smithi047.stdout: "tunables": { 2024-09-18T14:03:26.264 INFO:tasks.workunit.client.0.smithi047.stdout: "choose_local_tries": 0, 2024-09-18T14:03:26.264 INFO:tasks.workunit.client.0.smithi047.stdout: "choose_local_fallback_tries": 0, 2024-09-18T14:03:26.264 INFO:tasks.workunit.client.0.smithi047.stdout: "choose_total_tries": 50, 2024-09-18T14:03:26.264 INFO:tasks.workunit.client.0.smithi047.stdout: "chooseleaf_descend_once": 1, 2024-09-18T14:03:26.264 INFO:tasks.workunit.client.0.smithi047.stdout: "chooseleaf_vary_r": 1, 2024-09-18T14:03:26.264 INFO:tasks.workunit.client.0.smithi047.stdout: "chooseleaf_stable": 1, 2024-09-18T14:03:26.264 INFO:tasks.workunit.client.0.smithi047.stdout: "straw_calc_version": 1, 2024-09-18T14:03:26.264 INFO:tasks.workunit.client.0.smithi047.stdout: "allowed_bucket_algs": 54, 2024-09-18T14:03:26.264 INFO:tasks.workunit.client.0.smithi047.stdout: "profile": "jewel", 2024-09-18T14:03:26.264 INFO:tasks.workunit.client.0.smithi047.stdout: "optimal_tunables": 1, 2024-09-18T14:03:26.265 INFO:tasks.workunit.client.0.smithi047.stdout: "legacy_tunables": 0, 2024-09-18T14:03:26.265 INFO:tasks.workunit.client.0.smithi047.stdout: "minimum_required_version": "jewel", 2024-09-18T14:03:26.265 INFO:tasks.workunit.client.0.smithi047.stdout: "require_feature_tunables": 1, 2024-09-18T14:03:26.265 INFO:tasks.workunit.client.0.smithi047.stdout: "require_feature_tunables2": 1, 2024-09-18T14:03:26.265 INFO:tasks.workunit.client.0.smithi047.stdout: "has_v2_rules": 0, 2024-09-18T14:03:26.265 INFO:tasks.workunit.client.0.smithi047.stdout: "require_feature_tunables3": 1, 2024-09-18T14:03:26.265 INFO:tasks.workunit.client.0.smithi047.stdout: "has_v3_rules": 0, 2024-09-18T14:03:26.265 INFO:tasks.workunit.client.0.smithi047.stdout: "has_v4_buckets": 1, 2024-09-18T14:03:26.265 INFO:tasks.workunit.client.0.smithi047.stdout: "require_feature_tunables5": 1, 2024-09-18T14:03:26.265 INFO:tasks.workunit.client.0.smithi047.stdout: "has_v5_rules": 0 2024-09-18T14:03:26.265 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.265 INFO:tasks.workunit.client.0.smithi047.stdout: "choose_args": {} 2024-09-18T14:03:26.265 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.265 INFO:tasks.workunit.client.0.smithi047.stdout: "fsmap": { 2024-09-18T14:03:26.266 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 1, 2024-09-18T14:03:26.266 INFO:tasks.workunit.client.0.smithi047.stdout: "default_fscid": -1, 2024-09-18T14:03:26.266 INFO:tasks.workunit.client.0.smithi047.stdout: "compat": { 2024-09-18T14:03:26.266 INFO:tasks.workunit.client.0.smithi047.stdout: "compat": {}, 2024-09-18T14:03:26.266 INFO:tasks.workunit.client.0.smithi047.stdout: "ro_compat": {}, 2024-09-18T14:03:26.266 INFO:tasks.workunit.client.0.smithi047.stdout: "incompat": { 2024-09-18T14:03:26.266 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_1": "base v0.20", 2024-09-18T14:03:26.266 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_2": "client writeable ranges", 2024-09-18T14:03:26.266 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_3": "default file layouts on dirs", 2024-09-18T14:03:26.266 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_4": "dir inode in separate object", 2024-09-18T14:03:26.266 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_5": "mds uses versioned encoding", 2024-09-18T14:03:26.266 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-18T14:03:26.266 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_8": "no anchor table", 2024-09-18T14:03:26.266 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_9": "file layout v2", 2024-09-18T14:03:26.267 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_10": "snaprealm v2" 2024-09-18T14:03:26.267 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.267 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.267 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_flags": { 2024-09-18T14:03:26.267 INFO:tasks.workunit.client.0.smithi047.stdout: "enable_multiple": true, 2024-09-18T14:03:26.267 INFO:tasks.workunit.client.0.smithi047.stdout: "ever_enabled_multiple": true 2024-09-18T14:03:26.267 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.267 INFO:tasks.workunit.client.0.smithi047.stdout: "standbys": [], 2024-09-18T14:03:26.267 INFO:tasks.workunit.client.0.smithi047.stdout: "filesystems": [] 2024-09-18T14:03:26.267 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.267 INFO:tasks.workunit.client.0.smithi047.stdout: "mdsmap_first_committed": 1, 2024-09-18T14:03:26.267 INFO:tasks.workunit.client.0.smithi047.stdout: "mdsmap_last_committed": 1, 2024-09-18T14:03:26.268 INFO:tasks.workunit.client.0.smithi047.stdout: "auth": { 2024-09-18T14:03:26.268 INFO:tasks.workunit.client.0.smithi047.stdout: "first_committed": 1, 2024-09-18T14:03:26.268 INFO:tasks.workunit.client.0.smithi047.stdout: "last_committed": 5, 2024-09-18T14:03:26.268 INFO:tasks.workunit.client.0.smithi047.stdout: "num_secrets": 3 2024-09-18T14:03:26.268 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.268 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg": 1, 2024-09-18T14:03:26.268 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg_active": 1, 2024-09-18T14:03:26.268 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg_unknown": 0, 2024-09-18T14:03:26.268 INFO:tasks.workunit.client.0.smithi047.stdout: "num_osd": 3, 2024-09-18T14:03:26.268 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_sum": { 2024-09-18T14:03:26.268 INFO:tasks.workunit.client.0.smithi047.stdout: "stat_sum": { 2024-09-18T14:03:26.268 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes": 48300, 2024-09-18T14:03:26.269 INFO:tasks.workunit.client.0.smithi047.stderr:report 4080325241 2024-09-18T14:03:26.269 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects": 100, 2024-09-18T14:03:26.269 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_clones": 0, 2024-09-18T14:03:26.269 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_copies": 300, 2024-09-18T14:03:26.269 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing_on_primary": 0, 2024-09-18T14:03:26.269 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing": 0, 2024-09-18T14:03:26.270 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_degraded": 100, 2024-09-18T14:03:26.270 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_misplaced": 0, 2024-09-18T14:03:26.270 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_unfound": 0, 2024-09-18T14:03:26.270 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_dirty": 100, 2024-09-18T14:03:26.270 INFO:tasks.workunit.client.0.smithi047.stdout: "num_whiteouts": 0, 2024-09-18T14:03:26.270 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read": 0, 2024-09-18T14:03:26.270 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read_kb": 0, 2024-09-18T14:03:26.270 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write": 100, 2024-09-18T14:03:26.270 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write_kb": 100, 2024-09-18T14:03:26.270 INFO:tasks.workunit.client.0.smithi047.stdout: "num_scrub_errors": 0, 2024-09-18T14:03:26.270 INFO:tasks.workunit.client.0.smithi047.stdout: "num_shallow_scrub_errors": 0, 2024-09-18T14:03:26.271 INFO:tasks.workunit.client.0.smithi047.stdout: "num_deep_scrub_errors": 0, 2024-09-18T14:03:26.271 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_recovered": 0, 2024-09-18T14:03:26.271 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_recovered": 0, 2024-09-18T14:03:26.271 INFO:tasks.workunit.client.0.smithi047.stdout: "num_keys_recovered": 0, 2024-09-18T14:03:26.271 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_omap": 0, 2024-09-18T14:03:26.271 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_hit_set_archive": 0, 2024-09-18T14:03:26.271 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_hit_set_archive": 0, 2024-09-18T14:03:26.271 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush": 0, 2024-09-18T14:03:26.271 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_kb": 0, 2024-09-18T14:03:26.271 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict": 0, 2024-09-18T14:03:26.271 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_kb": 0, 2024-09-18T14:03:26.271 INFO:tasks.workunit.client.0.smithi047.stdout: "num_promote": 0, 2024-09-18T14:03:26.271 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_high": 0, 2024-09-18T14:03:26.272 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_low": 0, 2024-09-18T14:03:26.272 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_some": 0, 2024-09-18T14:03:26.272 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_full": 0, 2024-09-18T14:03:26.272 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_pinned": 0, 2024-09-18T14:03:26.272 INFO:tasks.workunit.client.0.smithi047.stdout: "num_legacy_snapsets": 0, 2024-09-18T14:03:26.272 INFO:tasks.workunit.client.0.smithi047.stdout: "num_large_omap_objects": 0, 2024-09-18T14:03:26.272 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_manifest": 0, 2024-09-18T14:03:26.272 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_bytes": 0, 2024-09-18T14:03:26.272 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_keys": 0, 2024-09-18T14:03:26.272 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_repaired": 0 2024-09-18T14:03:26.272 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.272 INFO:tasks.workunit.client.0.smithi047.stdout: "store_stats": { 2024-09-18T14:03:26.272 INFO:tasks.workunit.client.0.smithi047.stdout: "total": 0, 2024-09-18T14:03:26.272 INFO:tasks.workunit.client.0.smithi047.stdout: "available": 0, 2024-09-18T14:03:26.273 INFO:tasks.workunit.client.0.smithi047.stdout: "internally_reserved": 0, 2024-09-18T14:03:26.273 INFO:tasks.workunit.client.0.smithi047.stdout: "allocated": 0, 2024-09-18T14:03:26.273 INFO:tasks.workunit.client.0.smithi047.stdout: "data_stored": 0, 2024-09-18T14:03:26.273 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed": 0, 2024-09-18T14:03:26.273 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_allocated": 0, 2024-09-18T14:03:26.273 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_original": 0, 2024-09-18T14:03:26.273 INFO:tasks.workunit.client.0.smithi047.stdout: "omap_allocated": 0, 2024-09-18T14:03:26.273 INFO:tasks.workunit.client.0.smithi047.stdout: "internal_metadata": 0 2024-09-18T14:03:26.273 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.273 INFO:tasks.workunit.client.0.smithi047.stdout: "log_size": 16, 2024-09-18T14:03:26.273 INFO:tasks.workunit.client.0.smithi047.stdout: "ondisk_log_size": 16, 2024-09-18T14:03:26.273 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 2, 2024-09-18T14:03:26.273 INFO:tasks.workunit.client.0.smithi047.stdout: "acting": 2, 2024-09-18T14:03:26.273 INFO:tasks.workunit.client.0.smithi047.stdout: "num_store_stats": 0 2024-09-18T14:03:26.274 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.274 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_sum": { 2024-09-18T14:03:26.274 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 0, 2024-09-18T14:03:26.274 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 0, 2024-09-18T14:03:26.274 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pgs": 3, 2024-09-18T14:03:26.274 INFO:tasks.workunit.client.0.smithi047.stdout: "num_osds": 3, 2024-09-18T14:03:26.274 INFO:tasks.workunit.client.0.smithi047.stdout: "num_per_pool_osds": 3, 2024-09-18T14:03:26.274 INFO:tasks.workunit.client.0.smithi047.stdout: "num_per_pool_omap_osds": 3, 2024-09-18T14:03:26.274 INFO:tasks.workunit.client.0.smithi047.stdout: "kb": 314572800, 2024-09-18T14:03:26.274 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used": 894628, 2024-09-18T14:03:26.274 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_data": 2104, 2024-09-18T14:03:26.274 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_omap": 10, 2024-09-18T14:03:26.274 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_meta": 892469, 2024-09-18T14:03:26.274 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_avail": 313678172, 2024-09-18T14:03:26.275 INFO:tasks.workunit.client.0.smithi047.stdout: "statfs": { 2024-09-18T14:03:26.275 INFO:tasks.workunit.client.0.smithi047.stdout: "total": 322122547200, 2024-09-18T14:03:26.275 INFO:tasks.workunit.client.0.smithi047.stdout: "available": 321206448128, 2024-09-18T14:03:26.275 INFO:tasks.workunit.client.0.smithi047.stdout: "internally_reserved": 0, 2024-09-18T14:03:26.275 INFO:tasks.workunit.client.0.smithi047.stdout: "allocated": 2154496, 2024-09-18T14:03:26.275 INFO:tasks.workunit.client.0.smithi047.stdout: "data_stored": 417081, 2024-09-18T14:03:26.275 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed": 0, 2024-09-18T14:03:26.275 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_allocated": 0, 2024-09-18T14:03:26.275 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_original": 0, 2024-09-18T14:03:26.275 INFO:tasks.workunit.client.0.smithi047.stdout: "omap_allocated": 10790, 2024-09-18T14:03:26.275 INFO:tasks.workunit.client.0.smithi047.stdout: "internal_metadata": 913888730 2024-09-18T14:03:26.275 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.275 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_peers": [], 2024-09-18T14:03:26.275 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_trim_queue_len": 0, 2024-09-18T14:03:26.275 INFO:tasks.workunit.client.0.smithi047.stdout: "num_snap_trimming": 0, 2024-09-18T14:03:26.276 INFO:tasks.workunit.client.0.smithi047.stdout: "num_shards_repaired": 0, 2024-09-18T14:03:26.276 INFO:tasks.workunit.client.0.smithi047.stdout: "op_queue_age_hist": { 2024-09-18T14:03:26.276 INFO:tasks.workunit.client.0.smithi047.stdout: "histogram": [], 2024-09-18T14:03:26.276 INFO:tasks.workunit.client.0.smithi047.stdout: "upper_bound": 1 2024-09-18T14:03:26.276 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.276 INFO:tasks.workunit.client.0.smithi047.stdout: "perf_stat": { 2024-09-18T14:03:26.276 INFO:tasks.workunit.client.0.smithi047.stdout: "commit_latency_ms": 0, 2024-09-18T14:03:26.276 INFO:tasks.workunit.client.0.smithi047.stdout: "apply_latency_ms": 0, 2024-09-18T14:03:26.276 INFO:tasks.workunit.client.0.smithi047.stdout: "commit_latency_ns": 0, 2024-09-18T14:03:26.276 INFO:tasks.workunit.client.0.smithi047.stdout: "apply_latency_ns": 0 2024-09-18T14:03:26.276 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.276 INFO:tasks.workunit.client.0.smithi047.stdout: "alerts": [], 2024-09-18T14:03:26.276 INFO:tasks.workunit.client.0.smithi047.stdout: "network_ping_times": [] 2024-09-18T14:03:26.276 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.277 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_sum_by_class": { 2024-09-18T14:03:26.277 INFO:tasks.workunit.client.0.smithi047.stdout: "hdd": { 2024-09-18T14:03:26.277 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 0, 2024-09-18T14:03:26.277 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 0, 2024-09-18T14:03:26.277 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pgs": 3, 2024-09-18T14:03:26.277 INFO:tasks.workunit.client.0.smithi047.stdout: "num_osds": 3, 2024-09-18T14:03:26.277 INFO:tasks.workunit.client.0.smithi047.stdout: "num_per_pool_osds": 3, 2024-09-18T14:03:26.277 INFO:tasks.workunit.client.0.smithi047.stdout: "num_per_pool_omap_osds": 3, 2024-09-18T14:03:26.277 INFO:tasks.workunit.client.0.smithi047.stdout: "kb": 314572800, 2024-09-18T14:03:26.277 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used": 894628, 2024-09-18T14:03:26.277 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_data": 2104, 2024-09-18T14:03:26.277 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_omap": 10, 2024-09-18T14:03:26.277 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_meta": 892469, 2024-09-18T14:03:26.277 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_avail": 313678172, 2024-09-18T14:03:26.278 INFO:tasks.workunit.client.0.smithi047.stdout: "statfs": { 2024-09-18T14:03:26.278 INFO:tasks.workunit.client.0.smithi047.stdout: "total": 322122547200, 2024-09-18T14:03:26.278 INFO:tasks.workunit.client.0.smithi047.stdout: "available": 321206448128, 2024-09-18T14:03:26.278 INFO:tasks.workunit.client.0.smithi047.stdout: "internally_reserved": 0, 2024-09-18T14:03:26.278 INFO:tasks.workunit.client.0.smithi047.stdout: "allocated": 2154496, 2024-09-18T14:03:26.278 INFO:tasks.workunit.client.0.smithi047.stdout: "data_stored": 417081, 2024-09-18T14:03:26.278 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed": 0, 2024-09-18T14:03:26.278 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_allocated": 0, 2024-09-18T14:03:26.278 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_original": 0, 2024-09-18T14:03:26.278 INFO:tasks.workunit.client.0.smithi047.stdout: "omap_allocated": 10790, 2024-09-18T14:03:26.278 INFO:tasks.workunit.client.0.smithi047.stdout: "internal_metadata": 913888730 2024-09-18T14:03:26.278 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.278 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_peers": [], 2024-09-18T14:03:26.279 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_trim_queue_len": 0, 2024-09-18T14:03:26.279 INFO:tasks.workunit.client.0.smithi047.stdout: "num_snap_trimming": 0, 2024-09-18T14:03:26.279 INFO:tasks.workunit.client.0.smithi047.stdout: "num_shards_repaired": 0, 2024-09-18T14:03:26.279 INFO:tasks.workunit.client.0.smithi047.stdout: "op_queue_age_hist": { 2024-09-18T14:03:26.279 INFO:tasks.workunit.client.0.smithi047.stdout: "histogram": [], 2024-09-18T14:03:26.279 INFO:tasks.workunit.client.0.smithi047.stdout: "upper_bound": 1 2024-09-18T14:03:26.279 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.279 INFO:tasks.workunit.client.0.smithi047.stdout: "perf_stat": { 2024-09-18T14:03:26.279 INFO:tasks.workunit.client.0.smithi047.stdout: "commit_latency_ms": 0, 2024-09-18T14:03:26.279 INFO:tasks.workunit.client.0.smithi047.stdout: "apply_latency_ms": 0, 2024-09-18T14:03:26.279 INFO:tasks.workunit.client.0.smithi047.stdout: "commit_latency_ns": 0, 2024-09-18T14:03:26.279 INFO:tasks.workunit.client.0.smithi047.stdout: "apply_latency_ns": 0 2024-09-18T14:03:26.279 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.280 INFO:tasks.workunit.client.0.smithi047.stdout: "alerts": [], 2024-09-18T14:03:26.280 INFO:tasks.workunit.client.0.smithi047.stdout: "network_ping_times": [] 2024-09-18T14:03:26.280 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.280 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.280 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_stats": [ 2024-09-18T14:03:26.280 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.280 INFO:tasks.workunit.client.0.smithi047.stdout: "poolid": 1, 2024-09-18T14:03:26.280 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg": 1, 2024-09-18T14:03:26.280 INFO:tasks.workunit.client.0.smithi047.stdout: "stat_sum": { 2024-09-18T14:03:26.280 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes": 48300, 2024-09-18T14:03:26.280 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects": 100, 2024-09-18T14:03:26.280 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_clones": 0, 2024-09-18T14:03:26.280 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_copies": 300, 2024-09-18T14:03:26.280 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing_on_primary": 0, 2024-09-18T14:03:26.281 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing": 0, 2024-09-18T14:03:26.281 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_degraded": 100, 2024-09-18T14:03:26.281 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_misplaced": 0, 2024-09-18T14:03:26.281 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_unfound": 0, 2024-09-18T14:03:26.281 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_dirty": 100, 2024-09-18T14:03:26.281 INFO:tasks.workunit.client.0.smithi047.stdout: "num_whiteouts": 0, 2024-09-18T14:03:26.281 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read": 0, 2024-09-18T14:03:26.281 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read_kb": 0, 2024-09-18T14:03:26.281 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write": 100, 2024-09-18T14:03:26.281 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write_kb": 100, 2024-09-18T14:03:26.281 INFO:tasks.workunit.client.0.smithi047.stdout: "num_scrub_errors": 0, 2024-09-18T14:03:26.281 INFO:tasks.workunit.client.0.smithi047.stdout: "num_shallow_scrub_errors": 0, 2024-09-18T14:03:26.281 INFO:tasks.workunit.client.0.smithi047.stdout: "num_deep_scrub_errors": 0, 2024-09-18T14:03:26.281 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_recovered": 0, 2024-09-18T14:03:26.281 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_recovered": 0, 2024-09-18T14:03:26.282 INFO:tasks.workunit.client.0.smithi047.stdout: "num_keys_recovered": 0, 2024-09-18T14:03:26.282 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_omap": 0, 2024-09-18T14:03:26.282 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_hit_set_archive": 0, 2024-09-18T14:03:26.282 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_hit_set_archive": 0, 2024-09-18T14:03:26.282 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush": 0, 2024-09-18T14:03:26.282 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_kb": 0, 2024-09-18T14:03:26.282 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict": 0, 2024-09-18T14:03:26.282 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_kb": 0, 2024-09-18T14:03:26.282 INFO:tasks.workunit.client.0.smithi047.stdout: "num_promote": 0, 2024-09-18T14:03:26.282 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_high": 0, 2024-09-18T14:03:26.283 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_low": 0, 2024-09-18T14:03:26.283 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_some": 0, 2024-09-18T14:03:26.283 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_full": 0, 2024-09-18T14:03:26.283 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_pinned": 0, 2024-09-18T14:03:26.283 INFO:tasks.workunit.client.0.smithi047.stdout: "num_legacy_snapsets": 0, 2024-09-18T14:03:26.283 INFO:tasks.workunit.client.0.smithi047.stdout: "num_large_omap_objects": 0, 2024-09-18T14:03:26.283 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_manifest": 0, 2024-09-18T14:03:26.284 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_bytes": 0, 2024-09-18T14:03:26.284 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_keys": 0, 2024-09-18T14:03:26.284 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_repaired": 0 2024-09-18T14:03:26.284 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.284 INFO:tasks.workunit.client.0.smithi047.stdout: "store_stats": { 2024-09-18T14:03:26.284 INFO:tasks.workunit.client.0.smithi047.stdout: "total": 0, 2024-09-18T14:03:26.284 INFO:tasks.workunit.client.0.smithi047.stdout: "available": 0, 2024-09-18T14:03:26.284 INFO:tasks.workunit.client.0.smithi047.stdout: "internally_reserved": 0, 2024-09-18T14:03:26.284 INFO:tasks.workunit.client.0.smithi047.stdout: "allocated": 1220608, 2024-09-18T14:03:26.284 INFO:tasks.workunit.client.0.smithi047.stdout: "data_stored": 143276, 2024-09-18T14:03:26.285 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed": 0, 2024-09-18T14:03:26.285 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_allocated": 0, 2024-09-18T14:03:26.285 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_original": 0, 2024-09-18T14:03:26.285 INFO:tasks.workunit.client.0.smithi047.stdout: "omap_allocated": 0, 2024-09-18T14:03:26.285 INFO:tasks.workunit.client.0.smithi047.stdout: "internal_metadata": 0 2024-09-18T14:03:26.285 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.285 INFO:tasks.workunit.client.0.smithi047.stdout: "log_size": 16, 2024-09-18T14:03:26.285 INFO:tasks.workunit.client.0.smithi047.stdout: "ondisk_log_size": 16, 2024-09-18T14:03:26.285 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 2, 2024-09-18T14:03:26.285 INFO:tasks.workunit.client.0.smithi047.stdout: "acting": 2, 2024-09-18T14:03:26.285 INFO:tasks.workunit.client.0.smithi047.stdout: "num_store_stats": 3 2024-09-18T14:03:26.285 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.285 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:03:26.285 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_stats": [ 2024-09-18T14:03:26.286 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.286 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 0, 2024-09-18T14:03:26.286 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 154618822666 2024-09-18T14:03:26.286 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.286 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.286 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 1, 2024-09-18T14:03:26.286 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 42949672984 2024-09-18T14:03:26.286 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.286 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.286 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 2, 2024-09-18T14:03:26.286 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 163208757258 2024-09-18T14:03:26.286 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.286 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:03:26.286 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg_by_state": [ 2024-09-18T14:03:26.287 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.287 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "active+undersized+degraded", 2024-09-18T14:03:26.287 INFO:tasks.workunit.client.0.smithi047.stdout: "num": 1 2024-09-18T14:03:26.287 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.287 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:03:26.287 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg_by_osd": [ 2024-09-18T14:03:26.287 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.287 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 2, 2024-09-18T14:03:26.287 INFO:tasks.workunit.client.0.smithi047.stdout: "num_primary_pg": 0, 2024-09-18T14:03:26.287 INFO:tasks.workunit.client.0.smithi047.stdout: "num_acting_pg": 1, 2024-09-18T14:03:26.287 INFO:tasks.workunit.client.0.smithi047.stdout: "num_up_not_acting_pg": 0 2024-09-18T14:03:26.287 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.287 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.287 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 0, 2024-09-18T14:03:26.287 INFO:tasks.workunit.client.0.smithi047.stdout: "num_primary_pg": 1, 2024-09-18T14:03:26.288 INFO:tasks.workunit.client.0.smithi047.stdout: "num_acting_pg": 1, 2024-09-18T14:03:26.288 INFO:tasks.workunit.client.0.smithi047.stdout: "num_up_not_acting_pg": 0 2024-09-18T14:03:26.288 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.288 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.288 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 1, 2024-09-18T14:03:26.288 INFO:tasks.workunit.client.0.smithi047.stdout: "num_primary_pg": 0, 2024-09-18T14:03:26.288 INFO:tasks.workunit.client.0.smithi047.stdout: "num_acting_pg": 0, 2024-09-18T14:03:26.288 INFO:tasks.workunit.client.0.smithi047.stdout: "num_up_not_acting_pg": 0 2024-09-18T14:03:26.288 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.288 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:03:26.288 INFO:tasks.workunit.client.0.smithi047.stdout: "purged_snaps": [ 2024-09-18T14:03:26.288 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:03:26.288 INFO:tasks.workunit.client.0.smithi047.stdout: "pool": 1, 2024-09-18T14:03:26.289 INFO:tasks.workunit.client.0.smithi047.stdout: "purged_snaps": {} 2024-09-18T14:03:26.289 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.289 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:03:26.289 INFO:tasks.workunit.client.0.smithi047.stdout: "servicemap": { 2024-09-18T14:03:26.289 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 5, 2024-09-18T14:03:26.289 INFO:tasks.workunit.client.0.smithi047.stdout: "modified": "2024-09-18T14:02:40.391154+0000", 2024-09-18T14:03:26.289 INFO:tasks.workunit.client.0.smithi047.stdout: "services": {} 2024-09-18T14:03:26.289 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:03:26.289 INFO:tasks.workunit.client.0.smithi047.stdout: "paxos": { 2024-09-18T14:03:26.289 INFO:tasks.workunit.client.0.smithi047.stdout: "first_committed": 1, 2024-09-18T14:03:26.289 INFO:tasks.workunit.client.0.smithi047.stdout: "last_committed": 499, 2024-09-18T14:03:26.289 INFO:tasks.workunit.client.0.smithi047.stdout: "last_pn": 0, 2024-09-18T14:03:26.289 INFO:tasks.workunit.client.0.smithi047.stdout: "accepted_pn": 0 2024-09-18T14:03:26.289 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:03:26.289 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T14:03:26.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-18T14:03:26.290 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: expr 2 + 2 2024-09-18T14:03:26.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: objname=existing_4 2024-09-18T14:03:26.291 INFO:tasks.workunit.client.0.smithi047.stdout:writing non-divergent object existing_4 2024-09-18T14:03:26.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:529: TEST_divergent_2: echo 'writing non-divergent object existing_4' 2024-09-18T14:03:26.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:530: TEST_divergent_2: ceph pg dump pgs 2024-09-18T14:03:26.596 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:03:26.596 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 100 0 100 0 0 48300 0 0 16 84 16 active+undersized+degraded 2024-09-18T14:02:50.350708+0000 29'100 41:123 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:01:53.783298+0000 0'0 2024-09-18T14:01:53.783298+0000 0 0 periodic scrub scheduled @ 2024-09-19T15:26:35.100116+0000 0 0 2024-09-18T14:03:26.597 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:03:26.597 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:03:26.597 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:03:26.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:531: TEST_divergent_2: rados -p test put existing_4 /etc/resolv.conf 2024-09-18T14:03:26.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-18T14:03:26.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: wait_for_clean 2024-09-18T14:03:26.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:03:26.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:03:26.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:03:26.688 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-18T14:03:26.689 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:03:26.689 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:03:26.689 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:03:26.689 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:03:26.689 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:03:26.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-18T14:03:26.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:03:26.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:03:26.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:03:26.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:03:26.745 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:03:27.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:03:27.122 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T14:03:27.122 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T14:03:27.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:03:27.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:03:27.122 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:03:27.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=154618822669 2024-09-18T14:03:27.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 154618822669 2024-09-18T14:03:27.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-154618822669' 2024-09-18T14:03:27.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:03:27.238 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:03:27.323 INFO:tasks.workunit.client.0.smithi047.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-18T14:03:27.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq= 2024-09-18T14:03:27.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z '' 2024-09-18T14:03:27.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: flush_pg_stats: continue 2024-09-18T14:03:27.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:03:27.325 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:03:27.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=163208757260 2024-09-18T14:03:27.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 163208757260 2024-09-18T14:03:27.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-154618822669 2-163208757260' 2024-09-18T14:03:27.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:03:27.441 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-154618822669 2024-09-18T14:03:27.441 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:03:27.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:03:27.442 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-154618822669 2024-09-18T14:03:27.442 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:03:27.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=154618822669 2024-09-18T14:03:27.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 154618822669' 2024-09-18T14:03:27.444 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 154618822669 2024-09-18T14:03:27.444 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:03:27.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 154618822667 -lt 154618822669 2024-09-18T14:03:27.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:03:28.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:03:28.836 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:03:29.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 154618822669 -lt 154618822669 2024-09-18T14:03:29.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:03:29.270 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-163208757260 2024-09-18T14:03:29.270 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:03:29.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:03:29.271 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-163208757260 2024-09-18T14:03:29.271 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:03:29.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=163208757260 2024-09-18T14:03:29.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 163208757260' 2024-09-18T14:03:29.272 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 163208757260 2024-09-18T14:03:29.273 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:03:29.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 163208757261 -lt 163208757260 2024-09-18T14:03:29.729 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:03:29.729 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:03:29.729 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:03:30.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T14:03:30.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:03:30.229 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:03:30.229 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:03:30.229 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:03:30.229 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:03:30.230 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:03:30.230 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:03:30.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:03:30.625 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:03:30.626 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:03:30.626 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:03:31.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:03:31.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-18T14:03:31.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T14:03:31.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-18T14:03:31.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:03:31.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:03:31.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:03:31.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:03:31.232 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:03:31.233 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:03:31.233 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:03:31.233 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:03:31.233 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:03:31.233 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:03:31.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:03:31.604 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:03:31.604 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:03:31.604 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:03:32.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:03:32.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:03:32.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:03:32.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:03:32.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:03:32.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:03:32.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:03:32.108 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:03:32.108 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:03:32.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:03:32.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:03:32.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-09-18T14:03:32.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:03:32.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-18T14:03:32.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:03:32.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:03:32.856 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:03:32.856 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:03:32.856 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:03:32.856 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:03:32.856 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:03:32.856 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:03:33.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:03:33.219 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:03:33.219 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:03:33.220 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:03:33.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:03:33.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:03:33.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:03:33.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:03:33.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:03:33.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:03:33.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:03:33.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:03:33.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:03:34.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:03:34.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:03:34.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-09-18T14:03:34.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:03:34.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-18T14:03:34.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:03:34.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:03:34.662 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:03:34.662 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:03:34.662 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:03:34.662 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:03:34.662 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:03:34.662 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:03:35.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:03:35.027 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:03:35.028 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:03:35.028 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:03:35.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:03:35.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:03:35.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:03:35.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:03:35.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:03:35.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:03:35.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:03:35.529 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:03:35.529 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:03:36.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:03:36.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:03:36.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-09-18T14:03:36.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:03:36.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-09-18T14:03:36.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:03:36.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:03:36.910 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:03:36.910 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:03:36.911 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:03:36.911 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:03:36.911 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:03:36.911 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:03:37.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:03:37.281 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:03:37.281 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:03:37.281 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:03:37.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:03:37.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:03:37.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:03:37.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:03:37.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:03:37.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:03:37.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:03:37.784 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:03:37.784 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:03:38.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:03:38.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:03:38.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-09-18T14:03:38.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:03:38.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-09-18T14:03:39.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:03:39.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:03:39.889 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:03:39.889 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:03:39.889 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:03:39.889 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:03:39.889 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:03:39.889 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:03:40.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:03:40.271 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:03:40.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:03:40.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:03:40.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:03:40.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:03:40.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:03:40.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:03:40.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:03:40.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:03:40.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:03:40.769 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:03:40.770 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:03:41.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:03:41.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:03:41.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-09-18T14:03:41.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:03:41.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-09-18T14:03:44.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:03:44.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:03:44.476 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:03:44.476 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:03:44.476 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:03:44.476 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:03:44.476 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:03:44.476 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:03:44.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:03:44.857 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:03:44.857 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:03:44.858 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:03:45.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:03:45.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:03:45.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:03:45.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:03:45.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:03:45.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:03:45.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:03:45.353 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:03:45.353 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:03:45.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:03:45.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:03:45.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-09-18T14:03:45.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:03:45.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-09-18T14:03:52.299 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:03:52.299 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:03:52.300 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:03:52.300 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:03:52.300 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:03:52.300 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:03:52.300 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:03:52.300 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:03:52.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:03:52.674 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:03:52.674 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:03:52.674 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:03:53.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:03:53.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:03:53.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:03:53.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:03:53.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:03:53.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:03:53.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:03:53.170 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:03:53.170 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:03:53.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:03:53.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:03:53.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-09-18T14:03:53.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:03:53.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-09-18T14:04:00.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:04:00.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:04:00.999 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:04:00.999 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:04:00.999 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:04:01.000 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:04:01.000 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:04:01.000 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:04:01.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:04:01.383 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:04:01.383 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:04:01.383 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:04:01.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:04:01.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:04:01.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:04:01.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:04:01.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:04:01.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:04:01.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:04:01.895 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:04:01.895 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:04:02.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:04:02.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:04:02.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-09-18T14:04:02.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-18T14:04:02.934 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T14:04:02.934 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_fingerprint": "20585fb2-188b-4e34-98c2-e62d5079ffd5", 2024-09-18T14:04:02.934 INFO:tasks.workunit.client.0.smithi047.stdout: "version": "17.2.7-1645-g7061ea6c", 2024-09-18T14:04:02.934 INFO:tasks.workunit.client.0.smithi047.stdout: "commit": "7061ea6ca50908d104e9f3979dc615e118d9e153", 2024-09-18T14:04:02.935 INFO:tasks.workunit.client.0.smithi047.stdout: "timestamp": "2024-09-18T14:04:02.933928+0000", 2024-09-18T14:04:02.935 INFO:tasks.workunit.client.0.smithi047.stdout: "tag": "", 2024-09-18T14:04:02.935 INFO:tasks.workunit.client.0.smithi047.stdout: "health": { 2024-09-18T14:04:02.935 INFO:tasks.workunit.client.0.smithi047.stdout: "status": "HEALTH_WARN", 2024-09-18T14:04:02.935 INFO:tasks.workunit.client.0.smithi047.stdout: "checks": { 2024-09-18T14:04:02.935 INFO:tasks.workunit.client.0.smithi047.stdout: "OSDMAP_FLAGS": { 2024-09-18T14:04:02.935 INFO:tasks.workunit.client.0.smithi047.stdout: "severity": "HEALTH_WARN", 2024-09-18T14:04:02.935 INFO:tasks.workunit.client.0.smithi047.stdout: "summary": { 2024-09-18T14:04:02.935 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-18T14:04:02.935 INFO:tasks.workunit.client.0.smithi047.stdout: "count": 17 2024-09-18T14:04:02.935 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.935 INFO:tasks.workunit.client.0.smithi047.stdout: "detail": [], 2024-09-18T14:04:02.935 INFO:tasks.workunit.client.0.smithi047.stdout: "muted": false 2024-09-18T14:04:02.936 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.936 INFO:tasks.workunit.client.0.smithi047.stdout: "OSD_DOWN": { 2024-09-18T14:04:02.936 INFO:tasks.workunit.client.0.smithi047.stdout: "severity": "HEALTH_WARN", 2024-09-18T14:04:02.936 INFO:tasks.workunit.client.0.smithi047.stdout: "summary": { 2024-09-18T14:04:02.936 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "1 osds down", 2024-09-18T14:04:02.936 INFO:tasks.workunit.client.0.smithi047.stdout: "count": 1 2024-09-18T14:04:02.936 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.936 INFO:tasks.workunit.client.0.smithi047.stdout: "detail": [ 2024-09-18T14:04:02.936 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.936 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "osd.1 (root=default,host=smithi047) is down" 2024-09-18T14:04:02.936 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:02.936 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:04:02.936 INFO:tasks.workunit.client.0.smithi047.stdout: "muted": false 2024-09-18T14:04:02.937 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.937 INFO:tasks.workunit.client.0.smithi047.stdout: "PG_DEGRADED": { 2024-09-18T14:04:02.937 INFO:tasks.workunit.client.0.smithi047.stdout: "severity": "HEALTH_WARN", 2024-09-18T14:04:02.937 INFO:tasks.workunit.client.0.smithi047.stdout: "summary": { 2024-09-18T14:04:02.937 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded, 1 pg undersized", 2024-09-18T14:04:02.937 INFO:tasks.workunit.client.0.smithi047.stdout: "count": 2 2024-09-18T14:04:02.937 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.937 INFO:tasks.workunit.client.0.smithi047.stdout: "detail": [ 2024-09-18T14:04:02.937 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.937 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.0 is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2024-09-18T14:04:02.937 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:02.937 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:04:02.937 INFO:tasks.workunit.client.0.smithi047.stdout: "muted": false 2024-09-18T14:04:02.937 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.938 INFO:tasks.workunit.client.0.smithi047.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-18T14:04:02.938 INFO:tasks.workunit.client.0.smithi047.stdout: "severity": "HEALTH_WARN", 2024-09-18T14:04:02.938 INFO:tasks.workunit.client.0.smithi047.stdout: "summary": { 2024-09-18T14:04:02.938 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-18T14:04:02.938 INFO:tasks.workunit.client.0.smithi047.stdout: "count": 1 2024-09-18T14:04:02.938 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.938 INFO:tasks.workunit.client.0.smithi047.stdout: "detail": [ 2024-09-18T14:04:02.938 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.938 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "application not enabled on pool 'test'" 2024-09-18T14:04:02.938 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.938 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.938 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-18T14:04:02.938 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:02.938 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:04:02.938 INFO:tasks.workunit.client.0.smithi047.stdout: "muted": false 2024-09-18T14:04:02.939 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:02.939 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.939 INFO:tasks.workunit.client.0.smithi047.stdout: "mutes": [] 2024-09-18T14:04:02.939 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.939 INFO:tasks.workunit.client.0.smithi047.stdout: "monmap_first_committed": 1, 2024-09-18T14:04:02.939 INFO:tasks.workunit.client.0.smithi047.stdout: "monmap_last_committed": 1, 2024-09-18T14:04:02.939 INFO:tasks.workunit.client.0.smithi047.stdout: "monmap": { 2024-09-18T14:04:02.939 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 1, 2024-09-18T14:04:02.939 INFO:tasks.workunit.client.0.smithi047.stdout: "fsid": "d7330bd7-8062-424a-bbd8-8c587903c456", 2024-09-18T14:04:02.939 INFO:tasks.workunit.client.0.smithi047.stdout: "modified": "2024-09-18T14:00:11.473527Z", 2024-09-18T14:04:02.939 INFO:tasks.workunit.client.0.smithi047.stdout: "created": "2024-09-18T14:00:11.473527Z", 2024-09-18T14:04:02.939 INFO:tasks.workunit.client.0.smithi047.stdout: "min_mon_release": 17, 2024-09-18T14:04:02.939 INFO:tasks.workunit.client.0.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-18T14:04:02.939 INFO:tasks.workunit.client.0.smithi047.stdout: "election_strategy": 1, 2024-09-18T14:04:02.940 INFO:tasks.workunit.client.0.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-18T14:04:02.940 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_mode": false, 2024-09-18T14:04:02.940 INFO:tasks.workunit.client.0.smithi047.stdout: "tiebreaker_mon": "", 2024-09-18T14:04:02.940 INFO:tasks.workunit.client.0.smithi047.stdout: "removed_ranks: ": "", 2024-09-18T14:04:02.940 INFO:tasks.workunit.client.0.smithi047.stdout: "features": { 2024-09-18T14:04:02.940 INFO:tasks.workunit.client.0.smithi047.stdout: "persistent": [ 2024-09-18T14:04:02.940 INFO:tasks.workunit.client.0.smithi047.stdout: "kraken", 2024-09-18T14:04:02.940 INFO:tasks.workunit.client.0.smithi047.stdout: "luminous", 2024-09-18T14:04:02.940 INFO:tasks.workunit.client.0.smithi047.stdout: "mimic", 2024-09-18T14:04:02.940 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap-prune", 2024-09-18T14:04:02.940 INFO:tasks.workunit.client.0.smithi047.stdout: "nautilus", 2024-09-18T14:04:02.940 INFO:tasks.workunit.client.0.smithi047.stdout: "octopus", 2024-09-18T14:04:02.940 INFO:tasks.workunit.client.0.smithi047.stdout: "pacific", 2024-09-18T14:04:02.941 INFO:tasks.workunit.client.0.smithi047.stdout: "elector-pinging", 2024-09-18T14:04:02.941 INFO:tasks.workunit.client.0.smithi047.stdout: "quincy" 2024-09-18T14:04:02.941 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:04:02.941 INFO:tasks.workunit.client.0.smithi047.stdout: "optional": [] 2024-09-18T14:04:02.941 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.941 INFO:tasks.workunit.client.0.smithi047.stdout: "mons": [ 2024-09-18T14:04:02.941 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.941 INFO:tasks.workunit.client.0.smithi047.stdout: "rank": 0, 2024-09-18T14:04:02.941 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "a", 2024-09-18T14:04:02.941 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addrs": { 2024-09-18T14:04:02.941 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:04:02.941 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.941 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:04:02.941 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:7115", 2024-09-18T14:04:02.942 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 0 2024-09-18T14:04:02.942 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:02.942 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:04:02.942 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.942 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:7115/0", 2024-09-18T14:04:02.942 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-18T14:04:02.942 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 0, 2024-09-18T14:04:02.942 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 0, 2024-09-18T14:04:02.942 INFO:tasks.workunit.client.0.smithi047.stdout: "crush_location": "{}" 2024-09-18T14:04:02.942 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:02.942 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:04:02.942 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.942 INFO:tasks.workunit.client.0.smithi047.stdout: "quorum": [ 2024-09-18T14:04:02.942 INFO:tasks.workunit.client.0.smithi047.stdout: 0 2024-09-18T14:04:02.943 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:04:02.943 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap": { 2024-09-18T14:04:02.943 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 41, 2024-09-18T14:04:02.943 INFO:tasks.workunit.client.0.smithi047.stdout: "fsid": "d7330bd7-8062-424a-bbd8-8c587903c456", 2024-09-18T14:04:02.943 INFO:tasks.workunit.client.0.smithi047.stdout: "created": "2024-09-18T14:00:12.273086+0000", 2024-09-18T14:04:02.943 INFO:tasks.workunit.client.0.smithi047.stdout: "modified": "2024-09-18T14:02:50.040549+0000", 2024-09-18T14:04:02.943 INFO:tasks.workunit.client.0.smithi047.stdout: "last_up_change": "2024-09-18T14:02:47.419439+0000", 2024-09-18T14:04:02.943 INFO:tasks.workunit.client.0.smithi047.stdout: "last_in_change": "2024-09-18T14:02:49.728019+0000", 2024-09-18T14:04:02.943 INFO:tasks.workunit.client.0.smithi047.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-18T14:04:02.943 INFO:tasks.workunit.client.0.smithi047.stdout: "flags_num": 5800384, 2024-09-18T14:04:02.943 INFO:tasks.workunit.client.0.smithi047.stdout: "flags_set": [ 2024-09-18T14:04:02.943 INFO:tasks.workunit.client.0.smithi047.stdout: "nodown", 2024-09-18T14:04:02.943 INFO:tasks.workunit.client.0.smithi047.stdout: "noin", 2024-09-18T14:04:02.943 INFO:tasks.workunit.client.0.smithi047.stdout: "noout", 2024-09-18T14:04:02.943 INFO:tasks.workunit.client.0.smithi047.stdout: "pglog_hardlimit", 2024-09-18T14:04:02.944 INFO:tasks.workunit.client.0.smithi047.stdout: "purged_snapdirs", 2024-09-18T14:04:02.944 INFO:tasks.workunit.client.0.smithi047.stdout: "recovery_deletes", 2024-09-18T14:04:02.944 INFO:tasks.workunit.client.0.smithi047.stdout: "sortbitwise" 2024-09-18T14:04:02.944 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:04:02.944 INFO:tasks.workunit.client.0.smithi047.stdout: "crush_version": 9, 2024-09-18T14:04:02.944 INFO:tasks.workunit.client.0.smithi047.stdout: "full_ratio": 0.99000000953674316, 2024-09-18T14:04:02.944 INFO:tasks.workunit.client.0.smithi047.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-18T14:04:02.944 INFO:tasks.workunit.client.0.smithi047.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-18T14:04:02.944 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_snapshot": "", 2024-09-18T14:04:02.944 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_max": 1, 2024-09-18T14:04:02.944 INFO:tasks.workunit.client.0.smithi047.stdout: "max_osd": 3, 2024-09-18T14:04:02.944 INFO:tasks.workunit.client.0.smithi047.stdout: "require_min_compat_client": "luminous", 2024-09-18T14:04:02.944 INFO:tasks.workunit.client.0.smithi047.stdout: "min_compat_client": "jewel", 2024-09-18T14:04:02.945 INFO:tasks.workunit.client.0.smithi047.stdout: "require_osd_release": "quincy", 2024-09-18T14:04:02.945 INFO:tasks.workunit.client.0.smithi047.stdout: "pools": [ 2024-09-18T14:04:02.945 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.945 INFO:tasks.workunit.client.0.smithi047.stdout: "pool": 1, 2024-09-18T14:04:02.945 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_name": "test", 2024-09-18T14:04:02.945 INFO:tasks.workunit.client.0.smithi047.stdout: "create_time": "2024-09-18T14:01:53.733083+0000", 2024-09-18T14:04:02.945 INFO:tasks.workunit.client.0.smithi047.stdout: "flags": 1, 2024-09-18T14:04:02.945 INFO:tasks.workunit.client.0.smithi047.stdout: "flags_names": "hashpspool", 2024-09-18T14:04:02.945 INFO:tasks.workunit.client.0.smithi047.stdout: "type": 1, 2024-09-18T14:04:02.945 INFO:tasks.workunit.client.0.smithi047.stdout: "size": 3, 2024-09-18T14:04:02.945 INFO:tasks.workunit.client.0.smithi047.stdout: "min_size": 2, 2024-09-18T14:04:02.945 INFO:tasks.workunit.client.0.smithi047.stdout: "crush_rule": 0, 2024-09-18T14:04:02.945 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_count": 0, 2024-09-18T14:04:02.945 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_target": 0, 2024-09-18T14:04:02.946 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T14:04:02.946 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T14:04:02.946 INFO:tasks.workunit.client.0.smithi047.stdout: "object_hash": 2, 2024-09-18T14:04:02.946 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_autoscale_mode": "off", 2024-09-18T14:04:02.946 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_num": 1, 2024-09-18T14:04:02.946 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_placement_num": 1, 2024-09-18T14:04:02.946 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_placement_num_target": 1, 2024-09-18T14:04:02.946 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_num_target": 1, 2024-09-18T14:04:02.946 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_num_pending": 1, 2024-09-18T14:04:02.946 INFO:tasks.workunit.client.0.smithi047.stdout: "last_pg_merge_meta": { 2024-09-18T14:04:02.946 INFO:tasks.workunit.client.0.smithi047.stdout: "source_pgid": "0.0", 2024-09-18T14:04:02.946 INFO:tasks.workunit.client.0.smithi047.stdout: "ready_epoch": 0, 2024-09-18T14:04:02.946 INFO:tasks.workunit.client.0.smithi047.stdout: "last_epoch_started": 0, 2024-09-18T14:04:02.946 INFO:tasks.workunit.client.0.smithi047.stdout: "last_epoch_clean": 0, 2024-09-18T14:04:02.946 INFO:tasks.workunit.client.0.smithi047.stdout: "source_version": "0'0", 2024-09-18T14:04:02.947 INFO:tasks.workunit.client.0.smithi047.stdout: "target_version": "0'0" 2024-09-18T14:04:02.947 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.947 INFO:tasks.workunit.client.0.smithi047.stdout: "last_change": "29", 2024-09-18T14:04:02.947 INFO:tasks.workunit.client.0.smithi047.stdout: "last_force_op_resend": "0", 2024-09-18T14:04:02.947 INFO:tasks.workunit.client.0.smithi047.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T14:04:02.947 INFO:tasks.workunit.client.0.smithi047.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T14:04:02.947 INFO:tasks.workunit.client.0.smithi047.stdout: "auid": 0, 2024-09-18T14:04:02.947 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_mode": "selfmanaged", 2024-09-18T14:04:02.947 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_seq": 0, 2024-09-18T14:04:02.947 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_epoch": 0, 2024-09-18T14:04:02.947 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_snaps": [], 2024-09-18T14:04:02.947 INFO:tasks.workunit.client.0.smithi047.stdout: "removed_snaps": "[]", 2024-09-18T14:04:02.947 INFO:tasks.workunit.client.0.smithi047.stdout: "quota_max_bytes": 0, 2024-09-18T14:04:02.947 INFO:tasks.workunit.client.0.smithi047.stdout: "quota_max_objects": 0, 2024-09-18T14:04:02.948 INFO:tasks.workunit.client.0.smithi047.stdout: "tiers": [], 2024-09-18T14:04:02.948 INFO:tasks.workunit.client.0.smithi047.stdout: "tier_of": -1, 2024-09-18T14:04:02.948 INFO:tasks.workunit.client.0.smithi047.stdout: "read_tier": -1, 2024-09-18T14:04:02.948 INFO:tasks.workunit.client.0.smithi047.stdout: "write_tier": -1, 2024-09-18T14:04:02.948 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_mode": "none", 2024-09-18T14:04:02.948 INFO:tasks.workunit.client.0.smithi047.stdout: "target_max_bytes": 0, 2024-09-18T14:04:02.948 INFO:tasks.workunit.client.0.smithi047.stdout: "target_max_objects": 0, 2024-09-18T14:04:02.948 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T14:04:02.948 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T14:04:02.948 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T14:04:02.948 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_min_flush_age": 0, 2024-09-18T14:04:02.948 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_min_evict_age": 0, 2024-09-18T14:04:02.948 INFO:tasks.workunit.client.0.smithi047.stdout: "erasure_code_profile": "", 2024-09-18T14:04:02.948 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_params": { 2024-09-18T14:04:02.948 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "none" 2024-09-18T14:04:02.949 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.949 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_period": 0, 2024-09-18T14:04:02.949 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_count": 0, 2024-09-18T14:04:02.949 INFO:tasks.workunit.client.0.smithi047.stdout: "use_gmt_hitset": true, 2024-09-18T14:04:02.949 INFO:tasks.workunit.client.0.smithi047.stdout: "min_read_recency_for_promote": 0, 2024-09-18T14:04:02.949 INFO:tasks.workunit.client.0.smithi047.stdout: "min_write_recency_for_promote": 0, 2024-09-18T14:04:02.949 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T14:04:02.949 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_search_last_n": 0, 2024-09-18T14:04:02.949 INFO:tasks.workunit.client.0.smithi047.stdout: "grade_table": [], 2024-09-18T14:04:02.949 INFO:tasks.workunit.client.0.smithi047.stdout: "stripe_width": 0, 2024-09-18T14:04:02.949 INFO:tasks.workunit.client.0.smithi047.stdout: "expected_num_objects": 0, 2024-09-18T14:04:02.949 INFO:tasks.workunit.client.0.smithi047.stdout: "fast_read": false, 2024-09-18T14:04:02.949 INFO:tasks.workunit.client.0.smithi047.stdout: "options": {}, 2024-09-18T14:04:02.949 INFO:tasks.workunit.client.0.smithi047.stdout: "application_metadata": {} 2024-09-18T14:04:02.950 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:02.950 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:04:02.950 INFO:tasks.workunit.client.0.smithi047.stdout: "osds": [ 2024-09-18T14:04:02.950 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.950 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 0, 2024-09-18T14:04:02.950 INFO:tasks.workunit.client.0.smithi047.stdout: "uuid": "df87ad03-46d9-487e-8f7a-aab30f82a1bd", 2024-09-18T14:04:02.950 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 1, 2024-09-18T14:04:02.950 INFO:tasks.workunit.client.0.smithi047.stdout: "in": 1, 2024-09-18T14:04:02.950 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 1, 2024-09-18T14:04:02.950 INFO:tasks.workunit.client.0.smithi047.stdout: "primary_affinity": 1, 2024-09-18T14:04:02.950 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_begin": 5, 2024-09-18T14:04:02.950 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_end": 29, 2024-09-18T14:04:02.950 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 36, 2024-09-18T14:04:02.950 INFO:tasks.workunit.client.0.smithi047.stdout: "up_thru": 40, 2024-09-18T14:04:02.950 INFO:tasks.workunit.client.0.smithi047.stdout: "down_at": 30, 2024-09-18T14:04:02.950 INFO:tasks.workunit.client.0.smithi047.stdout: "lost_at": 0, 2024-09-18T14:04:02.951 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addrs": { 2024-09-18T14:04:02.951 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:04:02.951 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.951 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:04:02.951 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6802", 2024-09-18T14:04:02.951 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 53956 2024-09-18T14:04:02.951 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.951 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.951 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:04:02.951 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6803", 2024-09-18T14:04:02.951 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 53956 2024-09-18T14:04:02.951 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:02.951 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:04:02.951 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.952 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addrs": { 2024-09-18T14:04:02.952 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:04:02.952 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.952 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:04:02.952 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6804", 2024-09-18T14:04:02.952 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 53956 2024-09-18T14:04:02.952 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.952 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.952 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:04:02.952 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6805", 2024-09-18T14:04:02.952 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 53956 2024-09-18T14:04:02.952 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:02.952 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:04:02.952 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.953 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addrs": { 2024-09-18T14:04:02.953 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:04:02.953 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.953 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:04:02.953 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6808", 2024-09-18T14:04:02.953 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 53956 2024-09-18T14:04:02.953 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.953 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.953 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:04:02.953 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6809", 2024-09-18T14:04:02.953 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 53956 2024-09-18T14:04:02.953 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:02.953 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:04:02.953 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.953 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addrs": { 2024-09-18T14:04:02.954 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:04:02.954 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.954 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:04:02.954 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6806", 2024-09-18T14:04:02.954 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 53956 2024-09-18T14:04:02.954 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.954 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.954 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:04:02.954 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6807", 2024-09-18T14:04:02.954 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 53956 2024-09-18T14:04:02.954 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:02.954 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:04:02.954 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.954 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addr": "127.0.0.1:6803/53956", 2024-09-18T14:04:02.955 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addr": "127.0.0.1:6805/53956", 2024-09-18T14:04:02.955 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addr": "127.0.0.1:6809/53956", 2024-09-18T14:04:02.955 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addr": "127.0.0.1:6807/53956", 2024-09-18T14:04:02.955 INFO:tasks.workunit.client.0.smithi047.stdout: "state": [ 2024-09-18T14:04:02.955 INFO:tasks.workunit.client.0.smithi047.stdout: "exists", 2024-09-18T14:04:02.955 INFO:tasks.workunit.client.0.smithi047.stdout: "up" 2024-09-18T14:04:02.955 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:04:02.955 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.955 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.955 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 1, 2024-09-18T14:04:02.955 INFO:tasks.workunit.client.0.smithi047.stdout: "uuid": "b8177855-da06-4ebd-9d12-538bef0d43b8", 2024-09-18T14:04:02.955 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 0, 2024-09-18T14:04:02.955 INFO:tasks.workunit.client.0.smithi047.stdout: "in": 1, 2024-09-18T14:04:02.955 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 1, 2024-09-18T14:04:02.955 INFO:tasks.workunit.client.0.smithi047.stdout: "primary_affinity": 1, 2024-09-18T14:04:02.956 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_begin": 0, 2024-09-18T14:04:02.956 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_end": 0, 2024-09-18T14:04:02.956 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 10, 2024-09-18T14:04:02.956 INFO:tasks.workunit.client.0.smithi047.stdout: "up_thru": 23, 2024-09-18T14:04:02.956 INFO:tasks.workunit.client.0.smithi047.stdout: "down_at": 31, 2024-09-18T14:04:02.956 INFO:tasks.workunit.client.0.smithi047.stdout: "lost_at": 0, 2024-09-18T14:04:02.956 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addrs": { 2024-09-18T14:04:02.956 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:04:02.956 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.956 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:04:02.957 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6810", 2024-09-18T14:04:02.957 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 47896 2024-09-18T14:04:02.957 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.957 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.957 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:04:02.957 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6811", 2024-09-18T14:04:02.957 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 47896 2024-09-18T14:04:02.957 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:02.957 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:04:02.957 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.957 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addrs": { 2024-09-18T14:04:02.957 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:04:02.957 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.957 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:04:02.958 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6812", 2024-09-18T14:04:02.958 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 47896 2024-09-18T14:04:02.958 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.958 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.958 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:04:02.958 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6813", 2024-09-18T14:04:02.958 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 47896 2024-09-18T14:04:02.958 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:02.958 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:04:02.958 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.958 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addrs": { 2024-09-18T14:04:02.958 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:04:02.958 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.958 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:04:02.958 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6816", 2024-09-18T14:04:02.959 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 47896 2024-09-18T14:04:02.959 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.959 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.959 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:04:02.959 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6817", 2024-09-18T14:04:02.959 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 47896 2024-09-18T14:04:02.959 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:02.959 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:04:02.959 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.959 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addrs": { 2024-09-18T14:04:02.959 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:04:02.959 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.959 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:04:02.959 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6814", 2024-09-18T14:04:02.959 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 47896 2024-09-18T14:04:02.960 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.960 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.960 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:04:02.960 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6815", 2024-09-18T14:04:02.960 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 47896 2024-09-18T14:04:02.960 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:02.960 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:04:02.960 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.960 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addr": "127.0.0.1:6811/47896", 2024-09-18T14:04:02.960 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addr": "127.0.0.1:6813/47896", 2024-09-18T14:04:02.960 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addr": "127.0.0.1:6817/47896", 2024-09-18T14:04:02.960 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addr": "127.0.0.1:6815/47896", 2024-09-18T14:04:02.960 INFO:tasks.workunit.client.0.smithi047.stdout: "state": [ 2024-09-18T14:04:02.960 INFO:tasks.workunit.client.0.smithi047.stdout: "exists" 2024-09-18T14:04:02.960 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:04:02.961 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.961 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.961 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 2, 2024-09-18T14:04:02.961 INFO:tasks.workunit.client.0.smithi047.stdout: "uuid": "dec93258-345a-406b-85ef-8316978cb8b7", 2024-09-18T14:04:02.961 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 1, 2024-09-18T14:04:02.961 INFO:tasks.workunit.client.0.smithi047.stdout: "in": 1, 2024-09-18T14:04:02.961 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 1, 2024-09-18T14:04:02.961 INFO:tasks.workunit.client.0.smithi047.stdout: "primary_affinity": 1, 2024-09-18T14:04:02.961 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_begin": 15, 2024-09-18T14:04:02.961 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_end": 29, 2024-09-18T14:04:02.961 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 38, 2024-09-18T14:04:02.961 INFO:tasks.workunit.client.0.smithi047.stdout: "up_thru": 0, 2024-09-18T14:04:02.961 INFO:tasks.workunit.client.0.smithi047.stdout: "down_at": 32, 2024-09-18T14:04:02.961 INFO:tasks.workunit.client.0.smithi047.stdout: "lost_at": 0, 2024-09-18T14:04:02.962 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addrs": { 2024-09-18T14:04:02.962 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:04:02.962 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.962 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:04:02.962 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6810", 2024-09-18T14:04:02.962 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 54526 2024-09-18T14:04:02.962 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.962 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.962 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:04:02.962 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6811", 2024-09-18T14:04:02.962 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 54526 2024-09-18T14:04:02.962 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:02.962 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:04:02.963 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.963 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addrs": { 2024-09-18T14:04:02.963 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:04:02.963 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.963 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:04:02.963 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6812", 2024-09-18T14:04:02.963 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 54526 2024-09-18T14:04:02.963 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.963 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.963 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:04:02.963 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6813", 2024-09-18T14:04:02.963 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 54526 2024-09-18T14:04:02.964 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:02.964 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:04:02.964 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.964 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addrs": { 2024-09-18T14:04:02.964 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:04:02.964 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.964 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:04:02.964 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6816", 2024-09-18T14:04:02.964 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 54526 2024-09-18T14:04:02.964 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.964 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.964 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:04:02.964 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6817", 2024-09-18T14:04:02.964 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 54526 2024-09-18T14:04:02.965 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:02.965 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:04:02.965 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.965 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addrs": { 2024-09-18T14:04:02.965 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:04:02.965 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.965 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:04:02.965 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6814", 2024-09-18T14:04:02.965 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 54526 2024-09-18T14:04:02.965 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.965 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.965 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:04:02.965 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6815", 2024-09-18T14:04:02.965 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 54526 2024-09-18T14:04:02.966 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:02.966 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:04:02.966 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.966 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addr": "127.0.0.1:6811/54526", 2024-09-18T14:04:02.966 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addr": "127.0.0.1:6813/54526", 2024-09-18T14:04:02.966 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addr": "127.0.0.1:6817/54526", 2024-09-18T14:04:02.966 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addr": "127.0.0.1:6815/54526", 2024-09-18T14:04:02.966 INFO:tasks.workunit.client.0.smithi047.stdout: "state": [ 2024-09-18T14:04:02.966 INFO:tasks.workunit.client.0.smithi047.stdout: "exists", 2024-09-18T14:04:02.966 INFO:tasks.workunit.client.0.smithi047.stdout: "up" 2024-09-18T14:04:02.966 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:04:02.966 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:02.966 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:04:02.966 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_xinfo": [ 2024-09-18T14:04:02.966 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.966 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 0, 2024-09-18T14:04:02.967 INFO:tasks.workunit.client.0.smithi047.stdout: "down_stamp": "2024-09-18T14:02:37.095760+0000", 2024-09-18T14:04:02.967 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_probability": 0, 2024-09-18T14:04:02.967 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_interval": 0, 2024-09-18T14:04:02.967 INFO:tasks.workunit.client.0.smithi047.stdout: "features": 4540138320759226367, 2024-09-18T14:04:02.967 INFO:tasks.workunit.client.0.smithi047.stdout: "old_weight": 0, 2024-09-18T14:04:02.967 INFO:tasks.workunit.client.0.smithi047.stdout: "last_purged_snaps_scrub": "2024-09-18T14:00:43.377185+0000", 2024-09-18T14:04:02.967 INFO:tasks.workunit.client.0.smithi047.stdout: "dead_epoch": 0 2024-09-18T14:04:02.967 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.967 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.967 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 1, 2024-09-18T14:04:02.967 INFO:tasks.workunit.client.0.smithi047.stdout: "down_stamp": "2024-09-18T14:02:37.655555+0000", 2024-09-18T14:04:02.967 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_probability": 0, 2024-09-18T14:04:02.967 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_interval": 0, 2024-09-18T14:04:02.967 INFO:tasks.workunit.client.0.smithi047.stdout: "features": 4540138320759226367, 2024-09-18T14:04:02.967 INFO:tasks.workunit.client.0.smithi047.stdout: "old_weight": 0, 2024-09-18T14:04:02.968 INFO:tasks.workunit.client.0.smithi047.stdout: "last_purged_snaps_scrub": "2024-09-18T14:01:17.318061+0000", 2024-09-18T14:04:02.968 INFO:tasks.workunit.client.0.smithi047.stdout: "dead_epoch": 0 2024-09-18T14:04:02.968 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.968 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.968 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 2, 2024-09-18T14:04:02.968 INFO:tasks.workunit.client.0.smithi047.stdout: "down_stamp": "2024-09-18T14:02:38.212290+0000", 2024-09-18T14:04:02.968 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_probability": 0, 2024-09-18T14:04:02.968 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_interval": 0, 2024-09-18T14:04:02.968 INFO:tasks.workunit.client.0.smithi047.stdout: "features": 4540138320759226367, 2024-09-18T14:04:02.968 INFO:tasks.workunit.client.0.smithi047.stdout: "old_weight": 0, 2024-09-18T14:04:02.968 INFO:tasks.workunit.client.0.smithi047.stdout: "last_purged_snaps_scrub": "2024-09-18T14:01:50.075324+0000", 2024-09-18T14:04:02.968 INFO:tasks.workunit.client.0.smithi047.stdout: "dead_epoch": 0 2024-09-18T14:04:02.968 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:02.968 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:04:02.969 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_upmap": [], 2024-09-18T14:04:02.969 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_upmap_items": [], 2024-09-18T14:04:02.969 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_temp": [], 2024-09-18T14:04:02.969 INFO:tasks.workunit.client.0.smithi047.stdout: "primary_temp": [], 2024-09-18T14:04:02.969 INFO:tasks.workunit.client.0.smithi047.stdout: "blocklist": {}, 2024-09-18T14:04:02.969 INFO:tasks.workunit.client.0.smithi047.stdout: "range_blocklist": {}, 2024-09-18T14:04:02.969 INFO:tasks.workunit.client.0.smithi047.stdout: "erasure_code_profiles": { 2024-09-18T14:04:02.969 INFO:tasks.workunit.client.0.smithi047.stdout: "default": { 2024-09-18T14:04:02.969 INFO:tasks.workunit.client.0.smithi047.stdout: "k": "2", 2024-09-18T14:04:02.969 INFO:tasks.workunit.client.0.smithi047.stdout: "m": "2", 2024-09-18T14:04:02.969 INFO:tasks.workunit.client.0.smithi047.stdout: "plugin": "jerasure", 2024-09-18T14:04:02.969 INFO:tasks.workunit.client.0.smithi047.stdout: "technique": "reed_sol_van" 2024-09-18T14:04:02.969 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:02.969 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.969 INFO:tasks.workunit.client.0.smithi047.stdout: "removed_snaps_queue": [], 2024-09-18T14:04:02.970 INFO:tasks.workunit.client.0.smithi047.stdout: "new_removed_snaps": [], 2024-09-18T14:04:02.970 INFO:tasks.workunit.client.0.smithi047.stdout: "new_purged_snaps": [], 2024-09-18T14:04:02.970 INFO:tasks.workunit.client.0.smithi047.stdout: "crush_node_flags": {}, 2024-09-18T14:04:02.970 INFO:tasks.workunit.client.0.smithi047.stdout: "device_class_flags": {}, 2024-09-18T14:04:02.970 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_mode": { 2024-09-18T14:04:02.970 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_mode_enabled": false, 2024-09-18T14:04:02.970 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_bucket_count": 0, 2024-09-18T14:04:02.970 INFO:tasks.workunit.client.0.smithi047.stdout: "degraded_stretch_mode": 0, 2024-09-18T14:04:02.970 INFO:tasks.workunit.client.0.smithi047.stdout: "recovering_stretch_mode": 0, 2024-09-18T14:04:02.970 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_mode_bucket": 0 2024-09-18T14:04:02.970 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:02.970 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.970 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_metadata": [ 2024-09-18T14:04:02.970 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.970 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T14:04:02.970 INFO:tasks.workunit.client.0.smithi047.stdout: "arch": "x86_64", 2024-09-18T14:04:02.971 INFO:tasks.workunit.client.0.smithi047.stdout: "back_addr": "[v2:127.0.0.1:6804/53956,v1:127.0.0.1:6805/53956]", 2024-09-18T14:04:02.971 INFO:tasks.workunit.client.0.smithi047.stdout: "back_iface": "", 2024-09-18T14:04:02.971 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs": "1", 2024-09-18T14:04:02.971 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_db": "0", 2024-09-18T14:04:02.971 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_wal": "0", 2024-09-18T14:04:02.971 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_single_shared_device": "1", 2024-09-18T14:04:02.971 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_allocation_from_file": "0", 2024-09-18T14:04:02.971 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_access_mode": "file", 2024-09-18T14:04:02.971 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_block_size": "4096", 2024-09-18T14:04:02.971 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_devices": "sda", 2024-09-18T14:04:02.971 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-18T14:04:02.971 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-18T14:04:02.971 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-18T14:04:02.971 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_rotational": "1", 2024-09-18T14:04:02.971 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_size": "107374182400", 2024-09-18T14:04:02.972 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_support_discard": "1", 2024-09-18T14:04:02.972 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_type": "hdd", 2024-09-18T14:04:02.972 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_min_alloc_size": "4096", 2024-09-18T14:04:02.972 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_release": "quincy", 2024-09-18T14:04:02.972 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:04:02.972 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_short": "17.2.7-1645-g7061ea6c", 2024-09-18T14:04:02.972 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_when_created": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:04:02.972 INFO:tasks.workunit.client.0.smithi047.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-18T14:04:02.972 INFO:tasks.workunit.client.0.smithi047.stdout: "created_at": "2024-09-18T14:00:17.705689Z", 2024-09-18T14:04:02.972 INFO:tasks.workunit.client.0.smithi047.stdout: "default_device_class": "hdd", 2024-09-18T14:04:02.972 INFO:tasks.workunit.client.0.smithi047.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4HN80", 2024-09-18T14:04:02.972 INFO:tasks.workunit.client.0.smithi047.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-18T14:04:02.972 INFO:tasks.workunit.client.0.smithi047.stdout: "devices": "sda", 2024-09-18T14:04:02.972 INFO:tasks.workunit.client.0.smithi047.stdout: "distro": "centos", 2024-09-18T14:04:02.972 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_description": "CentOS Stream 9", 2024-09-18T14:04:02.973 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_version": "9", 2024-09-18T14:04:02.973 INFO:tasks.workunit.client.0.smithi047.stdout: "front_addr": "[v2:127.0.0.1:6802/53956,v1:127.0.0.1:6803/53956]", 2024-09-18T14:04:02.973 INFO:tasks.workunit.client.0.smithi047.stdout: "front_iface": "", 2024-09-18T14:04:02.973 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/53956,v1:127.0.0.1:6809/53956]", 2024-09-18T14:04:02.973 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/53956,v1:127.0.0.1:6807/53956]", 2024-09-18T14:04:02.973 INFO:tasks.workunit.client.0.smithi047.stdout: "hostname": "smithi047", 2024-09-18T14:04:02.973 INFO:tasks.workunit.client.0.smithi047.stdout: "journal_rotational": "1", 2024-09-18T14:04:02.973 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-18T14:04:02.973 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-18T14:04:02.973 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_swap_kb": "0", 2024-09-18T14:04:02.974 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_total_kb": "32493492", 2024-09-18T14:04:02.974 INFO:tasks.workunit.client.0.smithi047.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-18T14:04:02.974 INFO:tasks.workunit.client.0.smithi047.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-18T14:04:02.974 INFO:tasks.workunit.client.0.smithi047.stdout: "os": "Linux", 2024-09-18T14:04:02.974 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_data": "td/divergent-priors/0", 2024-09-18T14:04:02.974 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_objectstore": "bluestore", 2024-09-18T14:04:02.974 INFO:tasks.workunit.client.0.smithi047.stdout: "osdspec_affinity": "", 2024-09-18T14:04:02.974 INFO:tasks.workunit.client.0.smithi047.stdout: "rotational": "1" 2024-09-18T14:04:02.974 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.974 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.974 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 1, 2024-09-18T14:04:02.974 INFO:tasks.workunit.client.0.smithi047.stdout: "arch": "x86_64", 2024-09-18T14:04:02.974 INFO:tasks.workunit.client.0.smithi047.stdout: "back_addr": "[v2:127.0.0.1:6812/47896,v1:127.0.0.1:6813/47896]", 2024-09-18T14:04:02.974 INFO:tasks.workunit.client.0.smithi047.stdout: "back_iface": "", 2024-09-18T14:04:02.974 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs": "1", 2024-09-18T14:04:02.975 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_db": "0", 2024-09-18T14:04:02.975 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_wal": "0", 2024-09-18T14:04:02.975 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_single_shared_device": "1", 2024-09-18T14:04:02.975 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_allocation_from_file": "0", 2024-09-18T14:04:02.975 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_access_mode": "file", 2024-09-18T14:04:02.975 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_block_size": "4096", 2024-09-18T14:04:02.975 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_devices": "sda", 2024-09-18T14:04:02.975 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-18T14:04:02.975 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-18T14:04:02.975 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-18T14:04:02.975 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_rotational": "1", 2024-09-18T14:04:02.975 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_size": "107374182400", 2024-09-18T14:04:02.975 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_support_discard": "1", 2024-09-18T14:04:02.975 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_type": "hdd", 2024-09-18T14:04:02.975 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_min_alloc_size": "4096", 2024-09-18T14:04:02.976 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_release": "quincy", 2024-09-18T14:04:02.976 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:04:02.976 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_short": "17.2.7-1645-g7061ea6c", 2024-09-18T14:04:02.976 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_when_created": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:04:02.976 INFO:tasks.workunit.client.0.smithi047.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-18T14:04:02.976 INFO:tasks.workunit.client.0.smithi047.stdout: "created_at": "2024-09-18T14:00:50.562608Z", 2024-09-18T14:04:02.976 INFO:tasks.workunit.client.0.smithi047.stdout: "default_device_class": "hdd", 2024-09-18T14:04:02.976 INFO:tasks.workunit.client.0.smithi047.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4HN80", 2024-09-18T14:04:02.976 INFO:tasks.workunit.client.0.smithi047.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-18T14:04:02.976 INFO:tasks.workunit.client.0.smithi047.stdout: "devices": "sda", 2024-09-18T14:04:02.976 INFO:tasks.workunit.client.0.smithi047.stdout: "distro": "centos", 2024-09-18T14:04:02.976 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_description": "CentOS Stream 9", 2024-09-18T14:04:02.976 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_version": "9", 2024-09-18T14:04:02.976 INFO:tasks.workunit.client.0.smithi047.stdout: "front_addr": "[v2:127.0.0.1:6810/47896,v1:127.0.0.1:6811/47896]", 2024-09-18T14:04:02.977 INFO:tasks.workunit.client.0.smithi047.stdout: "front_iface": "", 2024-09-18T14:04:02.977 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/47896,v1:127.0.0.1:6817/47896]", 2024-09-18T14:04:02.977 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/47896,v1:127.0.0.1:6815/47896]", 2024-09-18T14:04:02.977 INFO:tasks.workunit.client.0.smithi047.stdout: "hostname": "smithi047", 2024-09-18T14:04:02.977 INFO:tasks.workunit.client.0.smithi047.stdout: "journal_rotational": "1", 2024-09-18T14:04:02.977 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-18T14:04:02.977 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-18T14:04:02.977 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_swap_kb": "0", 2024-09-18T14:04:02.977 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_total_kb": "32493492", 2024-09-18T14:04:02.977 INFO:tasks.workunit.client.0.smithi047.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-18T14:04:02.977 INFO:tasks.workunit.client.0.smithi047.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-18T14:04:02.977 INFO:tasks.workunit.client.0.smithi047.stdout: "os": "Linux", 2024-09-18T14:04:02.977 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_data": "td/divergent-priors/1", 2024-09-18T14:04:02.978 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_objectstore": "bluestore", 2024-09-18T14:04:02.978 INFO:tasks.workunit.client.0.smithi047.stdout: "osdspec_affinity": "", 2024-09-18T14:04:02.978 INFO:tasks.workunit.client.0.smithi047.stdout: "rotational": "1" 2024-09-18T14:04:02.978 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.978 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.978 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T14:04:02.978 INFO:tasks.workunit.client.0.smithi047.stdout: "arch": "x86_64", 2024-09-18T14:04:02.978 INFO:tasks.workunit.client.0.smithi047.stdout: "back_addr": "[v2:127.0.0.1:6812/54526,v1:127.0.0.1:6813/54526]", 2024-09-18T14:04:02.978 INFO:tasks.workunit.client.0.smithi047.stdout: "back_iface": "", 2024-09-18T14:04:02.978 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs": "1", 2024-09-18T14:04:02.978 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_db": "0", 2024-09-18T14:04:02.978 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_wal": "0", 2024-09-18T14:04:02.978 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_single_shared_device": "1", 2024-09-18T14:04:02.978 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_allocation_from_file": "0", 2024-09-18T14:04:02.979 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_access_mode": "file", 2024-09-18T14:04:02.979 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_block_size": "4096", 2024-09-18T14:04:02.979 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_devices": "sda", 2024-09-18T14:04:02.979 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-18T14:04:02.979 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-18T14:04:02.979 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-18T14:04:02.979 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_rotational": "1", 2024-09-18T14:04:02.979 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_size": "107374182400", 2024-09-18T14:04:02.979 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_support_discard": "1", 2024-09-18T14:04:02.979 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_type": "hdd", 2024-09-18T14:04:02.979 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_min_alloc_size": "4096", 2024-09-18T14:04:02.979 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_release": "quincy", 2024-09-18T14:04:02.979 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:04:02.979 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_short": "17.2.7-1645-g7061ea6c", 2024-09-18T14:04:02.980 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_when_created": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:04:02.980 INFO:tasks.workunit.client.0.smithi047.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-18T14:04:02.980 INFO:tasks.workunit.client.0.smithi047.stdout: "created_at": "2024-09-18T14:01:23.854123Z", 2024-09-18T14:04:02.980 INFO:tasks.workunit.client.0.smithi047.stdout: "default_device_class": "hdd", 2024-09-18T14:04:02.980 INFO:tasks.workunit.client.0.smithi047.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4HN80", 2024-09-18T14:04:02.980 INFO:tasks.workunit.client.0.smithi047.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-18T14:04:02.980 INFO:tasks.workunit.client.0.smithi047.stdout: "devices": "sda", 2024-09-18T14:04:02.980 INFO:tasks.workunit.client.0.smithi047.stdout: "distro": "centos", 2024-09-18T14:04:02.980 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_description": "CentOS Stream 9", 2024-09-18T14:04:02.980 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_version": "9", 2024-09-18T14:04:02.980 INFO:tasks.workunit.client.0.smithi047.stdout: "front_addr": "[v2:127.0.0.1:6810/54526,v1:127.0.0.1:6811/54526]", 2024-09-18T14:04:02.980 INFO:tasks.workunit.client.0.smithi047.stdout: "front_iface": "", 2024-09-18T14:04:02.980 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/54526,v1:127.0.0.1:6817/54526]", 2024-09-18T14:04:02.980 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/54526,v1:127.0.0.1:6815/54526]", 2024-09-18T14:04:02.981 INFO:tasks.workunit.client.0.smithi047.stdout: "hostname": "smithi047", 2024-09-18T14:04:02.981 INFO:tasks.workunit.client.0.smithi047.stdout: "journal_rotational": "1", 2024-09-18T14:04:02.981 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-18T14:04:02.981 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-18T14:04:02.981 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_swap_kb": "0", 2024-09-18T14:04:02.981 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_total_kb": "32493492", 2024-09-18T14:04:02.981 INFO:tasks.workunit.client.0.smithi047.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-18T14:04:02.981 INFO:tasks.workunit.client.0.smithi047.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-18T14:04:02.981 INFO:tasks.workunit.client.0.smithi047.stdout: "os": "Linux", 2024-09-18T14:04:02.981 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_data": "td/divergent-priors/2", 2024-09-18T14:04:02.981 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_objectstore": "bluestore", 2024-09-18T14:04:02.981 INFO:tasks.workunit.client.0.smithi047.stdout: "osdspec_affinity": "", 2024-09-18T14:04:02.981 INFO:tasks.workunit.client.0.smithi047.stdout: "rotational": "1" 2024-09-18T14:04:02.981 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:02.981 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:04:02.982 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap_clean_epochs": { 2024-09-18T14:04:02.982 INFO:tasks.workunit.client.0.smithi047.stdout: "min_last_epoch_clean": 0, 2024-09-18T14:04:02.982 INFO:tasks.workunit.client.0.smithi047.stdout: "last_epoch_clean": { 2024-09-18T14:04:02.982 INFO:tasks.workunit.client.0.smithi047.stdout: "per_pool": [] 2024-09-18T14:04:02.982 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.982 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_epochs": [ 2024-09-18T14:04:02.982 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.982 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T14:04:02.982 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 36 2024-09-18T14:04:02.982 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.982 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.982 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T14:04:02.982 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 38 2024-09-18T14:04:02.982 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:02.982 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:04:02.983 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.983 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap_first_committed": 1, 2024-09-18T14:04:02.983 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap_last_committed": 41, 2024-09-18T14:04:02.983 INFO:tasks.workunit.client.0.smithi047.stdout: "crushmap": { 2024-09-18T14:04:02.983 INFO:tasks.workunit.client.0.smithi047.stdout: "devices": [ 2024-09-18T14:04:02.983 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.983 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T14:04:02.983 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "osd.0", 2024-09-18T14:04:02.983 INFO:tasks.workunit.client.0.smithi047.stdout: "class": "hdd" 2024-09-18T14:04:02.983 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.983 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.983 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 1, 2024-09-18T14:04:02.983 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "osd.1", 2024-09-18T14:04:02.983 INFO:tasks.workunit.client.0.smithi047.stdout: "class": "hdd" 2024-09-18T14:04:02.984 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.984 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.984 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T14:04:02.984 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "osd.2", 2024-09-18T14:04:02.984 INFO:tasks.workunit.client.0.smithi047.stdout: "class": "hdd" 2024-09-18T14:04:02.984 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:02.984 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:04:02.984 INFO:tasks.workunit.client.0.smithi047.stdout: "types": [ 2024-09-18T14:04:02.984 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.984 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 0, 2024-09-18T14:04:02.984 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "osd" 2024-09-18T14:04:02.984 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.984 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.984 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 1, 2024-09-18T14:04:02.984 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "host" 2024-09-18T14:04:02.985 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.985 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.985 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 2, 2024-09-18T14:04:02.985 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "chassis" 2024-09-18T14:04:02.985 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.985 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.985 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 3, 2024-09-18T14:04:02.985 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "rack" 2024-09-18T14:04:02.985 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.985 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.985 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 4, 2024-09-18T14:04:02.985 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "row" 2024-09-18T14:04:02.985 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.985 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.985 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 5, 2024-09-18T14:04:02.986 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "pdu" 2024-09-18T14:04:02.986 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.986 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.986 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 6, 2024-09-18T14:04:02.986 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "pod" 2024-09-18T14:04:02.986 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.986 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.986 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 7, 2024-09-18T14:04:02.986 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "room" 2024-09-18T14:04:02.986 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.986 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.986 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 8, 2024-09-18T14:04:02.986 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "datacenter" 2024-09-18T14:04:02.986 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.987 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.987 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 9, 2024-09-18T14:04:02.987 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "zone" 2024-09-18T14:04:02.987 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.987 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.987 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 10, 2024-09-18T14:04:02.987 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "region" 2024-09-18T14:04:02.987 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.987 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.987 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 11, 2024-09-18T14:04:02.987 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "root" 2024-09-18T14:04:02.987 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:02.987 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:04:02.987 INFO:tasks.workunit.client.0.smithi047.stdout: "buckets": [ 2024-09-18T14:04:02.987 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.988 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -1, 2024-09-18T14:04:02.988 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "default", 2024-09-18T14:04:02.988 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 11, 2024-09-18T14:04:02.988 INFO:tasks.workunit.client.0.smithi047.stdout: "type_name": "root", 2024-09-18T14:04:02.988 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:04:02.988 INFO:tasks.workunit.client.0.smithi047.stdout: "alg": "straw2", 2024-09-18T14:04:02.988 INFO:tasks.workunit.client.0.smithi047.stdout: "hash": "rjenkins1", 2024-09-18T14:04:02.988 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T14:04:02.988 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.988 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -3, 2024-09-18T14:04:02.988 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:04:02.989 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 0 2024-09-18T14:04:02.989 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:02.989 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:04:02.989 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.989 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.989 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -2, 2024-09-18T14:04:02.989 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "default~hdd", 2024-09-18T14:04:02.989 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 11, 2024-09-18T14:04:02.989 INFO:tasks.workunit.client.0.smithi047.stdout: "type_name": "root", 2024-09-18T14:04:02.989 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:04:02.989 INFO:tasks.workunit.client.0.smithi047.stdout: "alg": "straw2", 2024-09-18T14:04:02.989 INFO:tasks.workunit.client.0.smithi047.stdout: "hash": "rjenkins1", 2024-09-18T14:04:02.989 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T14:04:02.990 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.990 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -4, 2024-09-18T14:04:02.990 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:04:02.990 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 0 2024-09-18T14:04:02.990 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:02.990 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:04:02.990 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.990 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.990 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -3, 2024-09-18T14:04:02.990 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "smithi047", 2024-09-18T14:04:02.990 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 1, 2024-09-18T14:04:02.990 INFO:tasks.workunit.client.0.smithi047.stdout: "type_name": "host", 2024-09-18T14:04:02.990 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:04:02.990 INFO:tasks.workunit.client.0.smithi047.stdout: "alg": "straw2", 2024-09-18T14:04:02.991 INFO:tasks.workunit.client.0.smithi047.stdout: "hash": "rjenkins1", 2024-09-18T14:04:02.991 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T14:04:02.991 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.991 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T14:04:02.991 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:04:02.991 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 0 2024-09-18T14:04:02.991 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.991 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.991 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 1, 2024-09-18T14:04:02.991 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:04:02.991 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 1 2024-09-18T14:04:02.991 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.991 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.991 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T14:04:02.992 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:04:02.992 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 2 2024-09-18T14:04:02.992 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:02.992 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:04:02.992 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.992 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.992 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -4, 2024-09-18T14:04:02.992 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "smithi047~hdd", 2024-09-18T14:04:02.992 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 1, 2024-09-18T14:04:02.992 INFO:tasks.workunit.client.0.smithi047.stdout: "type_name": "host", 2024-09-18T14:04:02.992 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:04:02.992 INFO:tasks.workunit.client.0.smithi047.stdout: "alg": "straw2", 2024-09-18T14:04:02.992 INFO:tasks.workunit.client.0.smithi047.stdout: "hash": "rjenkins1", 2024-09-18T14:04:02.992 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T14:04:02.993 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.993 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T14:04:02.993 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:04:02.993 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 0 2024-09-18T14:04:02.993 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.993 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.993 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 1, 2024-09-18T14:04:02.993 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:04:02.993 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 1 2024-09-18T14:04:02.993 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.993 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.993 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T14:04:02.993 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:04:02.993 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 2 2024-09-18T14:04:02.994 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:02.994 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:04:02.994 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:02.994 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:04:02.994 INFO:tasks.workunit.client.0.smithi047.stdout: "rules": [ 2024-09-18T14:04:02.994 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.994 INFO:tasks.workunit.client.0.smithi047.stdout: "rule_id": 0, 2024-09-18T14:04:02.994 INFO:tasks.workunit.client.0.smithi047.stdout: "rule_name": "replicated_rule", 2024-09-18T14:04:02.994 INFO:tasks.workunit.client.0.smithi047.stdout: "type": 1, 2024-09-18T14:04:02.994 INFO:tasks.workunit.client.0.smithi047.stdout: "steps": [ 2024-09-18T14:04:02.994 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.994 INFO:tasks.workunit.client.0.smithi047.stdout: "op": "take", 2024-09-18T14:04:02.994 INFO:tasks.workunit.client.0.smithi047.stdout: "item": -1, 2024-09-18T14:04:02.994 INFO:tasks.workunit.client.0.smithi047.stdout: "item_name": "default" 2024-09-18T14:04:02.995 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.995 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.995 INFO:tasks.workunit.client.0.smithi047.stdout: "op": "choose_firstn", 2024-09-18T14:04:02.995 INFO:tasks.workunit.client.0.smithi047.stdout: "num": 0, 2024-09-18T14:04:02.995 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "osd" 2024-09-18T14:04:02.995 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.995 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:02.995 INFO:tasks.workunit.client.0.smithi047.stdout: "op": "emit" 2024-09-18T14:04:02.995 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:02.995 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:04:02.995 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:02.995 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:04:02.995 INFO:tasks.workunit.client.0.smithi047.stdout: "tunables": { 2024-09-18T14:04:02.995 INFO:tasks.workunit.client.0.smithi047.stdout: "choose_local_tries": 0, 2024-09-18T14:04:02.995 INFO:tasks.workunit.client.0.smithi047.stdout: "choose_local_fallback_tries": 0, 2024-09-18T14:04:02.996 INFO:tasks.workunit.client.0.smithi047.stdout: "choose_total_tries": 50, 2024-09-18T14:04:02.996 INFO:tasks.workunit.client.0.smithi047.stdout: "chooseleaf_descend_once": 1, 2024-09-18T14:04:02.996 INFO:tasks.workunit.client.0.smithi047.stdout: "chooseleaf_vary_r": 1, 2024-09-18T14:04:02.996 INFO:tasks.workunit.client.0.smithi047.stdout: "chooseleaf_stable": 1, 2024-09-18T14:04:02.996 INFO:tasks.workunit.client.0.smithi047.stdout: "straw_calc_version": 1, 2024-09-18T14:04:02.996 INFO:tasks.workunit.client.0.smithi047.stdout: "allowed_bucket_algs": 54, 2024-09-18T14:04:02.996 INFO:tasks.workunit.client.0.smithi047.stdout: "profile": "jewel", 2024-09-18T14:04:02.996 INFO:tasks.workunit.client.0.smithi047.stdout: "optimal_tunables": 1, 2024-09-18T14:04:02.996 INFO:tasks.workunit.client.0.smithi047.stdout: "legacy_tunables": 0, 2024-09-18T14:04:02.996 INFO:tasks.workunit.client.0.smithi047.stdout: "minimum_required_version": "jewel", 2024-09-18T14:04:02.996 INFO:tasks.workunit.client.0.smithi047.stdout: "require_feature_tunables": 1, 2024-09-18T14:04:02.996 INFO:tasks.workunit.client.0.smithi047.stdout: "require_feature_tunables2": 1, 2024-09-18T14:04:02.996 INFO:tasks.workunit.client.0.smithi047.stdout: "has_v2_rules": 0, 2024-09-18T14:04:02.996 INFO:tasks.workunit.client.0.smithi047.stdout: "require_feature_tunables3": 1, 2024-09-18T14:04:02.997 INFO:tasks.workunit.client.0.smithi047.stdout: "has_v3_rules": 0, 2024-09-18T14:04:02.997 INFO:tasks.workunit.client.0.smithi047.stdout: "has_v4_buckets": 1, 2024-09-18T14:04:02.997 INFO:tasks.workunit.client.0.smithi047.stdout: "require_feature_tunables5": 1, 2024-09-18T14:04:02.997 INFO:tasks.workunit.client.0.smithi047.stdout: "has_v5_rules": 0 2024-09-18T14:04:02.997 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.997 INFO:tasks.workunit.client.0.smithi047.stdout: "choose_args": {} 2024-09-18T14:04:02.997 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.997 INFO:tasks.workunit.client.0.smithi047.stdout: "fsmap": { 2024-09-18T14:04:02.997 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 1, 2024-09-18T14:04:02.997 INFO:tasks.workunit.client.0.smithi047.stdout: "default_fscid": -1, 2024-09-18T14:04:02.997 INFO:tasks.workunit.client.0.smithi047.stdout: "compat": { 2024-09-18T14:04:02.997 INFO:tasks.workunit.client.0.smithi047.stdout: "compat": {}, 2024-09-18T14:04:02.997 INFO:tasks.workunit.client.0.smithi047.stdout: "ro_compat": {}, 2024-09-18T14:04:02.997 INFO:tasks.workunit.client.0.smithi047.stdout: "incompat": { 2024-09-18T14:04:02.997 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_1": "base v0.20", 2024-09-18T14:04:02.998 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_2": "client writeable ranges", 2024-09-18T14:04:02.998 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_3": "default file layouts on dirs", 2024-09-18T14:04:02.998 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_4": "dir inode in separate object", 2024-09-18T14:04:02.998 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_5": "mds uses versioned encoding", 2024-09-18T14:04:02.998 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-18T14:04:02.998 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_8": "no anchor table", 2024-09-18T14:04:02.998 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_9": "file layout v2", 2024-09-18T14:04:02.998 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_10": "snaprealm v2" 2024-09-18T14:04:02.998 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:02.998 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.998 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_flags": { 2024-09-18T14:04:02.998 INFO:tasks.workunit.client.0.smithi047.stdout: "enable_multiple": true, 2024-09-18T14:04:02.998 INFO:tasks.workunit.client.0.smithi047.stdout: "ever_enabled_multiple": true 2024-09-18T14:04:02.998 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.998 INFO:tasks.workunit.client.0.smithi047.stdout: "standbys": [], 2024-09-18T14:04:02.999 INFO:tasks.workunit.client.0.smithi047.stdout: "filesystems": [] 2024-09-18T14:04:02.999 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.999 INFO:tasks.workunit.client.0.smithi047.stdout: "mdsmap_first_committed": 1, 2024-09-18T14:04:02.999 INFO:tasks.workunit.client.0.smithi047.stdout: "mdsmap_last_committed": 1, 2024-09-18T14:04:02.999 INFO:tasks.workunit.client.0.smithi047.stdout: "auth": { 2024-09-18T14:04:02.999 INFO:tasks.workunit.client.0.smithi047.stdout: "first_committed": 1, 2024-09-18T14:04:02.999 INFO:tasks.workunit.client.0.smithi047.stdout: "last_committed": 5, 2024-09-18T14:04:02.999 INFO:tasks.workunit.client.0.smithi047.stdout: "num_secrets": 3 2024-09-18T14:04:02.999 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:02.999 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg": 1, 2024-09-18T14:04:02.999 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg_active": 1, 2024-09-18T14:04:02.999 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg_unknown": 0, 2024-09-18T14:04:02.999 INFO:tasks.workunit.client.0.smithi047.stdout: "num_osd": 3, 2024-09-18T14:04:03.000 INFO:tasks.workunit.client.0.smithi047.stderr:report 813825044 2024-09-18T14:04:03.000 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_sum": { 2024-09-18T14:04:03.000 INFO:tasks.workunit.client.0.smithi047.stdout: "stat_sum": { 2024-09-18T14:04:03.000 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes": 47971, 2024-09-18T14:04:03.000 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects": 100, 2024-09-18T14:04:03.000 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_clones": 0, 2024-09-18T14:04:03.001 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_copies": 300, 2024-09-18T14:04:03.001 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing_on_primary": 0, 2024-09-18T14:04:03.001 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing": 0, 2024-09-18T14:04:03.001 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_degraded": 100, 2024-09-18T14:04:03.001 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_misplaced": 0, 2024-09-18T14:04:03.001 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_unfound": 0, 2024-09-18T14:04:03.001 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_dirty": 100, 2024-09-18T14:04:03.001 INFO:tasks.workunit.client.0.smithi047.stdout: "num_whiteouts": 0, 2024-09-18T14:04:03.001 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read": 0, 2024-09-18T14:04:03.001 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read_kb": 0, 2024-09-18T14:04:03.001 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write": 101, 2024-09-18T14:04:03.001 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write_kb": 101, 2024-09-18T14:04:03.001 INFO:tasks.workunit.client.0.smithi047.stdout: "num_scrub_errors": 0, 2024-09-18T14:04:03.001 INFO:tasks.workunit.client.0.smithi047.stdout: "num_shallow_scrub_errors": 0, 2024-09-18T14:04:03.001 INFO:tasks.workunit.client.0.smithi047.stdout: "num_deep_scrub_errors": 0, 2024-09-18T14:04:03.002 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_recovered": 0, 2024-09-18T14:04:03.002 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_recovered": 0, 2024-09-18T14:04:03.002 INFO:tasks.workunit.client.0.smithi047.stdout: "num_keys_recovered": 0, 2024-09-18T14:04:03.002 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_omap": 0, 2024-09-18T14:04:03.002 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_hit_set_archive": 0, 2024-09-18T14:04:03.002 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_hit_set_archive": 0, 2024-09-18T14:04:03.002 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush": 0, 2024-09-18T14:04:03.002 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_kb": 0, 2024-09-18T14:04:03.002 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict": 0, 2024-09-18T14:04:03.002 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_kb": 0, 2024-09-18T14:04:03.002 INFO:tasks.workunit.client.0.smithi047.stdout: "num_promote": 0, 2024-09-18T14:04:03.002 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_high": 0, 2024-09-18T14:04:03.002 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_low": 0, 2024-09-18T14:04:03.002 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_some": 0, 2024-09-18T14:04:03.003 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_full": 0, 2024-09-18T14:04:03.003 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_pinned": 0, 2024-09-18T14:04:03.003 INFO:tasks.workunit.client.0.smithi047.stdout: "num_legacy_snapsets": 0, 2024-09-18T14:04:03.003 INFO:tasks.workunit.client.0.smithi047.stdout: "num_large_omap_objects": 0, 2024-09-18T14:04:03.003 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_manifest": 0, 2024-09-18T14:04:03.003 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_bytes": 0, 2024-09-18T14:04:03.003 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_keys": 0, 2024-09-18T14:04:03.003 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_repaired": 0 2024-09-18T14:04:03.003 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:03.003 INFO:tasks.workunit.client.0.smithi047.stdout: "store_stats": { 2024-09-18T14:04:03.003 INFO:tasks.workunit.client.0.smithi047.stdout: "total": 0, 2024-09-18T14:04:03.003 INFO:tasks.workunit.client.0.smithi047.stdout: "available": 0, 2024-09-18T14:04:03.003 INFO:tasks.workunit.client.0.smithi047.stdout: "internally_reserved": 0, 2024-09-18T14:04:03.003 INFO:tasks.workunit.client.0.smithi047.stdout: "allocated": 0, 2024-09-18T14:04:03.003 INFO:tasks.workunit.client.0.smithi047.stdout: "data_stored": 0, 2024-09-18T14:04:03.004 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed": 0, 2024-09-18T14:04:03.004 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_allocated": 0, 2024-09-18T14:04:03.004 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_original": 0, 2024-09-18T14:04:03.004 INFO:tasks.workunit.client.0.smithi047.stdout: "omap_allocated": 0, 2024-09-18T14:04:03.004 INFO:tasks.workunit.client.0.smithi047.stdout: "internal_metadata": 0 2024-09-18T14:04:03.004 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:03.004 INFO:tasks.workunit.client.0.smithi047.stdout: "log_size": 17, 2024-09-18T14:04:03.004 INFO:tasks.workunit.client.0.smithi047.stdout: "ondisk_log_size": 17, 2024-09-18T14:04:03.004 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 2, 2024-09-18T14:04:03.004 INFO:tasks.workunit.client.0.smithi047.stdout: "acting": 2, 2024-09-18T14:04:03.004 INFO:tasks.workunit.client.0.smithi047.stdout: "num_store_stats": 0 2024-09-18T14:04:03.004 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:03.004 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_sum": { 2024-09-18T14:04:03.004 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 0, 2024-09-18T14:04:03.004 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 0, 2024-09-18T14:04:03.005 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pgs": 3, 2024-09-18T14:04:03.005 INFO:tasks.workunit.client.0.smithi047.stdout: "num_osds": 3, 2024-09-18T14:04:03.005 INFO:tasks.workunit.client.0.smithi047.stdout: "num_per_pool_osds": 3, 2024-09-18T14:04:03.005 INFO:tasks.workunit.client.0.smithi047.stdout: "num_per_pool_omap_osds": 3, 2024-09-18T14:04:03.005 INFO:tasks.workunit.client.0.smithi047.stdout: "kb": 314572800, 2024-09-18T14:04:03.005 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used": 894628, 2024-09-18T14:04:03.005 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_data": 2104, 2024-09-18T14:04:03.005 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_omap": 10, 2024-09-18T14:04:03.005 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_meta": 892469, 2024-09-18T14:04:03.005 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_avail": 313678172, 2024-09-18T14:04:03.005 INFO:tasks.workunit.client.0.smithi047.stdout: "statfs": { 2024-09-18T14:04:03.005 INFO:tasks.workunit.client.0.smithi047.stdout: "total": 322122547200, 2024-09-18T14:04:03.005 INFO:tasks.workunit.client.0.smithi047.stdout: "available": 321206448128, 2024-09-18T14:04:03.005 INFO:tasks.workunit.client.0.smithi047.stdout: "internally_reserved": 0, 2024-09-18T14:04:03.005 INFO:tasks.workunit.client.0.smithi047.stdout: "allocated": 2154496, 2024-09-18T14:04:03.006 INFO:tasks.workunit.client.0.smithi047.stdout: "data_stored": 416423, 2024-09-18T14:04:03.006 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed": 0, 2024-09-18T14:04:03.006 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_allocated": 0, 2024-09-18T14:04:03.006 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_original": 0, 2024-09-18T14:04:03.006 INFO:tasks.workunit.client.0.smithi047.stdout: "omap_allocated": 10790, 2024-09-18T14:04:03.006 INFO:tasks.workunit.client.0.smithi047.stdout: "internal_metadata": 913888730 2024-09-18T14:04:03.006 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:03.006 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_peers": [], 2024-09-18T14:04:03.006 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_trim_queue_len": 0, 2024-09-18T14:04:03.006 INFO:tasks.workunit.client.0.smithi047.stdout: "num_snap_trimming": 0, 2024-09-18T14:04:03.006 INFO:tasks.workunit.client.0.smithi047.stdout: "num_shards_repaired": 0, 2024-09-18T14:04:03.006 INFO:tasks.workunit.client.0.smithi047.stdout: "op_queue_age_hist": { 2024-09-18T14:04:03.006 INFO:tasks.workunit.client.0.smithi047.stdout: "histogram": [], 2024-09-18T14:04:03.006 INFO:tasks.workunit.client.0.smithi047.stdout: "upper_bound": 1 2024-09-18T14:04:03.006 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:03.007 INFO:tasks.workunit.client.0.smithi047.stdout: "perf_stat": { 2024-09-18T14:04:03.007 INFO:tasks.workunit.client.0.smithi047.stdout: "commit_latency_ms": 0, 2024-09-18T14:04:03.007 INFO:tasks.workunit.client.0.smithi047.stdout: "apply_latency_ms": 0, 2024-09-18T14:04:03.007 INFO:tasks.workunit.client.0.smithi047.stdout: "commit_latency_ns": 0, 2024-09-18T14:04:03.007 INFO:tasks.workunit.client.0.smithi047.stdout: "apply_latency_ns": 0 2024-09-18T14:04:03.007 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:03.007 INFO:tasks.workunit.client.0.smithi047.stdout: "alerts": [], 2024-09-18T14:04:03.007 INFO:tasks.workunit.client.0.smithi047.stdout: "network_ping_times": [] 2024-09-18T14:04:03.007 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:03.007 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_sum_by_class": { 2024-09-18T14:04:03.007 INFO:tasks.workunit.client.0.smithi047.stdout: "hdd": { 2024-09-18T14:04:03.007 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 0, 2024-09-18T14:04:03.007 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 0, 2024-09-18T14:04:03.007 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pgs": 3, 2024-09-18T14:04:03.008 INFO:tasks.workunit.client.0.smithi047.stdout: "num_osds": 3, 2024-09-18T14:04:03.008 INFO:tasks.workunit.client.0.smithi047.stdout: "num_per_pool_osds": 3, 2024-09-18T14:04:03.008 INFO:tasks.workunit.client.0.smithi047.stdout: "num_per_pool_omap_osds": 3, 2024-09-18T14:04:03.008 INFO:tasks.workunit.client.0.smithi047.stdout: "kb": 314572800, 2024-09-18T14:04:03.008 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used": 894628, 2024-09-18T14:04:03.008 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_data": 2104, 2024-09-18T14:04:03.008 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_omap": 10, 2024-09-18T14:04:03.008 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_meta": 892469, 2024-09-18T14:04:03.008 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_avail": 313678172, 2024-09-18T14:04:03.008 INFO:tasks.workunit.client.0.smithi047.stdout: "statfs": { 2024-09-18T14:04:03.008 INFO:tasks.workunit.client.0.smithi047.stdout: "total": 322122547200, 2024-09-18T14:04:03.008 INFO:tasks.workunit.client.0.smithi047.stdout: "available": 321206448128, 2024-09-18T14:04:03.008 INFO:tasks.workunit.client.0.smithi047.stdout: "internally_reserved": 0, 2024-09-18T14:04:03.008 INFO:tasks.workunit.client.0.smithi047.stdout: "allocated": 2154496, 2024-09-18T14:04:03.009 INFO:tasks.workunit.client.0.smithi047.stdout: "data_stored": 416423, 2024-09-18T14:04:03.009 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed": 0, 2024-09-18T14:04:03.009 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_allocated": 0, 2024-09-18T14:04:03.009 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_original": 0, 2024-09-18T14:04:03.009 INFO:tasks.workunit.client.0.smithi047.stdout: "omap_allocated": 10790, 2024-09-18T14:04:03.009 INFO:tasks.workunit.client.0.smithi047.stdout: "internal_metadata": 913888730 2024-09-18T14:04:03.009 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:03.009 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_peers": [], 2024-09-18T14:04:03.009 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_trim_queue_len": 0, 2024-09-18T14:04:03.009 INFO:tasks.workunit.client.0.smithi047.stdout: "num_snap_trimming": 0, 2024-09-18T14:04:03.009 INFO:tasks.workunit.client.0.smithi047.stdout: "num_shards_repaired": 0, 2024-09-18T14:04:03.009 INFO:tasks.workunit.client.0.smithi047.stdout: "op_queue_age_hist": { 2024-09-18T14:04:03.009 INFO:tasks.workunit.client.0.smithi047.stdout: "histogram": [], 2024-09-18T14:04:03.010 INFO:tasks.workunit.client.0.smithi047.stdout: "upper_bound": 1 2024-09-18T14:04:03.010 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:03.010 INFO:tasks.workunit.client.0.smithi047.stdout: "perf_stat": { 2024-09-18T14:04:03.010 INFO:tasks.workunit.client.0.smithi047.stdout: "commit_latency_ms": 0, 2024-09-18T14:04:03.010 INFO:tasks.workunit.client.0.smithi047.stdout: "apply_latency_ms": 0, 2024-09-18T14:04:03.010 INFO:tasks.workunit.client.0.smithi047.stdout: "commit_latency_ns": 0, 2024-09-18T14:04:03.010 INFO:tasks.workunit.client.0.smithi047.stdout: "apply_latency_ns": 0 2024-09-18T14:04:03.010 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:03.010 INFO:tasks.workunit.client.0.smithi047.stdout: "alerts": [], 2024-09-18T14:04:03.010 INFO:tasks.workunit.client.0.smithi047.stdout: "network_ping_times": [] 2024-09-18T14:04:03.010 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:03.010 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:03.010 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_stats": [ 2024-09-18T14:04:03.010 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:03.011 INFO:tasks.workunit.client.0.smithi047.stdout: "poolid": 1, 2024-09-18T14:04:03.011 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg": 1, 2024-09-18T14:04:03.011 INFO:tasks.workunit.client.0.smithi047.stdout: "stat_sum": { 2024-09-18T14:04:03.011 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes": 47971, 2024-09-18T14:04:03.011 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects": 100, 2024-09-18T14:04:03.011 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_clones": 0, 2024-09-18T14:04:03.011 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_copies": 300, 2024-09-18T14:04:03.011 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing_on_primary": 0, 2024-09-18T14:04:03.011 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing": 0, 2024-09-18T14:04:03.011 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_degraded": 100, 2024-09-18T14:04:03.011 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_misplaced": 0, 2024-09-18T14:04:03.011 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_unfound": 0, 2024-09-18T14:04:03.011 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_dirty": 100, 2024-09-18T14:04:03.011 INFO:tasks.workunit.client.0.smithi047.stdout: "num_whiteouts": 0, 2024-09-18T14:04:03.011 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read": 0, 2024-09-18T14:04:03.012 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read_kb": 0, 2024-09-18T14:04:03.012 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write": 101, 2024-09-18T14:04:03.012 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write_kb": 101, 2024-09-18T14:04:03.012 INFO:tasks.workunit.client.0.smithi047.stdout: "num_scrub_errors": 0, 2024-09-18T14:04:03.012 INFO:tasks.workunit.client.0.smithi047.stdout: "num_shallow_scrub_errors": 0, 2024-09-18T14:04:03.012 INFO:tasks.workunit.client.0.smithi047.stdout: "num_deep_scrub_errors": 0, 2024-09-18T14:04:03.012 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_recovered": 0, 2024-09-18T14:04:03.012 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_recovered": 0, 2024-09-18T14:04:03.012 INFO:tasks.workunit.client.0.smithi047.stdout: "num_keys_recovered": 0, 2024-09-18T14:04:03.012 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_omap": 0, 2024-09-18T14:04:03.012 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_hit_set_archive": 0, 2024-09-18T14:04:03.012 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_hit_set_archive": 0, 2024-09-18T14:04:03.012 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush": 0, 2024-09-18T14:04:03.012 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_kb": 0, 2024-09-18T14:04:03.013 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict": 0, 2024-09-18T14:04:03.013 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_kb": 0, 2024-09-18T14:04:03.013 INFO:tasks.workunit.client.0.smithi047.stdout: "num_promote": 0, 2024-09-18T14:04:03.013 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_high": 0, 2024-09-18T14:04:03.013 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_low": 0, 2024-09-18T14:04:03.013 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_some": 0, 2024-09-18T14:04:03.013 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_full": 0, 2024-09-18T14:04:03.013 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_pinned": 0, 2024-09-18T14:04:03.013 INFO:tasks.workunit.client.0.smithi047.stdout: "num_legacy_snapsets": 0, 2024-09-18T14:04:03.013 INFO:tasks.workunit.client.0.smithi047.stdout: "num_large_omap_objects": 0, 2024-09-18T14:04:03.013 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_manifest": 0, 2024-09-18T14:04:03.013 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_bytes": 0, 2024-09-18T14:04:03.013 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_keys": 0, 2024-09-18T14:04:03.013 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_repaired": 0 2024-09-18T14:04:03.013 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:03.014 INFO:tasks.workunit.client.0.smithi047.stdout: "store_stats": { 2024-09-18T14:04:03.014 INFO:tasks.workunit.client.0.smithi047.stdout: "total": 0, 2024-09-18T14:04:03.014 INFO:tasks.workunit.client.0.smithi047.stdout: "available": 0, 2024-09-18T14:04:03.014 INFO:tasks.workunit.client.0.smithi047.stdout: "internally_reserved": 0, 2024-09-18T14:04:03.014 INFO:tasks.workunit.client.0.smithi047.stdout: "allocated": 1220608, 2024-09-18T14:04:03.014 INFO:tasks.workunit.client.0.smithi047.stdout: "data_stored": 142618, 2024-09-18T14:04:03.014 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed": 0, 2024-09-18T14:04:03.014 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_allocated": 0, 2024-09-18T14:04:03.014 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_original": 0, 2024-09-18T14:04:03.014 INFO:tasks.workunit.client.0.smithi047.stdout: "omap_allocated": 0, 2024-09-18T14:04:03.014 INFO:tasks.workunit.client.0.smithi047.stdout: "internal_metadata": 0 2024-09-18T14:04:03.014 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:03.014 INFO:tasks.workunit.client.0.smithi047.stdout: "log_size": 17, 2024-09-18T14:04:03.014 INFO:tasks.workunit.client.0.smithi047.stdout: "ondisk_log_size": 17, 2024-09-18T14:04:03.014 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 2, 2024-09-18T14:04:03.015 INFO:tasks.workunit.client.0.smithi047.stdout: "acting": 2, 2024-09-18T14:04:03.015 INFO:tasks.workunit.client.0.smithi047.stdout: "num_store_stats": 3 2024-09-18T14:04:03.015 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:03.015 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:04:03.015 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_stats": [ 2024-09-18T14:04:03.015 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:03.015 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 0, 2024-09-18T14:04:03.015 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 154618822676 2024-09-18T14:04:03.015 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:03.015 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:03.015 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 1, 2024-09-18T14:04:03.015 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 42949672984 2024-09-18T14:04:03.015 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:03.015 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:03.015 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 2, 2024-09-18T14:04:03.016 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 163208757267 2024-09-18T14:04:03.016 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:03.016 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:04:03.016 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg_by_state": [ 2024-09-18T14:04:03.016 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:03.016 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "active+undersized+degraded", 2024-09-18T14:04:03.016 INFO:tasks.workunit.client.0.smithi047.stdout: "num": 1 2024-09-18T14:04:03.016 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:03.016 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:04:03.016 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg_by_osd": [ 2024-09-18T14:04:03.016 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:03.016 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 2, 2024-09-18T14:04:03.016 INFO:tasks.workunit.client.0.smithi047.stdout: "num_primary_pg": 0, 2024-09-18T14:04:03.016 INFO:tasks.workunit.client.0.smithi047.stdout: "num_acting_pg": 1, 2024-09-18T14:04:03.017 INFO:tasks.workunit.client.0.smithi047.stdout: "num_up_not_acting_pg": 0 2024-09-18T14:04:03.017 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:03.017 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:03.017 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 0, 2024-09-18T14:04:03.017 INFO:tasks.workunit.client.0.smithi047.stdout: "num_primary_pg": 1, 2024-09-18T14:04:03.017 INFO:tasks.workunit.client.0.smithi047.stdout: "num_acting_pg": 1, 2024-09-18T14:04:03.017 INFO:tasks.workunit.client.0.smithi047.stdout: "num_up_not_acting_pg": 0 2024-09-18T14:04:03.017 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:03.017 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:03.017 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 1, 2024-09-18T14:04:03.017 INFO:tasks.workunit.client.0.smithi047.stdout: "num_primary_pg": 0, 2024-09-18T14:04:03.017 INFO:tasks.workunit.client.0.smithi047.stdout: "num_acting_pg": 0, 2024-09-18T14:04:03.017 INFO:tasks.workunit.client.0.smithi047.stdout: "num_up_not_acting_pg": 0 2024-09-18T14:04:03.017 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:03.018 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:04:03.018 INFO:tasks.workunit.client.0.smithi047.stdout: "purged_snaps": [ 2024-09-18T14:04:03.018 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:04:03.018 INFO:tasks.workunit.client.0.smithi047.stdout: "pool": 1, 2024-09-18T14:04:03.018 INFO:tasks.workunit.client.0.smithi047.stdout: "purged_snaps": {} 2024-09-18T14:04:03.018 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:03.018 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:04:03.018 INFO:tasks.workunit.client.0.smithi047.stdout: "servicemap": { 2024-09-18T14:04:03.018 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 7, 2024-09-18T14:04:03.018 INFO:tasks.workunit.client.0.smithi047.stdout: "modified": "2024-09-18T14:03:48.403225+0000", 2024-09-18T14:04:03.018 INFO:tasks.workunit.client.0.smithi047.stdout: "services": {} 2024-09-18T14:04:03.018 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:04:03.018 INFO:tasks.workunit.client.0.smithi047.stdout: "paxos": { 2024-09-18T14:04:03.018 INFO:tasks.workunit.client.0.smithi047.stdout: "first_committed": 1, 2024-09-18T14:04:03.018 INFO:tasks.workunit.client.0.smithi047.stdout: "last_committed": 585, 2024-09-18T14:04:03.018 INFO:tasks.workunit.client.0.smithi047.stdout: "last_pn": 0, 2024-09-18T14:04:03.019 INFO:tasks.workunit.client.0.smithi047.stdout: "accepted_pn": 0 2024-09-18T14:04:03.019 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:04:03.019 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T14:04:03.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-18T14:04:03.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:536: TEST_divergent_2: echo 'delay recovery' 2024-09-18T14:04:03.019 INFO:tasks.workunit.client.0.smithi047.stdout:delay recovery 2024-09-18T14:04:03.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:537: TEST_divergent_2: ceph pg dump pgs 2024-09-18T14:04:03.328 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:04:03.328 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-18T14:02:50.350708+0000 41'101 41:124 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:01:53.783298+0000 0'0 2024-09-18T14:01:53.783298+0000 0 0 periodic scrub scheduled @ 2024-09-19T15:26:35.100116+0000 0 0 2024-09-18T14:04:03.328 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:04:03.328 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:04:03.328 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:04:03.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2024-09-18T14:04:03.339 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.0 2024-09-18T14:04:03.339 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:04:03.339 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:04:03.339 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:04:03.339 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:04:03.340 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:04:03.340 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32609/ceph-osd.0.asok 2024-09-18T14:04:03.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2024-09-18T14:04:03.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.32609/ceph-osd.0.asok set_recovery_delay 100000 2024-09-18T14:04:03.418 INFO:tasks.workunit.client.0.smithi047.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-18T14:04:03.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2024-09-18T14:04:03.426 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.2 2024-09-18T14:04:03.426 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:04:03.426 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:04:03.426 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:04:03.426 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:04:03.426 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:04:03.426 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32609/ceph-osd.2.asok 2024-09-18T14:04:03.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2024-09-18T14:04:03.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.32609/ceph-osd.2.asok set_recovery_delay 100000 2024-09-18T14:04:03.504 INFO:tasks.workunit.client.0.smithi047.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-18T14:04:03.511 INFO:tasks.workunit.client.0.smithi047.stdout:revive divergent 1 2024-09-18T14:04:03.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:544: TEST_divergent_2: echo 'revive divergent 1' 2024-09-18T14:04:03.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:545: TEST_divergent_2: ceph pg dump pgs 2024-09-18T14:04:03.864 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:04:03.864 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-18T14:02:50.350708+0000 41'101 41:124 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:01:53.783298+0000 0'0 2024-09-18T14:01:53.783298+0000 0 0 periodic scrub scheduled @ 2024-09-19T15:26:35.100116+0000 0 0 2024-09-18T14:04:03.865 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:04:03.865 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:04:03.865 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:04:03.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:546: TEST_divergent_2: ceph osd set noup 2024-09-18T14:04:04.618 INFO:tasks.workunit.client.0.smithi047.stderr:noup is set 2024-09-18T14:04:04.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:547: TEST_divergent_2: activate_osd td/divergent-priors 1 2024-09-18T14:04:04.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-18T14:04:04.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:04:04.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T14:04:04.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:04:04.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-09-18T14:04:04.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T14:04:04.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:04:04.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:04:04.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:04:04.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-18T14:04:04.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-18T14:04:04.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:04:04.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:04:04.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-18T14:04:04.633 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:04:04.634 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:04:04.634 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:04:04.634 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:04:04.634 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:04:04.634 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:04:04.634 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:04:04.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:04:04.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:04:04.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-18T14:04:04.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-18T14:04:04.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:04:04.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:04:04.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:04:04.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:04:04.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:04:04.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:04:04.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-09-18T14:04:04.635 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T14:04:04.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T14:04:04.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:04:04.636 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-09-18T14:04:04.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T14:04:04.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:04:04.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:04:04.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:04:04.663 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:04:04.663+0000 7f90a4a4b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:04:04.665 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:04:04.665+0000 7f90a4a4b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:04:04.666 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:04:04.666+0000 7f90a4a4b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:04:05.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-09-18T14:04:05.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:548: TEST_divergent_2: sleep 5 2024-09-18T14:04:05.186 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:04:05.186+0000 7f90a4a4b440 -1 Falling back to public interface 2024-09-18T14:04:08.030 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:04:08.029+0000 7f90a4a4b440 -1 osd.1 29 log_to_monitors true 2024-09-18T14:04:10.024 INFO:tasks.workunit.client.0.smithi047.stdout:delay recovery divergent 2024-09-18T14:04:10.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:550: TEST_divergent_2: echo 'delay recovery divergent' 2024-09-18T14:04:10.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:551: TEST_divergent_2: ceph pg dump pgs 2024-09-18T14:04:10.393 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:04:10.393 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-18T14:02:50.350708+0000 41'101 41:125 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:01:53.783298+0000 0'0 2024-09-18T14:01:53.783298+0000 0 0 periodic scrub scheduled @ 2024-09-19T15:26:35.100116+0000 0 0 2024-09-18T14:04:10.393 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:04:10.393 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:04:10.393 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:04:10.405 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: get_asok_path osd.1 2024-09-18T14:04:10.405 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:04:10.405 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:04:10.405 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:04:10.405 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:04:10.405 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:04:10.405 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32609/ceph-osd.1.asok 2024-09-18T14:04:10.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: CEPH_ARGS= 2024-09-18T14:04:10.406 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.32609/ceph-osd.1.asok set_recovery_delay 100000 2024-09-18T14:04:10.483 INFO:tasks.workunit.client.0.smithi047.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-18T14:04:10.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:554: TEST_divergent_2: ceph osd unset noup 2024-09-18T14:04:11.337 INFO:tasks.workunit.client.0.smithi047.stderr:noup is unset 2024-09-18T14:04:11.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:556: TEST_divergent_2: wait_for_osd up 0 2024-09-18T14:04:11.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:04:11.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:04:11.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:04:11.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:04:11.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:04:11.350 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:04:11.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:04:11.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:04:11.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:04:11.810 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 36 up_thru 40 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/53956,v1:127.0.0.1:6803/53956] [v2:127.0.0.1:6804/53956,v1:127.0.0.1:6805/53956] exists,up df87ad03-46d9-487e-8f7a-aab30f82a1bd 2024-09-18T14:04:11.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:04:11.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:04:11.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:04:11.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:557: TEST_divergent_2: wait_for_osd up 1 2024-09-18T14:04:11.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:04:11.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:04:11.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:04:11.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:04:11.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:04:11.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:04:11.812 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:04:11.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:04:11.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:04:12.208 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/57072,v1:127.0.0.1:6819/57072] [v2:127.0.0.1:6820/57072,v1:127.0.0.1:6821/57072] exists,up b8177855-da06-4ebd-9d12-538bef0d43b8 2024-09-18T14:04:12.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:04:12.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:04:12.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:04:12.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:558: TEST_divergent_2: wait_for_osd up 2 2024-09-18T14:04:12.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:04:12.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T14:04:12.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:04:12.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:04:12.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:04:12.210 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:04:12.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:04:12.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:04:12.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:04:12.630 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/54526,v1:127.0.0.1:6811/54526] [v2:127.0.0.1:6812/54526,v1:127.0.0.1:6813/54526] exists,up dec93258-345a-406b-85ef-8316978cb8b7 2024-09-18T14:04:12.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:04:12.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:04:12.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:04:12.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:560: TEST_divergent_2: ceph pg dump pgs 2024-09-18T14:04:12.999 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:04:12.999 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-18T14:02:50.350708+0000 41'101 43:127 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:01:53.783298+0000 0'0 2024-09-18T14:01:53.783298+0000 0 0 periodic scrub scheduled @ 2024-09-19T15:26:35.100116+0000 0 0 2024-09-18T14:04:12.999 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:04:12.999 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:04:13.000 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:04:13.009 INFO:tasks.workunit.client.0.smithi047.stdout:wait for peering 2024-09-18T14:04:13.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:561: TEST_divergent_2: echo 'wait for peering' 2024-09-18T14:04:13.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:562: TEST_divergent_2: ceph pg dump pgs 2024-09-18T14:04:13.362 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:04:13.362 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-18T14:02:50.350708+0000 41'101 43:127 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:01:53.783298+0000 0'0 2024-09-18T14:01:53.783298+0000 0 0 periodic scrub scheduled @ 2024-09-19T15:26:35.100116+0000 0 0 2024-09-18T14:04:13.362 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:04:13.363 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:04:13.363 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:04:13.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:563: TEST_divergent_2: rados -p test put foo /etc/fstab 2024-09-18T14:04:13.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:567: TEST_divergent_2: echo 'killing divergent 1' 2024-09-18T14:04:13.472 INFO:tasks.workunit.client.0.smithi047.stdout:killing divergent 1 2024-09-18T14:04:13.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:568: TEST_divergent_2: ceph pg dump pgs 2024-09-18T14:04:13.825 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:04:13.826 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-18T14:02:50.350708+0000 41'101 43:127 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:01:53.783298+0000 0'0 2024-09-18T14:01:53.783298+0000 0 0 periodic scrub scheduled @ 2024-09-19T15:26:35.100116+0000 0 0 2024-09-18T14:04:13.826 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:04:13.826 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:04:13.826 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:04:13.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:569: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd.1 2024-09-18T14:04:13.836 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:04:13.836 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:04:13.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:04:13.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:04:13.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:04:13.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:04:13.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:572: TEST_divergent_2: expfile=td/divergent-priors/exp.32609.out 2024-09-18T14:04:13.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:573: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.32609.out 2024-09-18T14:04:13.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-18T14:04:13.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T14:04:13.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T14:04:13.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T14:04:13.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-18T14:04:13.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.32609.out 2024-09-18T14:04:15.413 INFO:tasks.workunit.client.0.smithi047.stderr:Exporting 1.0 info 1.0( v 48'102 lc 0'0 (29'90,48'102] local-lis/les=46/48 n=101 ec=23/23 lis/c=46/23 les/c/f=48/24/0 sis=46) 2024-09-18T14:04:15.430 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:023dd3d2:::existing_12:head# 2024-09-18T14:04:15.443 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:04c40a0c:::existing_93:head# 2024-09-18T14:04:15.463 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:064ff5fb:::existing_36:head# 2024-09-18T14:04:15.463 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:091f9f1d:::existing_94:head# 2024-09-18T14:04:15.471 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0a865a93:::existing_54:head# 2024-09-18T14:04:15.472 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0fed513c:::existing_41:head# 2024-09-18T14:04:15.488 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:13b9e0b8:::existing_92:head# 2024-09-18T14:04:15.496 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:169157fa:::existing_81:head# 2024-09-18T14:04:15.498 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:17731382:::existing_58:head# 2024-09-18T14:04:15.513 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:18a45a39:::existing_43:head# 2024-09-18T14:04:15.521 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1c160f01:::existing_66:head# 2024-09-18T14:04:15.523 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1dd6c249:::existing_97:head# 2024-09-18T14:04:15.547 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:227e0c77:::existing_7:head# 2024-09-18T14:04:15.559 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:23e1b063:::existing_59:head# 2024-09-18T14:04:15.560 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:28566edb:::existing_78:head# 2024-09-18T14:04:15.561 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2a101182:::existing_20:head# 2024-09-18T14:04:15.565 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2d1519da:::existing_96:head# 2024-09-18T14:04:15.565 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3078b87a:::existing_86:head# 2024-09-18T14:04:15.565 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:30bde01b:::existing_15:head# 2024-09-18T14:04:15.566 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:317a7e19:::existing_56:head# 2024-09-18T14:04:15.566 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:38b9c7fa:::existing_33:head# 2024-09-18T14:04:15.569 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3e16d84b:::existing_48:head# 2024-09-18T14:04:15.570 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:45fe8203:::existing_75:head# 2024-09-18T14:04:15.570 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4e82c9ba:::existing_50:head# 2024-09-18T14:04:15.570 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5298dcdb:::existing_57:head# 2024-09-18T14:04:15.570 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:52ebaa44:::existing_32:head# 2024-09-18T14:04:15.570 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:53588912:::existing_9:head# 2024-09-18T14:04:15.570 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5b7e2de6:::existing_84:head# 2024-09-18T14:04:15.570 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5da0255c:::existing_13:head# 2024-09-18T14:04:15.570 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5da2dab8:::existing_53:head# 2024-09-18T14:04:15.570 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5ee3514b:::existing_30:head# 2024-09-18T14:04:15.570 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:602f83fe:::foo:head# 2024-09-18T14:04:15.570 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:60426cc7:::existing_62:head# 2024-09-18T14:04:15.570 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:628918ca:::existing_79:head# 2024-09-18T14:04:15.570 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:63f8ab2d:::existing_77:head# 2024-09-18T14:04:15.570 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:67e1780f:::existing_27:head# 2024-09-18T14:04:15.571 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:68143ca1:::existing_71:head# 2024-09-18T14:04:15.571 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6bac9509:::existing_29:head# 2024-09-18T14:04:15.571 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6bbec798:::existing_45:head# 2024-09-18T14:04:15.571 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6cb23a40:::existing_65:head# 2024-09-18T14:04:15.571 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7055ec5e:::existing_83:head# 2024-09-18T14:04:15.571 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:70a54e2b:::existing_87:head# 2024-09-18T14:04:15.571 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:750ace21:::existing_19:head# 2024-09-18T14:04:15.571 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:78623b1d:::existing_6:head# 2024-09-18T14:04:15.584 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:79881152:::existing_63:head# 2024-09-18T14:04:15.585 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7a80c0d5:::existing_99:head# 2024-09-18T14:04:15.595 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:81490677:::existing_76:head# 2024-09-18T14:04:15.596 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:85ec8e5d:::existing_85:head# 2024-09-18T14:04:15.596 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8e2abbf0:::existing_49:head# 2024-09-18T14:04:15.596 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8ee0fe0f:::existing_52:head# 2024-09-18T14:04:15.596 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:923c5d4d:::existing_5:head# 2024-09-18T14:04:15.596 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:92731e71:::existing_89:head# 2024-09-18T14:04:15.596 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:928205cf:::existing_82:head# 2024-09-18T14:04:15.596 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:93be2267:::existing_24:head# 2024-09-18T14:04:15.596 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:99ceda2c:::existing_44:head# 2024-09-18T14:04:15.596 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9dd917ee:::existing_26:head# 2024-09-18T14:04:15.596 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a28805a6:::existing_72:head# 2024-09-18T14:04:15.596 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a294a8f6:::existing_16:head# 2024-09-18T14:04:15.596 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a2f5473a:::existing_42:head# 2024-09-18T14:04:15.596 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a504a533:::existing_90:head# 2024-09-18T14:04:15.597 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a691337e:::existing_61:head# 2024-09-18T14:04:15.597 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a6e8d7ac:::existing_28:head# 2024-09-18T14:04:15.597 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a9fde86e:::existing_18:head# 2024-09-18T14:04:15.597 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:aa309faa:::existing_68:head# 2024-09-18T14:04:15.597 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ae40b0ed:::existing_31:head# 2024-09-18T14:04:15.597 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b279b1fe:::existing_10:head# 2024-09-18T14:04:15.597 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b343e277:::existing_88:head# 2024-09-18T14:04:15.597 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b7130e90:::existing_80:head# 2024-09-18T14:04:15.597 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:bc6f7309:::existing_70:head# 2024-09-18T14:04:15.597 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:be8489b7:::existing_38:head# 2024-09-18T14:04:15.597 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:beb4e5e3:::existing_47:head# 2024-09-18T14:04:15.597 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:bebec877:::existing_74:head# 2024-09-18T14:04:15.597 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c04347d8:::existing_60:head# 2024-09-18T14:04:15.598 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c156033e:::existing_100:head# 2024-09-18T14:04:15.598 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c19f5a8e:::existing_8:head# 2024-09-18T14:04:15.598 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c3e6ebe4:::existing_17:head# 2024-09-18T14:04:15.598 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c6c5a7c2:::existing_39:head# 2024-09-18T14:04:15.598 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:cc1d0abf:::existing_95:head# 2024-09-18T14:04:15.598 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ccb3865b:::existing_73:head# 2024-09-18T14:04:15.598 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:cd84e19e:::existing_40:head# 2024-09-18T14:04:15.598 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d2cd64ce:::existing_37:head# 2024-09-18T14:04:15.598 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d41a25c9:::existing_22:head# 2024-09-18T14:04:15.598 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d7f088da:::existing_91:head# 2024-09-18T14:04:15.598 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d9cbd1d6:::existing_64:head# 2024-09-18T14:04:15.598 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:dcc4ff82:::existing_98:head# 2024-09-18T14:04:15.598 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:dd677bd3:::existing_35:head# 2024-09-18T14:04:15.598 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:dd6d90e7:::existing_46:head# 2024-09-18T14:04:15.599 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e0e22077:::existing_21:head# 2024-09-18T14:04:15.599 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e255b710:::existing_55:head# 2024-09-18T14:04:15.599 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:eff1d5db:::existing_14:head# 2024-09-18T14:04:15.599 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f1b156bf:::existing_23:head# 2024-09-18T14:04:15.599 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f5ce4199:::existing_34:head# 2024-09-18T14:04:15.599 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f6809456:::existing_11:head# 2024-09-18T14:04:15.599 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f699a2a9:::existing_67:head# 2024-09-18T14:04:15.599 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f9baac43:::existing_25:head# 2024-09-18T14:04:15.599 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fb76688e:::existing_51:head# 2024-09-18T14:04:15.599 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fbb92eec:::existing_69:head# 2024-09-18T14:04:15.600 INFO:tasks.workunit.client.0.smithi047.stderr:Export successful 2024-09-18T14:04:15.600 INFO:tasks.workunit.client.0.smithi047.stdout: marking collection for removal 2024-09-18T14:04:15.600 INFO:tasks.workunit.client.0.smithi047.stdout:setting '_remove' omap key 2024-09-18T14:04:15.601 INFO:tasks.workunit.client.0.smithi047.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-09-18T14:04:15.789 INFO:tasks.workunit.client.0.smithi047.stderr:Remove successful 2024-09-18T14:04:16.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:574: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.32609.out 2024-09-18T14:04:16.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-18T14:04:16.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T14:04:16.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T14:04:16.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T14:04:16.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-18T14:04:16.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.32609.out 2024-09-18T14:04:17.110 INFO:tasks.workunit.client.0.smithi047.stdout:get_pg_num_history pg_num_history pg_num_history(e48 pg_nums {1={23=1}} deleted_pools ) 2024-09-18T14:04:17.110 INFO:tasks.workunit.client.0.smithi047.stdout:Importing pgid 1.0 2024-09-18T14:04:17.110 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:023dd3d2:::existing_12:head# 2024-09-18T14:04:17.110 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:17.410 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:04c40a0c:::existing_93:head# 2024-09-18T14:04:17.410 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:17.460 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:064ff5fb:::existing_36:head# 2024-09-18T14:04:17.461 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:17.511 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:091f9f1d:::existing_94:head# 2024-09-18T14:04:17.511 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:17.561 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0a865a93:::existing_54:head# 2024-09-18T14:04:17.562 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:17.613 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0fed513c:::existing_41:head# 2024-09-18T14:04:17.613 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:17.645 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:13b9e0b8:::existing_92:head# 2024-09-18T14:04:17.645 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:17.678 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:169157fa:::existing_81:head# 2024-09-18T14:04:17.679 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:17.712 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:17731382:::existing_58:head# 2024-09-18T14:04:17.712 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:17.745 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:18a45a39:::existing_43:head# 2024-09-18T14:04:17.746 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:17.779 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1c160f01:::existing_66:head# 2024-09-18T14:04:17.779 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:17.812 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1dd6c249:::existing_97:head# 2024-09-18T14:04:17.812 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:17.846 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:227e0c77:::existing_7:head# 2024-09-18T14:04:17.846 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:17.879 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:23e1b063:::existing_59:head# 2024-09-18T14:04:17.879 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:17.913 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:28566edb:::existing_78:head# 2024-09-18T14:04:17.913 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:17.946 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2a101182:::existing_20:head# 2024-09-18T14:04:17.946 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:17.988 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2d1519da:::existing_96:head# 2024-09-18T14:04:17.988 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:18.024 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3078b87a:::existing_86:head# 2024-09-18T14:04:18.025 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:18.055 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:30bde01b:::existing_15:head# 2024-09-18T14:04:18.055 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:18.088 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:317a7e19:::existing_56:head# 2024-09-18T14:04:18.089 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:18.122 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:38b9c7fa:::existing_33:head# 2024-09-18T14:04:18.122 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:18.155 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3e16d84b:::existing_48:head# 2024-09-18T14:04:18.156 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:18.206 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:45fe8203:::existing_75:head# 2024-09-18T14:04:18.206 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:18.239 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4e82c9ba:::existing_50:head# 2024-09-18T14:04:18.239 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:18.272 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5298dcdb:::existing_57:head# 2024-09-18T14:04:18.273 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:18.306 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:52ebaa44:::existing_32:head# 2024-09-18T14:04:18.306 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:18.339 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:53588912:::existing_9:head# 2024-09-18T14:04:18.340 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:18.373 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5b7e2de6:::existing_84:head# 2024-09-18T14:04:18.373 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:18.406 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5da0255c:::existing_13:head# 2024-09-18T14:04:18.407 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:18.441 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5da2dab8:::existing_53:head# 2024-09-18T14:04:18.441 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:18.482 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5ee3514b:::existing_30:head# 2024-09-18T14:04:18.482 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:18.540 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:602f83fe:::foo:head# 2024-09-18T14:04:18.540 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:18.582 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:60426cc7:::existing_62:head# 2024-09-18T14:04:18.582 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:18.607 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:628918ca:::existing_79:head# 2024-09-18T14:04:18.608 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:18.632 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:63f8ab2d:::existing_77:head# 2024-09-18T14:04:18.633 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:18.658 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:67e1780f:::existing_27:head# 2024-09-18T14:04:18.658 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:18.683 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:68143ca1:::existing_71:head# 2024-09-18T14:04:18.683 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:18.708 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6bac9509:::existing_29:head# 2024-09-18T14:04:18.708 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:18.733 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6bbec798:::existing_45:head# 2024-09-18T14:04:18.733 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:18.758 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6cb23a40:::existing_65:head# 2024-09-18T14:04:18.758 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:18.783 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7055ec5e:::existing_83:head# 2024-09-18T14:04:18.783 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:18.808 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:70a54e2b:::existing_87:head# 2024-09-18T14:04:18.808 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:18.834 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:750ace21:::existing_19:head# 2024-09-18T14:04:18.834 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:18.860 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:78623b1d:::existing_6:head# 2024-09-18T14:04:18.860 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:18.909 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:79881152:::existing_63:head# 2024-09-18T14:04:18.909 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:18.951 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7a80c0d5:::existing_99:head# 2024-09-18T14:04:18.951 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:19.001 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:81490677:::existing_76:head# 2024-09-18T14:04:19.001 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:19.026 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:85ec8e5d:::existing_85:head# 2024-09-18T14:04:19.026 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:19.051 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8e2abbf0:::existing_49:head# 2024-09-18T14:04:19.052 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:19.076 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8ee0fe0f:::existing_52:head# 2024-09-18T14:04:19.076 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:19.102 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:923c5d4d:::existing_5:head# 2024-09-18T14:04:19.102 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:19.126 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:92731e71:::existing_89:head# 2024-09-18T14:04:19.127 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:19.151 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:928205cf:::existing_82:head# 2024-09-18T14:04:19.152 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:19.177 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:93be2267:::existing_24:head# 2024-09-18T14:04:19.177 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:19.202 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:99ceda2c:::existing_44:head# 2024-09-18T14:04:19.202 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:19.227 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9dd917ee:::existing_26:head# 2024-09-18T14:04:19.227 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:19.252 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a28805a6:::existing_72:head# 2024-09-18T14:04:19.253 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:19.277 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a294a8f6:::existing_16:head# 2024-09-18T14:04:19.277 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:19.302 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a2f5473a:::existing_42:head# 2024-09-18T14:04:19.303 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:19.327 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a504a533:::existing_90:head# 2024-09-18T14:04:19.328 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:19.353 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a691337e:::existing_61:head# 2024-09-18T14:04:19.353 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:19.378 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a6e8d7ac:::existing_28:head# 2024-09-18T14:04:19.378 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:19.403 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a9fde86e:::existing_18:head# 2024-09-18T14:04:19.403 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:19.429 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:aa309faa:::existing_68:head# 2024-09-18T14:04:19.429 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:19.464 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ae40b0ed:::existing_31:head# 2024-09-18T14:04:19.465 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:19.521 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b279b1fe:::existing_10:head# 2024-09-18T14:04:19.521 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:19.554 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b343e277:::existing_88:head# 2024-09-18T14:04:19.554 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:19.588 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b7130e90:::existing_80:head# 2024-09-18T14:04:19.588 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:19.621 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:bc6f7309:::existing_70:head# 2024-09-18T14:04:19.621 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:19.654 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:be8489b7:::existing_38:head# 2024-09-18T14:04:19.655 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:19.697 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:beb4e5e3:::existing_47:head# 2024-09-18T14:04:19.697 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:19.730 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:bebec877:::existing_74:head# 2024-09-18T14:04:19.730 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:19.764 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c04347d8:::existing_60:head# 2024-09-18T14:04:19.764 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:19.797 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c156033e:::existing_100:head# 2024-09-18T14:04:19.797 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:19.830 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c19f5a8e:::existing_8:head# 2024-09-18T14:04:19.831 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:19.864 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c3e6ebe4:::existing_17:head# 2024-09-18T14:04:19.864 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:19.897 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c6c5a7c2:::existing_39:head# 2024-09-18T14:04:19.898 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:19.931 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cc1d0abf:::existing_95:head# 2024-09-18T14:04:19.931 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:19.964 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ccb3865b:::existing_73:head# 2024-09-18T14:04:19.964 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:19.998 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cd84e19e:::existing_40:head# 2024-09-18T14:04:19.998 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:20.031 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d2cd64ce:::existing_37:head# 2024-09-18T14:04:20.031 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:20.065 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d41a25c9:::existing_22:head# 2024-09-18T14:04:20.065 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:20.098 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d7f088da:::existing_91:head# 2024-09-18T14:04:20.098 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:20.132 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d9cbd1d6:::existing_64:head# 2024-09-18T14:04:20.132 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:20.165 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:dcc4ff82:::existing_98:head# 2024-09-18T14:04:20.165 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:20.199 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:dd677bd3:::existing_35:head# 2024-09-18T14:04:20.199 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:20.232 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:dd6d90e7:::existing_46:head# 2024-09-18T14:04:20.233 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:20.265 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e0e22077:::existing_21:head# 2024-09-18T14:04:20.266 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:20.299 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e255b710:::existing_55:head# 2024-09-18T14:04:20.299 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:20.332 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:eff1d5db:::existing_14:head# 2024-09-18T14:04:20.333 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:20.366 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f1b156bf:::existing_23:head# 2024-09-18T14:04:20.366 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:20.399 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f5ce4199:::existing_34:head# 2024-09-18T14:04:20.399 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:20.432 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f6809456:::existing_11:head# 2024-09-18T14:04:20.433 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:20.474 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f699a2a9:::existing_67:head# 2024-09-18T14:04:20.474 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:20.541 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f9baac43:::existing_25:head# 2024-09-18T14:04:20.541 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:20.583 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fb76688e:::existing_51:head# 2024-09-18T14:04:20.583 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:20.608 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fbb92eec:::existing_69:head# 2024-09-18T14:04:20.608 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:04:20.633 INFO:tasks.workunit.client.0.smithi047.stdout:write_pg epoch 48 info 1.0( v 48'102 lc 0'0 (29'90,48'102] local-lis/les=46/48 n=101 ec=23/23 lis/c=46/23 les/c/f=48/24/0 sis=46) 2024-09-18T14:04:20.658 INFO:tasks.workunit.client.0.smithi047.stdout:Import successful 2024-09-18T14:04:21.174 INFO:tasks.workunit.client.0.smithi047.stdout:reviving divergent 1 2024-09-18T14:04:21.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:576: TEST_divergent_2: echo 'reviving divergent 1' 2024-09-18T14:04:21.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:577: TEST_divergent_2: ceph pg dump pgs 2024-09-18T14:04:21.540 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:04:21.541 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 100 4 4 0 0 47971 0 0 11 90 11 active+recovering+degraded 2024-09-18T14:04:11.883764+0000 41'101 48:129 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:01:53.783298+0000 0'0 2024-09-18T14:01:53.783298+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:02:45.364750+0000 0 0 2024-09-18T14:04:21.541 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:04:21.541 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:04:21.541 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:04:21.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:578: TEST_divergent_2: activate_osd td/divergent-priors 1 2024-09-18T14:04:21.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-18T14:04:21.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:04:21.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T14:04:21.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:04:21.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-09-18T14:04:21.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T14:04:21.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:04:21.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:04:21.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:04:21.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-18T14:04:21.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-18T14:04:21.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:04:21.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:04:21.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-18T14:04:21.557 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:04:21.557 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:04:21.558 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:04:21.558 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:04:21.558 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:04:21.558 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:04:21.558 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:04:21.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:04:21.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:04:21.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-18T14:04:21.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-18T14:04:21.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:04:21.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:04:21.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:04:21.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:04:21.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:04:21.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:04:21.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-09-18T14:04:21.561 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T14:04:21.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T14:04:21.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:04:21.563 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-09-18T14:04:21.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T14:04:21.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:04:21.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:04:21.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:04:21.584 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:04:21.584+0000 7fd75d926440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:04:21.586 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:04:21.586+0000 7fd75d926440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:04:21.587 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:04:21.587+0000 7fd75d926440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:04:21.948 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T14:04:21.948 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:04:21.948 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:04:21.948 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:04:21.948 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:04:21.948 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:04:21.949 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:04:21.949 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:04:21.949 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:04:21.949 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:04:22.105 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:04:22.105+0000 7fd75d926440 -1 Falling back to public interface 2024-09-18T14:04:22.473 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/57072,v1:127.0.0.1:6819/57072] [v2:127.0.0.1:6820/57072,v1:127.0.0.1:6821/57072] exists,up b8177855-da06-4ebd-9d12-538bef0d43b8 2024-09-18T14:04:22.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:04:22.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:04:22.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:04:22.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:579: TEST_divergent_2: wait_for_osd up 1 2024-09-18T14:04:22.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:04:22.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:04:22.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:04:22.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:04:22.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:04:22.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:04:22.475 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:04:22.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:04:22.476 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:04:22.873 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/57072,v1:127.0.0.1:6819/57072] [v2:127.0.0.1:6820/57072,v1:127.0.0.1:6821/57072] exists,up b8177855-da06-4ebd-9d12-538bef0d43b8 2024-09-18T14:04:22.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:04:22.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:04:22.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:04:22.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:581: TEST_divergent_2: sleep 20 2024-09-18T14:04:24.878 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:04:24.877+0000 7fd75d926440 -1 osd.1 48 log_to_monitors true 2024-09-18T14:04:42.876 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: get_asok_path osd.1 2024-09-18T14:04:42.876 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:04:42.877 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:04:42.877 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:04:42.877 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:04:42.877 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:04:42.877 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32609/ceph-osd.1.asok 2024-09-18T14:04:42.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: CEPH_ARGS= 2024-09-18T14:04:42.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.32609/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T14:04:42.961 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T14:04:42.962 INFO:tasks.workunit.client.0.smithi047.stdout: "ops": [], 2024-09-18T14:04:42.962 INFO:tasks.workunit.client.0.smithi047.stdout: "num_ops": 0 2024-09-18T14:04:42.962 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T14:04:42.969 INFO:tasks.workunit.client.0.smithi047.stdout:allowing recovery 2024-09-18T14:04:42.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:584: TEST_divergent_2: echo 'allowing recovery' 2024-09-18T14:04:42.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:585: TEST_divergent_2: ceph pg dump pgs 2024-09-18T14:04:43.328 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:04:43.328 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 101 0 0 0 0 48454 0 0 12 90 12 active+clean 2024-09-18T14:04:27.016170+0000 48'102 53:153 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:01:53.783298+0000 0'0 2024-09-18T14:01:53.783298+0000 0 0 periodic scrub scheduled @ 2024-09-19T14:54:12.489147+0000 0 0 2024-09-18T14:04:43.328 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:04:43.328 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:04:43.329 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:04:43.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-09-18T14:04:43.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-18T14:04:43.446 INFO:tasks.workunit.client.0.smithi047.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-18T14:04:43.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-09-18T14:04:43.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-18T14:04:43.562 INFO:tasks.workunit.client.0.smithi047.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-18T14:04:43.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-09-18T14:04:43.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-18T14:04:43.678 INFO:tasks.workunit.client.0.smithi047.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-18T14:04:43.686 INFO:tasks.workunit.client.0.smithi047.stdout:reading divergent objects 2024-09-18T14:04:43.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:592: TEST_divergent_2: echo 'reading divergent objects' 2024-09-18T14:04:43.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:593: TEST_divergent_2: ceph pg dump pgs 2024-09-18T14:04:44.038 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:04:44.038 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 101 0 0 0 0 48454 0 0 12 90 12 active+clean 2024-09-18T14:04:27.016170+0000 48'102 53:153 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:01:53.783298+0000 0'0 2024-09-18T14:01:53.783298+0000 0 0 periodic scrub scheduled @ 2024-09-19T14:54:12.489147+0000 0 0 2024-09-18T14:04:44.038 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:04:44.039 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:04:44.039 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:04:44.049 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: expr 2 + 2 2024-09-18T14:04:44.050 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: seq 1 4 2024-09-18T14:04:44.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-18T14:04:44.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_1 td/divergent-priors/existing 2024-09-18T14:04:44.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-18T14:04:44.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_2 td/divergent-priors/existing 2024-09-18T14:04:44.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-18T14:04:44.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_3 td/divergent-priors/existing 2024-09-18T14:04:44.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-18T14:04:44.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_4 td/divergent-priors/existing 2024-09-18T14:04:44.176 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: seq 1 2 2024-09-18T14:04:44.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-18T14:04:44.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_1 td/divergent-priors/existing 2024-09-18T14:04:44.206 INFO:tasks.workunit.client.0.smithi047.stderr:error getting test/newobject_1: (2) No such file or directory 2024-09-18T14:04:44.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-18T14:04:44.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_2 td/divergent-priors/existing 2024-09-18T14:04:44.237 INFO:tasks.workunit.client.0.smithi047.stderr:error getting test/newobject_2: (2) No such file or directory 2024-09-18T14:04:44.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:602: TEST_divergent_2: rm -f td/divergent-priors/existing 2024-09-18T14:04:44.240 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-09-18T14:04:44.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-18T14:04:44.245 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 29'105 (29'4) delete 1:70b3bf12:::existing_4:head by client.4632.0:1 2024-09-18T14:02:26.328845+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2024-09-18T14:04:44.245 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 20 _merge_object_divergent_entries: keeping 29'105 (29'4) delete 1:70b3bf12:::existing_4:head by client.4632.0:1 2024-09-18T14:02:26.328845+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2024-09-18T14:04:44.245 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-18T14:04:44.245 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 29'4 first_divergent_update: 29'105 last_divergent_update: 29'105 2024-09-18T14:04:44.245 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 10 _merge_object_divergent_entries: more recent entry found: 41'101 (29'4) modify 1:70b3bf12:::existing_4:head by client.4748.0:1 2024-09-18T14:03:26.637146+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0, already merged 2024-09-18T14:04:44.245 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 29'102 (0'0) modify 1:75ac5091:::newobject_2:head by client.4624.0:1 2024-09-18T14:02:26.320248+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2024-09-18T14:04:44.246 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 20 _merge_object_divergent_entries: keeping 29'102 (0'0) modify 1:75ac5091:::newobject_2:head by client.4624.0:1 2024-09-18T14:02:26.320248+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2024-09-18T14:04:44.246 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-18T14:04:44.246 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 29'102 last_divergent_update: 29'102 2024-09-18T14:04:44.246 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2024-09-18T14:04:44.246 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2024-09-18T14:04:44.246 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 29'106 (29'3) delete 1:8881e6d8:::existing_3:head by client.4633.0:1 2024-09-18T14:02:26.329373+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2024-09-18T14:04:44.246 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 20 _merge_object_divergent_entries: keeping 29'106 (29'3) delete 1:8881e6d8:::existing_3:head by client.4633.0:1 2024-09-18T14:02:26.329373+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2024-09-18T14:04:44.246 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-09-18T14:04:44.246 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 29'3 first_divergent_update: 29'106 last_divergent_update: 29'106 2024-09-18T14:04:44.246 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-09-18T14:04:44.246 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-09-18T14:04:44.246 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 29'106 2024-09-18T14:04:44.247 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 29'106 (29'3) delete 1:8881e6d8:::existing_3:head by client.4633.0:1 2024-09-18T14:02:26.329373+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2024-09-18T14:04:44.247 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-09-18T14:04:44.247 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 29'3 <= info.log_tail 29'90 2024-09-18T14:04:44.247 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 29'101 (29'2) modify 1:97db2a0f:::existing_2:head by client.4618.0:1 2024-09-18T14:02:26.304792+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-18T14:04:44.247 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 20 _merge_object_divergent_entries: keeping 29'101 (29'2) modify 1:97db2a0f:::existing_2:head by client.4618.0:1 2024-09-18T14:02:26.304792+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-18T14:04:44.247 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-18T14:04:44.247 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 29'2 first_divergent_update: 29'101 last_divergent_update: 29'101 2024-09-18T14:04:44.247 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2024-09-18T14:04:44.247 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2024-09-18T14:04:44.247 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 29'106 2024-09-18T14:04:44.247 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 29'101 (29'2) modify 1:97db2a0f:::existing_2:head by client.4618.0:1 2024-09-18T14:02:26.304792+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-18T14:04:44.247 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2024-09-18T14:04:44.247 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 29'2 <= info.log_tail 29'90 2024-09-18T14:04:44.248 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 29'103 (29'1) modify 1:d858c035:::existing_1:head by client.4625.0:1 2024-09-18T14:02:26.321866+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-18T14:04:44.248 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 20 _merge_object_divergent_entries: keeping 29'103 (29'1) modify 1:d858c035:::existing_1:head by client.4625.0:1 2024-09-18T14:02:26.321866+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-18T14:04:44.248 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-18T14:04:44.248 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 29'1 first_divergent_update: 29'103 last_divergent_update: 29'103 2024-09-18T14:04:44.248 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2024-09-18T14:04:44.248 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2024-09-18T14:04:44.248 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 29'106 2024-09-18T14:04:44.248 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 29'103 (29'1) modify 1:d858c035:::existing_1:head by client.4625.0:1 2024-09-18T14:02:26.321866+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-18T14:04:44.249 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2024-09-18T14:04:44.249 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 29'1 <= info.log_tail 29'90 2024-09-18T14:04:44.249 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 29'104 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4627.0:1 2024-09-18T14:02:26.322957+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2024-09-18T14:04:44.249 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 20 _merge_object_divergent_entries: keeping 29'104 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4627.0:1 2024-09-18T14:02:26.322957+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2024-09-18T14:04:44.249 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-18T14:04:44.249 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 29'104 last_divergent_update: 29'104 2024-09-18T14:04:44.249 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2024-09-18T14:04:44.249 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:04:11.613+0000 7f90879c7640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2024-09-18T14:04:44.249 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-09-18T14:04:44.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-18T14:04:44.252 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-09-18T14:04:44.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-18T14:04:44.257 INFO:tasks.workunit.client.0.smithi047.stdout:success 2024-09-18T14:04:44.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:617: TEST_divergent_2: echo success 2024-09-18T14:04:44.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:619: TEST_divergent_2: rm td/divergent-priors/td/divergent-priors/exp.32609.out 2024-09-18T14:04:44.258 INFO:tasks.workunit.client.0.smithi047.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.32609.out': No such file or directory 2024-09-18T14:04:44.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:621: TEST_divergent_2: delete_pool test 2024-09-18T14:04:44.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-18T14:04:44.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-18T14:04:44.882 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test' does not exist 2024-09-18T14:04:44.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:622: TEST_divergent_2: kill_daemons td/divergent-priors 2024-09-18T14:04:44.894 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:04:44.894 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:04:44.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:04:44.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:04:44.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:04:45.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:04:45.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-18T14:04:45.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-18T14:04:45.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:04:45.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-18T14:04:45.205 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:04:45.205 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:04:45.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:04:45.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:04:45.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:04:45.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:04:45.209 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:04:45.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:04:45.210 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:04:45.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:04:45.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:04:45.212 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:04:45.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:04:45.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:04:45.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:04:45.213 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:04:45.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:04:45.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:04:45.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-18T14:04:45.236 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:04:45.236 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:04:45.236 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:04:45.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.32609 2024-09-18T14:04:45.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:04:45.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:04:45.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-18T14:04:45.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-18T14:04:45.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-09-18T14:04:45.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-09-18T14:04:45.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-18T14:04:45.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:04:45.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-18T14:04:45.239 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:04:45.239 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:04:45.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:04:45.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:04:45.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:04:45.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:04:45.240 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:04:45.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:04:45.241 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:04:45.242 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:04:45.242 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:04:45.242 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:04:45.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:04:45.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:04:45.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:04:45.243 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:04:45.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:04:45.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:04:45.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-18T14:04:45.249 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:04:45.249 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:04:45.249 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:04:45.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.32609 2024-09-18T14:04:45.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:04:45.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:04:45.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-09-18T14:04:45.250 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T14:04:45.250 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:04:45.250 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:04:45.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.32609 2024-09-18T14:04:45.251 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T14:04:45.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T14:04:45.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T14:04:45.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-18T14:04:45.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_3 td/divergent-priors 2024-09-18T14:04:45.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:628: TEST_divergent_3: local dir=td/divergent-priors 2024-09-18T14:04:45.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:631: TEST_divergent_3: local dummyfile=/etc/fstab 2024-09-18T14:04:45.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:632: TEST_divergent_3: local dummyfile2=/etc/resolv.conf 2024-09-18T14:04:45.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:634: TEST_divergent_3: local num_osds=3 2024-09-18T14:04:45.252 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: expr 3 - 1 2024-09-18T14:04:45.253 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: seq 0 2 2024-09-18T14:04:45.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: local 'osds=0 2024-09-18T14:04:45.253 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T14:04:45.253 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T14:04:45.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:636: TEST_divergent_3: run_mon td/divergent-priors a 2024-09-18T14:04:45.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-09-18T14:04:45.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T14:04:45.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T14:04:45.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T14:04:45.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-09-18T14:04:45.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-18T14:04:45.616 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T14:04:45.616 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:04:45.617 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:04:45.617 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:04:45.617 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:04:45.617 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:04:45.617 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:04:45.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T14:04:45.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T14:04:45.877 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T14:04:45.877 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:04:45.877 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:04:45.877 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T14:04:45.877 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T14:04:45.877 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:04:45.877 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:04:45.877 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:04:45.878 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:04:45.878 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:04:45.878 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:04:45.878 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32609/ceph-mon.a.asok 2024-09-18T14:04:45.878 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:04:45.879 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.32609/ceph-mon.a.asok config get fsid 2024-09-18T14:04:45.958 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T14:04:45.958 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:04:45.958 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:04:45.958 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T14:04:45.958 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T14:04:45.959 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:04:45.959 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:04:45.959 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:04:45.959 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:04:45.959 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:04:45.959 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:04:45.959 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32609/ceph-mon.a.asok 2024-09-18T14:04:45.959 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:04:45.960 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.32609/ceph-mon.a.asok config get mon_host 2024-09-18T14:04:46.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:637: TEST_divergent_3: run_mgr td/divergent-priors x 2024-09-18T14:04:46.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-09-18T14:04:46.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T14:04:46.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T14:04:46.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T14:04:46.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-09-18T14:04:46.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T14:04:46.890 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T14:04:46.890 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:04:46.890 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:04:46.890 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:04:46.890 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:04:46.891 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:04:46.891 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:04:46.891 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:04:46.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:04:46.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-09-18T14:04:46.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 0 2024-09-18T14:04:46.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-18T14:04:46.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:04:46.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T14:04:46.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:04:46.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-09-18T14:04:46.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T14:04:46.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:04:46.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:04:46.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:04:46.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-18T14:04:46.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-18T14:04:46.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:04:46.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:04:46.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-18T14:04:46.925 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:04:46.925 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:04:46.925 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:04:46.925 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:04:46.925 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:04:46.925 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:04:46.925 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:04:46.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:04:46.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:04:46.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:04:46.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:04:46.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-18T14:04:46.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-18T14:04:46.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:04:46.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:04:46.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:04:46.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:04:46.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:04:46.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:04:46.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-09-18T14:04:46.927 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:04:46.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2d333a34-fa14-4e6f-aca9-f14a57e36c76 2024-09-18T14:04:46.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2d333a34-fa14-4e6f-aca9-f14a57e36c76' 2024-09-18T14:04:46.927 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 2d333a34-fa14-4e6f-aca9-f14a57e36c76 2024-09-18T14:04:46.928 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:04:46.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD+3epmy1FWOBAAve3aFVFNQlrqfABT0kJ+2g== 2024-09-18T14:04:46.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD+3epmy1FWOBAAve3aFVFNQlrqfABT0kJ+2g=="}' 2024-09-18T14:04:46.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2d333a34-fa14-4e6f-aca9-f14a57e36c76 -i td/divergent-priors/0/new.json 2024-09-18T14:04:47.232 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:04:47.242 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-09-18T14:04:47.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD+3epmy1FWOBAAve3aFVFNQlrqfABT0kJ+2g== --osd-uuid 2d333a34-fa14-4e6f-aca9-f14a57e36c76 2024-09-18T14:04:47.276 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:04:47.275+0000 7ff27c81a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:04:47.278 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:04:47.277+0000 7ff27c81a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:04:47.279 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:04:47.278+0000 7ff27c81a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:04:47.279 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:04:47.279+0000 7ff27c81a440 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-18T14:04:47.279 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:04:47.279+0000 7ff27c81a440 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-18T14:04:47.279 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:04:47.279+0000 7ff27c81a440 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-18T14:04:47.317 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:04:47.316+0000 7ff27c81a440 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-18T14:04:52.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-18T14:04:52.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:04:52.647 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T14:04:52.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T14:04:52.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:04:53.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T14:04:53.182 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T14:04:53.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:04:53.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:04:53.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:04:53.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:04:53.213 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:04:53.212+0000 7f360ee10440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:04:53.231 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:04:53.231+0000 7f360ee10440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:04:53.233 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:04:53.232+0000 7f360ee10440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:04:53.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T14:04:53.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:04:53.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:04:53.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:04:53.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:04:53.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:04:53.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:04:53.638 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:04:53.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:04:53.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:04:53.763 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:04:53.763+0000 7f360ee10440 -1 Falling back to public interface 2024-09-18T14:04:54.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:04:55.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:04:55.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:04:55.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:04:55.055 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:04:55.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:04:55.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:04:55.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:04:56.244 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:04:56.244+0000 7f360ee10440 -1 osd.0 0 log_to_monitors true 2024-09-18T14:04:56.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:04:56.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:04:56.481 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:04:56.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:04:56.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:04:56.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:04:56.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:04:57.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:04:57.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:04:57.945 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:04:57.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:04:57.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:04:57.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:04:58.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:04:59.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:04:59.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:04:59.361 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:04:59.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:04:59.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:04:59.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:04:59.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:05:00.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:05:00.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:05:00.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:05:00.750 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:05:00.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:05:00.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:05:01.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:05:02.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:05:02.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:05:02.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:05:02.204 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:05:02.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:05:02.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:05:02.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:05:03.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:05:03.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:05:03.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:05:03.609 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:05:03.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:05:03.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:05:04.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:05:05.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:05:05.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:05:05.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:05:05.059 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:05:05.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:05:05.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:05:05.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:05:06.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:05:06.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:05:06.475 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:05:06.476 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:05:06.476 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:05:06.476 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:05:06.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:05:07.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:05:07.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:05:07.881 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:05:07.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:05:07.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:05:07.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:05:08.746 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:05:09.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:05:09.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:05:09.748 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:05:09.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:05:09.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:05:09.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:05:10.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:05:11.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:05:11.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:05:11.282 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:05:11.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:05:11.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:05:11.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:05:11.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:05:12.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:05:12.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:05:12.981 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:05:12.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:05:12.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:05:12.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:05:13.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:05:14.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:05:14.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:05:14.882 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:05:14.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:05:14.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:05:14.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:05:15.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:05:16.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:05:16.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:05:16.329 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:05:16.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:05:16.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:05:16.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:05:16.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:05:17.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:05:17.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:05:17.876 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:05:17.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:05:17.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:05:17.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:05:18.330 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/60488,v1:127.0.0.1:6803/60488] [v2:127.0.0.1:6804/60488,v1:127.0.0.1:6805/60488] exists,up 2d333a34-fa14-4e6f-aca9-f14a57e36c76 2024-09-18T14:05:18.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:05:18.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:05:18.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:05:18.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-09-18T14:05:18.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 1 2024-09-18T14:05:18.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-18T14:05:18.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:05:18.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T14:05:18.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:05:18.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-09-18T14:05:18.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T14:05:18.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:05:18.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:05:18.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:05:18.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-18T14:05:18.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-18T14:05:18.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:05:18.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:05:18.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-18T14:05:18.336 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:05:18.336 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:05:18.336 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:05:18.336 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:05:18.336 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:05:18.336 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:05:18.336 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:05:18.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:05:18.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:05:18.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:05:18.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:05:18.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-18T14:05:18.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-18T14:05:18.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:05:18.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:05:18.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:05:18.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:05:18.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:05:18.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:05:18.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-09-18T14:05:18.337 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:05:18.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d16b04c1-1e01-47e1-87e5-e8058bd968e4 2024-09-18T14:05:18.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d16b04c1-1e01-47e1-87e5-e8058bd968e4' 2024-09-18T14:05:18.338 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 d16b04c1-1e01-47e1-87e5-e8058bd968e4 2024-09-18T14:05:18.338 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:05:18.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAe3upmBZ4gFRAAn5ULctR5WcZbgq/TUbamdw== 2024-09-18T14:05:18.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAe3upmBZ4gFRAAn5ULctR5WcZbgq/TUbamdw=="}' 2024-09-18T14:05:18.355 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d16b04c1-1e01-47e1-87e5-e8058bd968e4 -i td/divergent-priors/1/new.json 2024-09-18T14:05:18.956 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:05:18.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-09-18T14:05:18.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAe3upmBZ4gFRAAn5ULctR5WcZbgq/TUbamdw== --osd-uuid d16b04c1-1e01-47e1-87e5-e8058bd968e4 2024-09-18T14:05:19.000 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:05:19.000+0000 7f485ff2f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:05:19.002 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:05:19.002+0000 7f485ff2f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:05:19.003 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:05:19.003+0000 7f485ff2f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:05:19.004 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:05:19.003+0000 7f485ff2f440 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-18T14:05:19.004 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:05:19.003+0000 7f485ff2f440 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-18T14:05:19.004 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:05:19.003+0000 7f485ff2f440 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-18T14:05:19.071 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:05:19.072+0000 7f485ff2f440 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-18T14:05:24.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-18T14:05:24.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:05:24.886 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T14:05:24.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T14:05:24.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:05:25.421 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T14:05:25.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T14:05:25.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:05:25.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:05:25.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:05:25.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:05:25.452 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:05:25.452+0000 7fc287b1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:05:25.469 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:05:25.468+0000 7fc287b1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:05:25.470 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:05:25.470+0000 7fc287b1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:05:25.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T14:05:25.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:05:25.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:05:25.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:05:25.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:05:25.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:05:25.883 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:05:25.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:05:25.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:05:25.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:05:26.012 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:05:26.012+0000 7fc287b1d440 -1 Falling back to public interface 2024-09-18T14:05:26.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:05:27.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:05:27.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:05:27.254 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:05:27.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:05:27.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:05:27.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:05:27.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:05:28.481 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:05:28.481+0000 7fc287b1d440 -1 osd.1 0 log_to_monitors true 2024-09-18T14:05:28.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:05:28.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:05:28.687 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:05:28.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:05:28.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:05:28.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:05:29.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:05:30.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:05:30.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:05:30.168 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:05:30.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:05:30.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:05:30.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:05:30.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:05:31.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:05:31.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:05:31.566 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:05:31.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:05:31.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:05:31.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:05:31.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:05:32.985 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:05:32.985 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:05:32.986 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:05:32.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:05:32.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:05:32.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:05:33.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:05:34.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:05:34.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:05:34.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:05:34.373 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:05:34.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:05:34.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:05:34.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:05:35.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:05:35.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:05:35.867 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:05:35.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:05:35.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:05:35.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:05:36.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:05:37.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:05:37.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:05:37.359 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:05:37.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:05:37.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:05:37.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:05:37.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:05:38.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:05:38.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:05:38.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:05:38.776 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:05:38.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:05:38.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:05:39.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:05:40.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:05:40.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:05:40.229 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:05:40.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:05:40.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:05:40.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:05:41.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:05:42.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:05:42.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:05:42.003 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:05:42.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:05:42.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:05:42.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:05:42.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:05:43.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:05:43.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:05:43.797 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:05:43.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:05:43.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:05:43.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:05:44.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:05:45.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:05:45.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:05:45.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:05:45.526 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:05:45.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:05:45.527 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:05:46.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:05:47.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:05:47.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:05:47.271 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:05:47.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:05:47.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:05:47.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:05:48.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:05:49.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:05:49.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:05:49.050 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:05:49.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:05:49.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:05:49.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:05:49.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:05:50.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:05:50.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:05:50.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:05:50.744 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:05:50.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:05:50.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:05:51.132 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/62217,v1:127.0.0.1:6811/62217] [v2:127.0.0.1:6812/62217,v1:127.0.0.1:6813/62217] exists,up d16b04c1-1e01-47e1-87e5-e8058bd968e4 2024-09-18T14:05:51.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:05:51.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:05:51.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:05:51.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-09-18T14:05:51.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 2 2024-09-18T14:05:51.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-18T14:05:51.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:05:51.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T14:05:51.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:05:51.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-09-18T14:05:51.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T14:05:51.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:05:51.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:05:51.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:05:51.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-18T14:05:51.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-18T14:05:51.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:05:51.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:05:51.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-18T14:05:51.137 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:05:51.137 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:05:51.138 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:05:51.138 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:05:51.138 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:05:51.138 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:05:51.138 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:05:51.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:05:51.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:05:51.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:05:51.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:05:51.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-18T14:05:51.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-18T14:05:51.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:05:51.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:05:51.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:05:51.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:05:51.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:05:51.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:05:51.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-09-18T14:05:51.139 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:05:51.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=20ba0a43-7c90-449e-bd64-6203deaebae3 2024-09-18T14:05:51.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 20ba0a43-7c90-449e-bd64-6203deaebae3' 2024-09-18T14:05:51.140 INFO:tasks.workunit.client.0.smithi047.stdout:add osd2 20ba0a43-7c90-449e-bd64-6203deaebae3 2024-09-18T14:05:51.140 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:05:51.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA/3upmZGRSCRAAQ9+usUT5GclxwLALnX7hfg== 2024-09-18T14:05:51.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA/3upmZGRSCRAAQ9+usUT5GclxwLALnX7hfg=="}' 2024-09-18T14:05:51.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 20ba0a43-7c90-449e-bd64-6203deaebae3 -i td/divergent-priors/2/new.json 2024-09-18T14:05:51.721 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:05:51.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-09-18T14:05:51.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA/3upmZGRSCRAAQ9+usUT5GclxwLALnX7hfg== --osd-uuid 20ba0a43-7c90-449e-bd64-6203deaebae3 2024-09-18T14:05:51.765 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:05:51.764+0000 7f865a222440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:05:51.766 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:05:51.766+0000 7f865a222440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:05:51.767 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:05:51.767+0000 7f865a222440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:05:51.768 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:05:51.768+0000 7f865a222440 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-18T14:05:51.768 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:05:51.768+0000 7f865a222440 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-18T14:05:51.768 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:05:51.768+0000 7f865a222440 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-18T14:05:51.836 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:05:51.836+0000 7f865a222440 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-18T14:05:57.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-18T14:05:57.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:05:57.446 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd2 key to auth repository 2024-09-18T14:05:57.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T14:05:57.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:05:57.965 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T14:05:57.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T14:05:57.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:05:57.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:05:57.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:05:57.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:05:57.996 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:05:57.996+0000 7f94d8ffe440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:05:58.015 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:05:58.015+0000 7f94d8ffe440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:05:58.016 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:05:58.016+0000 7f94d8ffe440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:05:58.420 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T14:05:58.420 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:05:58.420 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T14:05:58.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:05:58.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:05:58.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:05:58.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:05:58.421 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:05:58.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:05:58.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:05:58.546 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:05:58.546+0000 7f94d8ffe440 -1 Falling back to public interface 2024-09-18T14:05:58.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:05:59.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:05:59.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:05:59.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:05:59.804 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:05:59.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:05:59.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:06:00.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:06:01.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:06:01.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:06:01.188 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:06:01.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:06:01.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:06:01.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:06:01.237 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:06:01.237+0000 7f94d8ffe440 -1 osd.2 0 log_to_monitors true 2024-09-18T14:06:01.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:06:02.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:06:02.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:06:02.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:06:02.577 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:06:02.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:06:02.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:06:03.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:06:04.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:06:04.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:06:04.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:06:04.040 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:06:04.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:06:04.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:06:04.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:06:05.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:06:05.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:06:05.435 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:06:05.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:06:05.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:06:05.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:06:05.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:06:06.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:06:06.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:06:06.830 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:06:06.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:06:06.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:06:06.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:06:07.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:06:08.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:06:08.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:06:08.223 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:06:08.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:06:08.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:06:08.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:06:08.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:06:09.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:06:09.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:06:09.642 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:06:09.642 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:06:09.642 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:06:09.642 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:06:10.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:06:11.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:06:11.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:06:11.102 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:06:11.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:06:11.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:06:11.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:06:11.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:06:12.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:06:12.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:06:12.530 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:06:12.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:06:12.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:06:12.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:06:12.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:06:13.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:06:13.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:06:13.962 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:06:13.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:06:13.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:06:13.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:06:14.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:06:15.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:06:15.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:06:15.689 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:06:15.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:06:15.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:06:15.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:06:16.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:06:17.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:06:17.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:06:17.517 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:06:17.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:06:17.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:06:17.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:06:18.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:06:19.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:06:19.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:06:19.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:06:19.275 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:06:19.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:06:19.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:06:20.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:06:21.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:06:21.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:06:21.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:06:21.191 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:06:21.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:06:21.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:06:21.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:06:22.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:06:22.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:06:22.587 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:06:22.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:06:22.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:06:22.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:06:23.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:06:24.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:06:24.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:06:24.148 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T14:06:24.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:06:24.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:06:24.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:06:24.532 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/63946,v1:127.0.0.1:6819/63946] [v2:127.0.0.1:6820/63946,v1:127.0.0.1:6821/63946] exists,up 20ba0a43-7c90-449e-bd64-6203deaebae3 2024-09-18T14:06:24.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:06:24.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:06:24.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:06:24.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:643: TEST_divergent_3: ceph osd set noout 2024-09-18T14:06:25.300 INFO:tasks.workunit.client.0.smithi047.stderr:noout is set 2024-09-18T14:06:25.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:644: TEST_divergent_3: ceph osd set noin 2024-09-18T14:06:26.095 INFO:tasks.workunit.client.0.smithi047.stderr:noin is set 2024-09-18T14:06:26.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:645: TEST_divergent_3: ceph osd set nodown 2024-09-18T14:06:26.926 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T14:06:26.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:646: TEST_divergent_3: create_pool test 1 1 2024-09-18T14:06:26.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-18T14:06:27.647 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test' already exists 2024-09-18T14:06:27.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T14:06:28.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:647: TEST_divergent_3: ceph osd pool set test size 3 2024-09-18T14:06:29.328 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 1 size to 3 2024-09-18T14:06:29.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:648: TEST_divergent_3: ceph osd pool set test min_size 2 2024-09-18T14:06:30.056 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 1 min_size to 2 2024-09-18T14:06:30.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:651: TEST_divergent_3: ceph osd pool set test pg_autoscale_mode on 2024-09-18T14:06:30.794 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 1 pg_autoscale_mode to on 2024-09-18T14:06:30.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:653: TEST_divergent_3: divergent=-1 2024-09-18T14:06:30.805 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: date +%s 2024-09-18T14:06:30.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: start_time=1726668390 2024-09-18T14:06:30.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:655: TEST_divergent_3: max_duration=300 2024-09-18T14:06:30.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: '[' -1 -le -1 ']' 2024-09-18T14:06:30.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:659: TEST_divergent_3: flush_pg_stats 2024-09-18T14:06:30.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:06:30.806 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:06:31.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:06:31.185 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T14:06:31.185 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T14:06:31.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:06:31.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:06:31.185 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:06:31.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836498 2024-09-18T14:06:31.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836498 2024-09-18T14:06:31.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836498' 2024-09-18T14:06:31.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:06:31.314 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:06:31.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672975 2024-09-18T14:06:31.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672975 2024-09-18T14:06:31.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836498 1-42949672975' 2024-09-18T14:06:31.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:06:31.465 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:06:31.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509449 2024-09-18T14:06:31.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509449 2024-09-18T14:06:31.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836498 1-42949672975 2-64424509449' 2024-09-18T14:06:31.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:06:31.580 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836498 2024-09-18T14:06:31.580 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:06:31.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:06:31.582 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836498 2024-09-18T14:06:31.582 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:06:31.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836498 2024-09-18T14:06:31.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-09-18T14:06:31.583 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836498 2024-09-18T14:06:31.584 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:06:31.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836496 -lt 21474836498 2024-09-18T14:06:31.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:06:32.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:06:32.961 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:06:33.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836498 -lt 21474836498 2024-09-18T14:06:33.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:06:33.412 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672975 2024-09-18T14:06:33.412 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:06:33.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:06:33.413 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672975 2024-09-18T14:06:33.413 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:06:33.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672975 2024-09-18T14:06:33.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-09-18T14:06:33.414 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672975 2024-09-18T14:06:33.415 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:06:33.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672976 -lt 42949672975 2024-09-18T14:06:33.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:06:33.838 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509449 2024-09-18T14:06:33.838 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:06:33.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:06:33.840 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509449 2024-09-18T14:06:33.840 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:06:33.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509449 2024-09-18T14:06:33.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-09-18T14:06:33.841 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509449 2024-09-18T14:06:33.841 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:06:34.247 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509449 -lt 64424509449 2024-09-18T14:06:34.248 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:660: TEST_divergent_3: wait_for_clean 2024-09-18T14:06:34.248 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:06:34.248 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:06:34.248 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:06:34.248 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:06:34.249 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:06:34.249 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:06:34.249 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:06:34.249 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:06:34.249 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:06:34.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T14:06:34.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:06:34.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:06:34.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:06:34.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:06:34.341 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:06:34.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:06:34.764 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T14:06:34.764 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T14:06:34.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:06:34.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:06:34.764 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:06:34.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836501 2024-09-18T14:06:34.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836501 2024-09-18T14:06:34.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836501' 2024-09-18T14:06:34.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:06:34.922 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:06:35.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672978 2024-09-18T14:06:35.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672978 2024-09-18T14:06:35.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836501 1-42949672978' 2024-09-18T14:06:35.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:06:35.073 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:06:35.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509452 2024-09-18T14:06:35.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509452 2024-09-18T14:06:35.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836501 1-42949672978 2-64424509452' 2024-09-18T14:06:35.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:06:35.189 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836501 2024-09-18T14:06:35.189 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:06:35.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:06:35.191 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836501 2024-09-18T14:06:35.191 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:06:35.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836501 2024-09-18T14:06:35.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-09-18T14:06:35.192 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836501 2024-09-18T14:06:35.193 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:06:35.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836499 -lt 21474836501 2024-09-18T14:06:35.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:06:36.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:06:36.610 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:06:36.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836501 -lt 21474836501 2024-09-18T14:06:36.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:06:36.994 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672978 2024-09-18T14:06:36.994 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:06:36.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:06:36.996 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672978 2024-09-18T14:06:36.996 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:06:36.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672978 2024-09-18T14:06:36.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-09-18T14:06:36.997 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672978 2024-09-18T14:06:36.997 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:06:37.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672978 -lt 42949672978 2024-09-18T14:06:37.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:06:37.369 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509452 2024-09-18T14:06:37.369 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:06:37.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:06:37.370 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509452 2024-09-18T14:06:37.370 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:06:37.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509452 2024-09-18T14:06:37.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-09-18T14:06:37.372 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509452 2024-09-18T14:06:37.372 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:06:37.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509452 -lt 64424509452 2024-09-18T14:06:37.787 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:06:37.788 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:06:37.788 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:06:38.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T14:06:38.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:06:38.286 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:06:38.286 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:06:38.286 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:06:38.286 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:06:38.286 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:06:38.286 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:06:38.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T14:06:38.678 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:06:38.679 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:06:38.679 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:06:39.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T14:06:39.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:06:39.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:06:39.193 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: ceph pg dump pgs --format=json 2024-09-18T14:06:39.193 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: jq '.pg_stats[0].up_primary' 2024-09-18T14:06:39.544 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:06:39.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: divergent=1 2024-09-18T14:06:39.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:664: TEST_divergent_3: echo 'primary and soon to be divergent is 1' 2024-09-18T14:06:39.555 INFO:tasks.workunit.client.0.smithi047.stdout:primary and soon to be divergent is 1 2024-09-18T14:06:39.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:665: TEST_divergent_3: ceph pg dump pgs 2024-09-18T14:06:39.905 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:06:39.905 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:06:28.458244+0000 0'0 28:11 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:05:06.732590+0000 0 0 2024-09-18T14:06:39.905 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:06:39.906 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:06:39.906 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:06:39.916 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:667: TEST_divergent_3: date +%s 2024-09-18T14:06:39.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:667: TEST_divergent_3: current_time=1726668399 2024-09-18T14:06:39.917 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:668: TEST_divergent_3: expr 1726668399 - 1726668390 2024-09-18T14:06:39.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:668: TEST_divergent_3: elapsed_time=9 2024-09-18T14:06:39.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:669: TEST_divergent_3: '[' 9 -gt 300 ']' 2024-09-18T14:06:39.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: '[' 1 -le -1 ']' 2024-09-18T14:06:39.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:675: TEST_divergent_3: local non_divergent= 2024-09-18T14:06:39.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-09-18T14:06:39.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 0 = 1 ']' 2024-09-18T14:06:39.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: non_divergent=' 0' 2024-09-18T14:06:39.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-09-18T14:06:39.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 1 = 1 ']' 2024-09-18T14:06:39.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:679: TEST_divergent_3: continue 2024-09-18T14:06:39.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-09-18T14:06:39.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 2 = 1 ']' 2024-09-18T14:06:39.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: non_divergent=' 0 2' 2024-09-18T14:06:39.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:684: TEST_divergent_3: echo 'writing initial objects' 2024-09-18T14:06:39.920 INFO:tasks.workunit.client.0.smithi047.stdout:writing initial objects 2024-09-18T14:06:39.921 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: seq 1 100 2024-09-18T14:06:39.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:39.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_1 /etc/fstab 2024-09-18T14:06:40.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:40.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_2 /etc/fstab 2024-09-18T14:06:40.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:40.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_3 /etc/fstab 2024-09-18T14:06:40.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:40.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_4 /etc/fstab 2024-09-18T14:06:40.551 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:40.551 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_5 /etc/fstab 2024-09-18T14:06:40.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:40.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_6 /etc/fstab 2024-09-18T14:06:40.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:40.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_7 /etc/fstab 2024-09-18T14:06:41.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:41.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_8 /etc/fstab 2024-09-18T14:06:41.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:41.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_9 /etc/fstab 2024-09-18T14:06:41.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:41.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_10 /etc/fstab 2024-09-18T14:06:41.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:41.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_11 /etc/fstab 2024-09-18T14:06:41.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:41.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_12 /etc/fstab 2024-09-18T14:06:41.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:41.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_13 /etc/fstab 2024-09-18T14:06:42.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:42.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_14 /etc/fstab 2024-09-18T14:06:42.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:42.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_15 /etc/fstab 2024-09-18T14:06:42.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:42.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_16 /etc/fstab 2024-09-18T14:06:42.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:42.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_17 /etc/fstab 2024-09-18T14:06:42.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:42.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_18 /etc/fstab 2024-09-18T14:06:42.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:42.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_19 /etc/fstab 2024-09-18T14:06:43.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:43.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_20 /etc/fstab 2024-09-18T14:06:43.157 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:43.157 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_21 /etc/fstab 2024-09-18T14:06:43.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:43.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_22 /etc/fstab 2024-09-18T14:06:43.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:43.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_23 /etc/fstab 2024-09-18T14:06:43.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:43.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_24 /etc/fstab 2024-09-18T14:06:43.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:43.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_25 /etc/fstab 2024-09-18T14:06:43.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:43.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_26 /etc/fstab 2024-09-18T14:06:44.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:44.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_27 /etc/fstab 2024-09-18T14:06:44.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:44.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_28 /etc/fstab 2024-09-18T14:06:44.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:44.406 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_29 /etc/fstab 2024-09-18T14:06:44.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:44.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_30 /etc/fstab 2024-09-18T14:06:44.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:44.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_31 /etc/fstab 2024-09-18T14:06:44.875 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:44.875 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_32 /etc/fstab 2024-09-18T14:06:45.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:45.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_33 /etc/fstab 2024-09-18T14:06:45.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:45.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_34 /etc/fstab 2024-09-18T14:06:45.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:45.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_35 /etc/fstab 2024-09-18T14:06:45.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:45.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_36 /etc/fstab 2024-09-18T14:06:45.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:45.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_37 /etc/fstab 2024-09-18T14:06:45.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:45.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_38 /etc/fstab 2024-09-18T14:06:45.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:45.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_39 /etc/fstab 2024-09-18T14:06:46.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:46.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_40 /etc/fstab 2024-09-18T14:06:46.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:46.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_41 /etc/fstab 2024-09-18T14:06:46.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:46.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_42 /etc/fstab 2024-09-18T14:06:46.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:46.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_43 /etc/fstab 2024-09-18T14:06:46.761 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:46.761 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_44 /etc/fstab 2024-09-18T14:06:46.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:46.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_45 /etc/fstab 2024-09-18T14:06:47.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:47.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_46 /etc/fstab 2024-09-18T14:06:47.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:47.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_47 /etc/fstab 2024-09-18T14:06:47.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:47.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_48 /etc/fstab 2024-09-18T14:06:47.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:47.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_49 /etc/fstab 2024-09-18T14:06:47.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:47.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_50 /etc/fstab 2024-09-18T14:06:47.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:47.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_51 /etc/fstab 2024-09-18T14:06:48.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:48.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_52 /etc/fstab 2024-09-18T14:06:48.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:48.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_53 /etc/fstab 2024-09-18T14:06:48.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:48.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_54 /etc/fstab 2024-09-18T14:06:48.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:48.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_55 /etc/fstab 2024-09-18T14:06:48.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:48.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_56 /etc/fstab 2024-09-18T14:06:48.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:48.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_57 /etc/fstab 2024-09-18T14:06:48.948 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:48.948 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_58 /etc/fstab 2024-09-18T14:06:49.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:49.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_59 /etc/fstab 2024-09-18T14:06:49.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:49.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_60 /etc/fstab 2024-09-18T14:06:49.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:49.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_61 /etc/fstab 2024-09-18T14:06:49.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:49.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_62 /etc/fstab 2024-09-18T14:06:49.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:49.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_63 /etc/fstab 2024-09-18T14:06:49.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:49.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_64 /etc/fstab 2024-09-18T14:06:50.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:50.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_65 /etc/fstab 2024-09-18T14:06:50.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:50.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_66 /etc/fstab 2024-09-18T14:06:50.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:50.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_67 /etc/fstab 2024-09-18T14:06:50.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:50.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_68 /etc/fstab 2024-09-18T14:06:50.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:50.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_69 /etc/fstab 2024-09-18T14:06:50.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:50.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_70 /etc/fstab 2024-09-18T14:06:51.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:51.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_71 /etc/fstab 2024-09-18T14:06:51.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:51.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_72 /etc/fstab 2024-09-18T14:06:51.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:51.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_73 /etc/fstab 2024-09-18T14:06:51.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:51.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_74 /etc/fstab 2024-09-18T14:06:51.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:51.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_75 /etc/fstab 2024-09-18T14:06:52.095 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:52.095 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_76 /etc/fstab 2024-09-18T14:06:52.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:52.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_77 /etc/fstab 2024-09-18T14:06:52.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:52.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_78 /etc/fstab 2024-09-18T14:06:52.551 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:52.551 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_79 /etc/fstab 2024-09-18T14:06:52.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:52.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_80 /etc/fstab 2024-09-18T14:06:52.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:52.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_81 /etc/fstab 2024-09-18T14:06:54.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:54.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_82 /etc/fstab 2024-09-18T14:06:54.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:54.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_83 /etc/fstab 2024-09-18T14:06:54.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:54.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_84 /etc/fstab 2024-09-18T14:06:54.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:54.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_85 /etc/fstab 2024-09-18T14:06:54.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:54.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_86 /etc/fstab 2024-09-18T14:06:54.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:54.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_87 /etc/fstab 2024-09-18T14:06:55.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:55.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_88 /etc/fstab 2024-09-18T14:06:55.285 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:55.285 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_89 /etc/fstab 2024-09-18T14:06:55.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:55.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_90 /etc/fstab 2024-09-18T14:06:55.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:55.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_91 /etc/fstab 2024-09-18T14:06:55.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:55.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_92 /etc/fstab 2024-09-18T14:06:55.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:55.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_93 /etc/fstab 2024-09-18T14:06:56.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:56.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_94 /etc/fstab 2024-09-18T14:06:56.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:56.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_95 /etc/fstab 2024-09-18T14:06:56.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:56.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_96 /etc/fstab 2024-09-18T14:06:56.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:56.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_97 /etc/fstab 2024-09-18T14:06:56.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:56.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_98 /etc/fstab 2024-09-18T14:06:56.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:56.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_99 /etc/fstab 2024-09-18T14:06:57.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-18T14:06:57.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_100 /etc/fstab 2024-09-18T14:06:57.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:691: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-18T14:06:57.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:691: TEST_divergent_3: wait_for_clean 2024-09-18T14:06:57.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:06:57.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:06:57.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:06:57.298 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-18T14:06:57.298 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:06:57.298 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:06:57.299 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:06:57.299 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:06:57.299 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:06:57.355 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-18T14:06:57.355 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:06:57.355 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:06:57.355 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:06:57.355 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:06:57.355 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:06:57.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:06:57.814 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T14:06:57.814 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T14:06:57.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:06:57.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:06:57.814 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:06:57.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836507 2024-09-18T14:06:57.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836507 2024-09-18T14:06:57.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836507' 2024-09-18T14:06:57.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:06:57.930 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:06:58.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672985 2024-09-18T14:06:58.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672985 2024-09-18T14:06:58.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836507 1-42949672985' 2024-09-18T14:06:58.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:06:58.048 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:06:58.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509458 2024-09-18T14:06:58.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509458 2024-09-18T14:06:58.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836507 1-42949672985 2-64424509458' 2024-09-18T14:06:58.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:06:58.164 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836507 2024-09-18T14:06:58.164 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:06:58.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:06:58.166 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836507 2024-09-18T14:06:58.166 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:06:58.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836507 2024-09-18T14:06:58.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-09-18T14:06:58.167 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836507 2024-09-18T14:06:58.167 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:06:58.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836505 -lt 21474836507 2024-09-18T14:06:58.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:06:59.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:06:59.541 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:06:59.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836507 -lt 21474836507 2024-09-18T14:06:59.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:06:59.937 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672985 2024-09-18T14:06:59.937 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:06:59.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:06:59.938 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672985 2024-09-18T14:06:59.938 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:06:59.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672985 2024-09-18T14:06:59.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-09-18T14:06:59.940 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672985 2024-09-18T14:06:59.940 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:07:00.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672985 -lt 42949672985 2024-09-18T14:07:00.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:07:00.345 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509458 2024-09-18T14:07:00.345 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:07:00.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:07:00.347 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509458 2024-09-18T14:07:00.347 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:07:00.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509458 2024-09-18T14:07:00.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-09-18T14:07:00.349 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509458 2024-09-18T14:07:00.349 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:07:00.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509459 -lt 64424509458 2024-09-18T14:07:00.737 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:07:00.738 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:07:00.738 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:07:01.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-09-18T14:07:01.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:07:01.276 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:07:01.276 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:07:01.276 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:07:01.276 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:07:01.276 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:07:01.276 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:07:01.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=21 2024-09-18T14:07:01.678 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:07:01.678 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:07:01.678 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:07:02.197 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 21 = 32 2024-09-18T14:07:02.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 21 '!=' -1 2024-09-18T14:07:02.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T14:07:02.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=21 2024-09-18T14:07:02.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:07:02.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:07:02.299 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:07:02.299 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:07:02.299 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:07:02.299 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:07:02.299 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:07:02.299 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:07:02.300 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:07:02.300 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:07:02.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=21 2024-09-18T14:07:02.677 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:07:02.677 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:07:02.677 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:07:03.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 21 = 32 2024-09-18T14:07:03.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 21 '!=' 21 2024-09-18T14:07:03.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:07:03.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:07:03.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:07:03.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:07:03.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:07:03.205 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:07:03.205 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:07:03.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:07:03.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:07:03.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-09-18T14:07:03.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:07:03.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-18T14:07:03.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:07:03.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:07:03.911 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:07:03.911 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:07:03.911 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:07:03.911 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:07:03.911 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:07:03.911 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:07:04.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=21 2024-09-18T14:07:04.280 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:07:04.280 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:07:04.280 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:07:04.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 21 = 32 2024-09-18T14:07:04.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 21 '!=' 21 2024-09-18T14:07:04.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:07:04.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:07:04.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:07:04.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:07:04.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:07:04.849 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:07:04.849 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:07:05.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:07:05.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:07:05.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-09-18T14:07:05.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:07:05.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-18T14:07:05.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:07:05.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:07:05.760 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:07:05.760 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:07:05.760 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:07:05.760 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:07:05.760 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:07:05.761 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:07:06.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=21 2024-09-18T14:07:06.135 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:07:06.135 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:07:06.135 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:07:06.642 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 21 = 32 2024-09-18T14:07:06.642 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 21 '!=' 21 2024-09-18T14:07:06.642 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:07:06.642 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:07:06.642 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:07:06.642 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:07:06.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:07:06.643 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:07:06.643 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:07:07.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:07:07.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:07:07.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-09-18T14:07:07.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:07:07.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-09-18T14:07:08.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:07:08.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:07:08.016 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:07:08.016 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:07:08.016 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:07:08.016 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:07:08.017 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:07:08.017 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:07:08.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-09-18T14:07:08.397 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:07:08.398 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:07:08.398 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:07:08.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-09-18T14:07:08.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:07:08.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:07:08.926 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:693: TEST_divergent_3: get_pg test existing_1 2024-09-18T14:07:08.926 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-18T14:07:08.926 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-09-18T14:07:08.926 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-09-18T14:07:08.926 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T14:07:09.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:693: TEST_divergent_3: local pgid=1.1b 2024-09-18T14:07:09.394 INFO:tasks.workunit.client.0.smithi047.stdout:blackholing osds 0 2 2024-09-18T14:07:09.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: echo 'blackholing osds 0 2' 2024-09-18T14:07:09.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:697: TEST_divergent_3: ceph pg dump pgs 2024-09-18T14:07:09.828 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:07:09.828 INFO:tasks.workunit.client.0.smithi047.stdout:1.1f 2 0 0 0 0 1207 0 0 14 66 14 active+clean 2024-09-18T14:06:59.727759+0000 32'80 36:103 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:09.828 INFO:tasks.workunit.client.0.smithi047.stdout:1.1e 3 0 0 0 0 1690 0 0 15 66 15 active+clean 2024-09-18T14:07:07.768142+0000 34'81 38:116 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:09.828 INFO:tasks.workunit.client.0.smithi047.stdout:1.1d 2 0 0 0 0 1207 0 0 14 66 14 active+clean 2024-09-18T14:06:59.726438+0000 32'80 36:103 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:09.828 INFO:tasks.workunit.client.0.smithi047.stdout:1.b 4 0 0 0 0 1691 0 0 15 66 15 active+clean 2024-09-18T14:06:59.727500+0000 34'81 36:116 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:09.828 INFO:tasks.workunit.client.0.smithi047.stdout:1.a 3 0 0 0 0 1208 0 0 14 66 14 active+clean 2024-09-18T14:07:07.767880+0000 32'80 38:103 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:09.828 INFO:tasks.workunit.client.0.smithi047.stdout:1.9 5 0 0 0 0 2174 0 0 16 66 16 active+clean 2024-09-18T14:06:59.726256+0000 34'82 36:117 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:09.828 INFO:tasks.workunit.client.0.smithi047.stdout:1.8 5 0 0 0 0 2174 0 0 16 66 16 active+clean 2024-09-18T14:06:53.868909+0000 34'82 35:108 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:09.829 INFO:tasks.workunit.client.0.smithi047.stdout:1.7 3 0 0 0 0 1208 0 0 14 66 14 active+clean 2024-09-18T14:06:59.728036+0000 32'80 36:103 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:09.829 INFO:tasks.workunit.client.0.smithi047.stdout:1.6 3 0 0 0 0 1208 0 0 14 66 14 active+clean 2024-09-18T14:07:07.767115+0000 32'80 38:103 [0,1,2] 0 [0,1,2] 0 32'80 2024-09-18T14:06:58.202394+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:09.829 INFO:tasks.workunit.client.0.smithi047.stdout:1.5 4 0 0 0 0 1691 0 0 15 66 15 active+clean 2024-09-18T14:06:59.726090+0000 34'81 36:116 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:09.829 INFO:tasks.workunit.client.0.smithi047.stdout:1.4 3 0 0 0 0 1208 0 0 14 66 14 active+clean 2024-09-18T14:06:53.869682+0000 32'80 35:106 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:09.829 INFO:tasks.workunit.client.0.smithi047.stdout:1.3 4 0 0 0 0 1691 0 0 15 66 15 active+clean 2024-09-18T14:06:59.726487+0000 34'81 36:116 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:09.829 INFO:tasks.workunit.client.0.smithi047.stdout:1.2 3 0 0 0 0 1208 0 0 14 66 14 active+clean 2024-09-18T14:07:07.767167+0000 32'80 38:103 [0,1,2] 0 [0,1,2] 0 32'80 2024-09-18T14:07:04.230275+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:09.829 INFO:tasks.workunit.client.0.smithi047.stdout:1.c 4 0 0 0 0 1691 0 0 15 66 15 active+clean 2024-09-18T14:06:53.871157+0000 34'81 35:107 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:09.829 INFO:tasks.workunit.client.0.smithi047.stdout:1.1 4 0 0 0 0 1691 0 0 15 66 15 active+clean 2024-09-18T14:06:59.625240+0000 34'81 36:116 [2,0,1] 2 [2,0,1] 2 34'81 2024-09-18T14:06:56.389936+0000 34'81 2024-09-18T14:06:56.389936+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:09.829 INFO:tasks.workunit.client.0.smithi047.stdout:1.1c 2 0 0 0 0 1207 0 0 14 66 14 active+clean 2024-09-18T14:06:53.870016+0000 32'80 35:106 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:09.830 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 3 0 0 0 0 1449 0 0 15 66 15 active+clean 2024-09-18T14:06:54.303752+0000 34'81 35:118 [1,0,2] 1 [1,0,2] 1 32'80 2024-09-18T14:06:54.303691+0000 32'80 2024-09-18T14:06:54.303691+0000 0 1 periodic scrub scheduled @ 2024-09-20T01:43:21.549213+0000 2 0 2024-09-18T14:07:09.830 INFO:tasks.workunit.client.0.smithi047.stdout:1.d 4 0 0 0 0 1691 0 0 15 66 15 active+clean 2024-09-18T14:06:59.726679+0000 34'81 36:116 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:09.830 INFO:tasks.workunit.client.0.smithi047.stdout:1.e 5 0 0 0 0 2174 0 0 16 66 16 active+clean 2024-09-18T14:07:07.767927+0000 34'82 38:117 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:09.830 INFO:tasks.workunit.client.0.smithi047.stdout:1.f 4 0 0 0 0 1932 0 0 14 66 14 active+clean 2024-09-18T14:07:00.547631+0000 32'80 36:113 [2,0,1] 2 [2,0,1] 2 32'80 2024-09-18T14:07:00.547567+0000 32'80 2024-09-18T14:07:00.547567+0000 0 1 periodic scrub scheduled @ 2024-09-19T16:32:07.301994+0000 4 0 2024-09-18T14:07:09.830 INFO:tasks.workunit.client.0.smithi047.stdout:1.10 4 0 0 0 0 1691 0 0 15 66 15 active+clean 2024-09-18T14:06:53.872008+0000 34'81 35:107 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:09.830 INFO:tasks.workunit.client.0.smithi047.stdout:1.11 2 0 0 0 0 1207 0 0 14 66 14 active+clean 2024-09-18T14:06:59.588575+0000 32'80 36:103 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:09.830 INFO:tasks.workunit.client.0.smithi047.stdout:1.12 1 0 0 0 0 483 0 0 14 66 14 active+clean 2024-09-18T14:07:08.487386+0000 32'80 38:113 [0,1,2] 0 [0,1,2] 0 32'80 2024-09-18T14:07:08.487332+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T19:19:08.320223+0000 1 0 2024-09-18T14:07:09.830 INFO:tasks.workunit.client.0.smithi047.stdout:1.13 3 0 0 0 0 1690 0 0 15 66 15 active+clean 2024-09-18T14:06:59.625670+0000 34'81 36:116 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:09.830 INFO:tasks.workunit.client.0.smithi047.stdout:1.14 3 0 0 0 0 1690 0 0 15 66 15 active+clean 2024-09-18T14:06:53.871541+0000 34'81 35:107 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:09.830 INFO:tasks.workunit.client.0.smithi047.stdout:1.15 2 0 0 0 0 1207 0 0 14 66 14 active+clean 2024-09-18T14:06:59.727249+0000 32'80 36:103 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:09.830 INFO:tasks.workunit.client.0.smithi047.stdout:1.16 2 0 0 0 0 1207 0 0 14 66 14 active+clean 2024-09-18T14:07:07.767353+0000 32'80 38:103 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:09.831 INFO:tasks.workunit.client.0.smithi047.stdout:1.17 2 0 0 0 0 1207 0 0 14 66 14 active+clean 2024-09-18T14:06:59.625194+0000 32'80 36:103 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:09.831 INFO:tasks.workunit.client.0.smithi047.stdout:1.18 3 0 0 0 0 1690 0 0 15 66 15 active+clean 2024-09-18T14:06:53.871248+0000 34'81 35:107 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:09.831 INFO:tasks.workunit.client.0.smithi047.stdout:1.19 2 0 0 0 0 1207 0 0 14 66 14 active+clean 2024-09-18T14:06:59.726283+0000 32'80 36:103 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:09.831 INFO:tasks.workunit.client.0.smithi047.stdout:1.1a 3 0 0 0 0 1690 0 0 15 66 15 active+clean 2024-09-18T14:07:07.767321+0000 34'81 38:116 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:09.831 INFO:tasks.workunit.client.0.smithi047.stdout:1.1b 3 0 0 0 0 1690 0 0 15 66 15 active+clean 2024-09-18T14:06:59.728298+0000 34'81 36:116 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:09.831 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:07:09.831 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:07:09.831 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:07:09.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: for i in $non_divergent 2024-09-18T14:07:09.839 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: get_asok_path osd.0 2024-09-18T14:07:09.839 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:07:09.839 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:07:09.839 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:07:09.839 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:07:09.839 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:07:09.839 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32609/ceph-osd.0.asok 2024-09-18T14:07:09.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: CEPH_ARGS= 2024-09-18T14:07:09.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.32609/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-18T14:07:09.914 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T14:07:09.915 INFO:tasks.workunit.client.0.smithi047.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-18T14:07:09.915 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T14:07:09.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: for i in $non_divergent 2024-09-18T14:07:09.922 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: get_asok_path osd.2 2024-09-18T14:07:09.922 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:07:09.923 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:07:09.923 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:07:09.923 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:07:09.923 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:07:09.923 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32609/ceph-osd.2.asok 2024-09-18T14:07:09.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: CEPH_ARGS= 2024-09-18T14:07:09.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.32609/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-18T14:07:09.997 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T14:07:09.997 INFO:tasks.workunit.client.0.smithi047.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-18T14:07:09.997 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T14:07:10.004 INFO:tasks.workunit.client.0.smithi047.stdout:create new divergent objects 2024-09-18T14:07:10.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: echo 'create new divergent objects' 2024-09-18T14:07:10.004 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: seq 1 2 2024-09-18T14:07:10.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-18T14:07:10.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-18T14:07:10.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: rados -p test create newobject_1 2024-09-18T14:07:10.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:710: TEST_divergent_3: echo 'writing divergent objects' 2024-09-18T14:07:10.006 INFO:tasks.workunit.client.0.smithi047.stdout:writing divergent objects 2024-09-18T14:07:10.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: rados -p test create newobject_2 2024-09-18T14:07:10.006 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: seq 1 2 2024-09-18T14:07:10.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-18T14:07:10.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-18T14:07:10.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: rados -p test put existing_1 /etc/resolv.conf 2024-09-18T14:07:10.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: echo 'remove divergent objects' 2024-09-18T14:07:10.007 INFO:tasks.workunit.client.0.smithi047.stdout:remove divergent objects 2024-09-18T14:07:10.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: rados -p test put existing_2 /etc/resolv.conf 2024-09-18T14:07:10.008 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: seq 1 2 2024-09-18T14:07:10.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-18T14:07:10.009 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: expr 1 + 2 2024-09-18T14:07:10.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: rmi=3 2024-09-18T14:07:10.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-18T14:07:10.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: rados -p test rm existing_3 2024-09-18T14:07:10.011 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: expr 2 + 2 2024-09-18T14:07:10.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: rmi=4 2024-09-18T14:07:10.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:722: TEST_divergent_3: sleep 10 2024-09-18T14:07:10.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: rados -p test rm existing_4 2024-09-18T14:07:20.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:723: TEST_divergent_3: killall -9 rados 2024-09-18T14:07:20.018 INFO:tasks.workunit.client.0.smithi047.stderr:rados: no process found 2024-09-18T14:07:20.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: echo 'killing all the osds' 2024-09-18T14:07:20.019 INFO:tasks.workunit.client.0.smithi047.stdout:killing all the osds 2024-09-18T14:07:20.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:727: TEST_divergent_3: ceph pg dump pgs 2024-09-18T14:07:20.386 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:07:20.386 INFO:tasks.workunit.client.0.smithi047.stdout:1.1f 2 0 0 0 0 1207 0 0 14 66 14 active+clean 2024-09-18T14:07:09.761988+0000 32'80 43:120 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:20.386 INFO:tasks.workunit.client.0.smithi047.stdout:1.1e 3 0 0 0 0 1690 0 0 15 66 15 active+clean 2024-09-18T14:07:15.417434+0000 34'81 44:119 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:20.386 INFO:tasks.workunit.client.0.smithi047.stdout:1.1d 2 0 0 0 0 1207 0 0 14 66 14 active+clean 2024-09-18T14:06:59.726438+0000 32'80 41:109 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:20.386 INFO:tasks.workunit.client.0.smithi047.stdout:1.b 4 0 0 0 0 1691 0 0 15 66 15 active+clean 2024-09-18T14:07:09.761777+0000 34'81 43:121 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:20.387 INFO:tasks.workunit.client.0.smithi047.stdout:1.a 3 0 0 0 0 1208 0 0 14 66 14 active+clean 2024-09-18T14:07:07.767880+0000 32'80 44:110 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:20.387 INFO:tasks.workunit.client.0.smithi047.stdout:1.9 5 0 0 0 0 2086 0 0 17 66 17 active+clean 2024-09-18T14:07:11.436936+0000 40'83 41:134 [2,0,1] 2 [2,0,1] 2 40'83 2024-09-18T14:07:11.436862+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-20T00:39:30.586095+0000 5 0 2024-09-18T14:07:20.387 INFO:tasks.workunit.client.0.smithi047.stdout:1.8 3 0 0 0 0 1449 0 0 16 66 16 active+clean 2024-09-18T14:07:15.131335+0000 34'82 43:126 [1,0,2] 1 [1,0,2] 1 34'82 2024-09-18T14:07:15.131268+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T17:23:40.244015+0000 3 0 2024-09-18T14:07:20.387 INFO:tasks.workunit.client.0.smithi047.stdout:1.7 3 0 0 0 0 1208 0 0 14 66 14 active+clean 2024-09-18T14:07:09.761807+0000 32'80 43:120 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:20.387 INFO:tasks.workunit.client.0.smithi047.stdout:1.6 4 0 0 0 0 1932 0 0 14 66 14 active+clean 2024-09-18T14:07:15.509884+0000 32'80 44:128 [1,0,2] 1 [1,0,2] 1 32'80 2024-09-18T14:06:58.202394+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T20:55:47.501665+0000 4 0 2024-09-18T14:07:20.387 INFO:tasks.workunit.client.0.smithi047.stdout:1.5 6 0 0 0 0 2898 0 0 15 66 15 active+clean 2024-09-18T14:07:06.420594+0000 34'81 41:132 [2,0,1] 2 [2,0,1] 2 34'81 2024-09-18T14:07:06.420533+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T21:49:34.160706+0000 6 0 2024-09-18T14:07:20.387 INFO:tasks.workunit.client.0.smithi047.stdout:1.4 2 0 0 0 0 966 0 0 14 66 14 active+clean 2024-09-18T14:07:08.170532+0000 32'80 43:124 [1,0,2] 1 [1,0,2] 1 32'80 2024-09-18T14:07:08.170467+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T16:10:40.115462+0000 2 0 2024-09-18T14:07:20.387 INFO:tasks.workunit.client.0.smithi047.stdout:1.3 4 0 0 0 0 1691 0 0 15 66 15 active+clean 2024-09-18T14:07:09.761872+0000 34'81 43:121 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:20.387 INFO:tasks.workunit.client.0.smithi047.stdout:1.2 1 0 0 0 0 483 0 0 14 66 14 active+clean 2024-09-18T14:07:18.506369+0000 32'80 45:120 [0,1,2] 0 [0,1,2] 0 32'80 2024-09-18T14:07:18.506303+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T23:05:42.596177+0000 1 0 2024-09-18T14:07:20.387 INFO:tasks.workunit.client.0.smithi047.stdout:1.c 3 0 0 0 0 1449 0 0 15 66 15 active+clean 2024-09-18T14:07:12.133223+0000 34'81 43:125 [1,0,2] 1 [1,0,2] 1 34'81 2024-09-18T14:07:12.133156+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T19:41:59.371626+0000 3 0 2024-09-18T14:07:20.387 INFO:tasks.workunit.client.0.smithi047.stdout:1.1 2 0 0 0 0 966 0 0 15 66 15 active+clean 2024-09-18T14:07:13.379547+0000 34'81 42:132 [2,0,1] 2 [2,0,1] 2 34'81 2024-09-18T14:07:13.379437+0000 34'81 2024-09-18T14:06:56.389936+0000 0 1 periodic scrub scheduled @ 2024-09-20T00:39:31.640524+0000 2 0 2024-09-18T14:07:20.387 INFO:tasks.workunit.client.0.smithi047.stdout:1.1c 2 0 0 0 0 966 0 0 14 66 14 active+clean 2024-09-18T14:07:16.092389+0000 32'80 44:124 [1,0,2] 1 [1,0,2] 1 32'80 2024-09-18T14:07:16.092318+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T15:53:36.670105+0000 2 0 2024-09-18T14:07:20.388 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 3 0 0 0 0 1449 0 0 15 66 15 active+clean 2024-09-18T14:06:54.303752+0000 34'81 43:126 [1,0,2] 1 [1,0,2] 1 32'80 2024-09-18T14:06:54.303691+0000 32'80 2024-09-18T14:06:54.303691+0000 0 1 periodic scrub scheduled @ 2024-09-20T01:43:21.549213+0000 2 0 2024-09-18T14:07:20.388 INFO:tasks.workunit.client.0.smithi047.stdout:1.d 3 0 0 0 0 1449 0 0 15 66 15 active+clean 2024-09-18T14:07:09.554414+0000 34'81 41:132 [2,0,1] 2 [2,0,1] 2 34'81 2024-09-18T14:07:09.554363+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T22:18:26.253697+0000 3 0 2024-09-18T14:07:20.388 INFO:tasks.workunit.client.0.smithi047.stdout:1.e 5 0 0 0 0 1691 0 0 18 66 18 active+clean 2024-09-18T14:07:15.509879+0000 40'84 44:131 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:20.388 INFO:tasks.workunit.client.0.smithi047.stdout:1.f 3 0 0 0 0 1208 0 0 14 66 14 active+clean 2024-09-18T14:07:09.761357+0000 32'80 43:120 [1,2,0] 1 [1,2,0] 1 32'80 2024-09-18T14:07:00.547567+0000 32'80 2024-09-18T14:07:00.547567+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:20.388 INFO:tasks.workunit.client.0.smithi047.stdout:1.10 3 0 0 0 0 1449 0 0 15 66 15 active+clean 2024-09-18T14:07:07.204835+0000 34'81 43:125 [1,0,2] 1 [1,0,2] 1 34'81 2024-09-18T14:07:07.204718+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T19:00:25.640613+0000 3 0 2024-09-18T14:07:20.388 INFO:tasks.workunit.client.0.smithi047.stdout:1.11 1 0 0 0 0 724 0 0 15 66 15 active+clean 2024-09-18T14:06:59.588575+0000 40'81 41:110 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:20.388 INFO:tasks.workunit.client.0.smithi047.stdout:1.12 1 0 0 0 0 483 0 0 14 66 14 active+clean 2024-09-18T14:07:08.487386+0000 32'80 44:120 [0,1,2] 0 [0,1,2] 0 32'80 2024-09-18T14:07:08.487332+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T19:19:08.320223+0000 1 0 2024-09-18T14:07:20.388 INFO:tasks.workunit.client.0.smithi047.stdout:1.13 3 0 0 0 0 1690 0 0 15 66 15 active+clean 2024-09-18T14:07:09.761314+0000 34'81 43:121 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:20.388 INFO:tasks.workunit.client.0.smithi047.stdout:1.14 3 0 0 0 0 1690 0 0 15 66 15 active+clean 2024-09-18T14:06:53.871541+0000 34'81 43:115 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:20.388 INFO:tasks.workunit.client.0.smithi047.stdout:1.15 2 0 0 0 0 1207 0 0 14 66 14 active+clean 2024-09-18T14:06:59.727249+0000 32'80 41:109 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:20.388 INFO:tasks.workunit.client.0.smithi047.stdout:1.16 2 0 0 0 0 1207 0 0 14 66 14 active+clean 2024-09-18T14:07:15.417490+0000 32'80 44:118 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:20.388 INFO:tasks.workunit.client.0.smithi047.stdout:1.17 2 0 0 0 0 1207 0 0 14 66 14 active+clean 2024-09-18T14:07:09.762065+0000 32'80 43:120 [1,2,0] 1 [1,2,0] 1 32'80 2024-09-18T14:07:05.418931+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:20.389 INFO:tasks.workunit.client.0.smithi047.stdout:1.18 3 0 0 0 0 1690 0 0 15 66 15 active+clean 2024-09-18T14:06:53.871248+0000 34'81 43:115 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:20.389 INFO:tasks.workunit.client.0.smithi047.stdout:1.19 2 0 0 0 0 966 0 0 14 66 14 active+clean 2024-09-18T14:07:07.546465+0000 32'80 41:119 [2,0,1] 2 [2,0,1] 2 32'80 2024-09-18T14:07:07.546410+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T15:57:14.042002+0000 2 0 2024-09-18T14:07:20.389 INFO:tasks.workunit.client.0.smithi047.stdout:1.1a 3 0 0 0 0 1690 0 0 15 66 15 active+clean 2024-09-18T14:07:07.767321+0000 34'81 44:123 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:20.389 INFO:tasks.workunit.client.0.smithi047.stdout:1.1b 4 0 0 0 0 1361 0 0 17 66 17 active+clean 2024-09-18T14:07:09.762009+0000 40'83 43:123 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:20.389 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:07:20.389 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:07:20.389 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:07:20.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd 2024-09-18T14:07:20.397 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:07:20.397 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:07:20.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:07:20.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:07:20.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:07:20.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:07:20.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-09-18T14:07:20.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.0 2024-09-18T14:07:21.142 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already down. 2024-09-18T14:07:21.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-09-18T14:07:21.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.1 2024-09-18T14:07:21.720 INFO:tasks.workunit.client.0.smithi047.stderr:osd.1 is already down. 2024-09-18T14:07:21.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-09-18T14:07:21.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.2 2024-09-18T14:07:22.277 INFO:tasks.workunit.client.0.smithi047.stderr:osd.2 is already down. 2024-09-18T14:07:22.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:733: TEST_divergent_3: for i in $non_divergent 2024-09-18T14:07:22.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: ceph osd out osd.0 2024-09-18T14:07:22.830 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already out. 2024-09-18T14:07:22.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:733: TEST_divergent_3: for i in $non_divergent 2024-09-18T14:07:22.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: ceph osd out osd.2 2024-09-18T14:07:23.582 INFO:tasks.workunit.client.0.smithi047.stderr:osd.2 is already out. 2024-09-18T14:07:23.592 INFO:tasks.workunit.client.0.smithi047.stdout:bringing up non_divergent 0 2 2024-09-18T14:07:23.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:739: TEST_divergent_3: echo 'bringing up non_divergent 0 2' 2024-09-18T14:07:23.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:740: TEST_divergent_3: ceph pg dump pgs 2024-09-18T14:07:23.992 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:07:23.992 INFO:tasks.workunit.client.0.smithi047.stdout:1.1f 2 0 0 0 0 1207 0 0 14 66 14 stale+active+clean 2024-09-18T14:07:09.761988+0000 32'80 43:120 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:23.992 INFO:tasks.workunit.client.0.smithi047.stdout:1.1e 3 0 0 0 0 1690 0 0 15 66 15 stale+active+clean 2024-09-18T14:07:15.417434+0000 34'81 44:119 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:23.992 INFO:tasks.workunit.client.0.smithi047.stdout:1.1d 4 0 0 0 0 1932 0 0 14 66 14 stale+active+clean 2024-09-18T14:07:18.379784+0000 32'80 46:124 [2,0,1] 2 [2,0,1] 2 32'80 2024-09-18T14:07:18.379714+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-20T01:11:09.411303+0000 4 0 2024-09-18T14:07:23.992 INFO:tasks.workunit.client.0.smithi047.stdout:1.b 4 0 0 0 0 1691 0 0 15 66 15 stale+active+clean 2024-09-18T14:07:09.761777+0000 34'81 43:121 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:23.992 INFO:tasks.workunit.client.0.smithi047.stdout:1.a 3 0 0 0 0 1208 0 0 14 66 14 stale+active+clean 2024-09-18T14:07:07.767880+0000 32'80 44:110 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:23.993 INFO:tasks.workunit.client.0.smithi047.stdout:1.9 5 0 0 0 0 2086 0 0 17 66 17 stale+active+clean 2024-09-18T14:07:11.436936+0000 40'83 46:139 [2,0,1] 2 [2,0,1] 2 40'83 2024-09-18T14:07:11.436862+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-20T00:39:30.586095+0000 5 0 2024-09-18T14:07:23.993 INFO:tasks.workunit.client.0.smithi047.stdout:1.8 3 0 0 0 0 1449 0 0 16 66 16 stale+active+clean 2024-09-18T14:07:15.131335+0000 34'82 43:126 [1,0,2] 1 [1,0,2] 1 34'82 2024-09-18T14:07:15.131268+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T17:23:40.244015+0000 3 0 2024-09-18T14:07:23.993 INFO:tasks.workunit.client.0.smithi047.stdout:1.7 3 0 0 0 0 1208 0 0 14 66 14 stale+active+clean 2024-09-18T14:07:09.761807+0000 32'80 43:120 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:23.993 INFO:tasks.workunit.client.0.smithi047.stdout:1.6 4 0 0 0 0 1932 0 0 14 66 14 stale+active+clean 2024-09-18T14:07:15.509884+0000 32'80 44:128 [1,0,2] 1 [1,0,2] 1 32'80 2024-09-18T14:06:58.202394+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T20:55:47.501665+0000 4 0 2024-09-18T14:07:23.993 INFO:tasks.workunit.client.0.smithi047.stdout:1.5 6 0 0 0 0 2898 0 0 15 66 15 stale+active+clean 2024-09-18T14:07:06.420594+0000 34'81 46:137 [2,0,1] 2 [2,0,1] 2 34'81 2024-09-18T14:07:06.420533+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T21:49:34.160706+0000 6 0 2024-09-18T14:07:23.993 INFO:tasks.workunit.client.0.smithi047.stdout:1.4 2 0 0 0 0 966 0 0 14 66 14 stale+active+clean 2024-09-18T14:07:08.170532+0000 32'80 43:124 [1,0,2] 1 [1,0,2] 1 32'80 2024-09-18T14:07:08.170467+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T16:10:40.115462+0000 2 0 2024-09-18T14:07:23.993 INFO:tasks.workunit.client.0.smithi047.stdout:1.3 4 0 0 0 0 1691 0 0 15 66 15 stale+active+clean 2024-09-18T14:07:09.761872+0000 34'81 43:121 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:23.993 INFO:tasks.workunit.client.0.smithi047.stdout:1.2 1 0 0 0 0 483 0 0 14 66 14 stale+active+clean 2024-09-18T14:07:18.506369+0000 32'80 45:120 [0,1,2] 0 [0,1,2] 0 32'80 2024-09-18T14:07:18.506303+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T23:05:42.596177+0000 1 0 2024-09-18T14:07:23.993 INFO:tasks.workunit.client.0.smithi047.stdout:1.c 3 0 0 0 0 1449 0 0 15 66 15 stale+active+clean 2024-09-18T14:07:12.133223+0000 34'81 43:125 [1,0,2] 1 [1,0,2] 1 34'81 2024-09-18T14:07:12.133156+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T19:41:59.371626+0000 3 0 2024-09-18T14:07:23.993 INFO:tasks.workunit.client.0.smithi047.stdout:1.1 2 0 0 0 0 966 0 0 15 66 15 stale+active+clean 2024-09-18T14:07:13.379547+0000 34'81 46:137 [2,0,1] 2 [2,0,1] 2 34'81 2024-09-18T14:07:13.379437+0000 34'81 2024-09-18T14:06:56.389936+0000 0 1 periodic scrub scheduled @ 2024-09-20T00:39:31.640524+0000 2 0 2024-09-18T14:07:23.993 INFO:tasks.workunit.client.0.smithi047.stdout:1.1c 2 0 0 0 0 966 0 0 14 66 14 stale+active+clean 2024-09-18T14:07:16.092389+0000 32'80 44:124 [1,0,2] 1 [1,0,2] 1 32'80 2024-09-18T14:07:16.092318+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T15:53:36.670105+0000 2 0 2024-09-18T14:07:23.993 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 3 0 0 0 0 1449 0 0 15 66 15 stale+active+clean 2024-09-18T14:06:54.303752+0000 34'81 43:126 [1,0,2] 1 [1,0,2] 1 32'80 2024-09-18T14:06:54.303691+0000 32'80 2024-09-18T14:06:54.303691+0000 0 1 periodic scrub scheduled @ 2024-09-20T01:43:21.549213+0000 2 0 2024-09-18T14:07:23.994 INFO:tasks.workunit.client.0.smithi047.stdout:1.d 3 0 0 0 0 1449 0 0 15 66 15 stale+active+clean 2024-09-18T14:07:09.554414+0000 34'81 46:137 [2,0,1] 2 [2,0,1] 2 34'81 2024-09-18T14:07:09.554363+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T22:18:26.253697+0000 3 0 2024-09-18T14:07:23.994 INFO:tasks.workunit.client.0.smithi047.stdout:1.e 5 0 0 0 0 1691 0 0 18 66 18 stale+active+clean 2024-09-18T14:07:15.509879+0000 40'84 44:131 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:23.994 INFO:tasks.workunit.client.0.smithi047.stdout:1.f 3 0 0 0 0 1208 0 0 14 66 14 stale+active+clean 2024-09-18T14:07:09.761357+0000 32'80 43:120 [1,2,0] 1 [1,2,0] 1 32'80 2024-09-18T14:07:00.547567+0000 32'80 2024-09-18T14:07:00.547567+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:23.994 INFO:tasks.workunit.client.0.smithi047.stdout:1.10 3 0 0 0 0 1449 0 0 15 66 15 stale+active+clean 2024-09-18T14:07:07.204835+0000 34'81 43:125 [1,0,2] 1 [1,0,2] 1 34'81 2024-09-18T14:07:07.204718+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T19:00:25.640613+0000 3 0 2024-09-18T14:07:23.994 INFO:tasks.workunit.client.0.smithi047.stdout:1.11 2 0 0 0 0 966 0 0 15 66 15 stale+active+clean 2024-09-18T14:07:15.411331+0000 40'81 46:125 [2,0,1] 2 [2,0,1] 2 40'81 2024-09-18T14:07:15.411220+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-20T01:58:26.321473+0000 2 0 2024-09-18T14:07:23.994 INFO:tasks.workunit.client.0.smithi047.stdout:1.12 1 0 0 0 0 483 0 0 14 66 14 stale+active+clean 2024-09-18T14:07:08.487386+0000 32'80 44:120 [0,1,2] 0 [0,1,2] 0 32'80 2024-09-18T14:07:08.487332+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T19:19:08.320223+0000 1 0 2024-09-18T14:07:23.994 INFO:tasks.workunit.client.0.smithi047.stdout:1.13 3 0 0 0 0 1690 0 0 15 66 15 stale+active+clean 2024-09-18T14:07:09.761314+0000 34'81 43:121 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:23.994 INFO:tasks.workunit.client.0.smithi047.stdout:1.14 3 0 0 0 0 1690 0 0 15 66 15 stale+active+clean 2024-09-18T14:06:53.871541+0000 34'81 43:115 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:23.994 INFO:tasks.workunit.client.0.smithi047.stdout:1.15 2 0 0 0 0 1207 0 0 14 66 14 stale+active+clean 2024-09-18T14:06:59.727249+0000 32'80 46:114 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:23.994 INFO:tasks.workunit.client.0.smithi047.stdout:1.16 2 0 0 0 0 1207 0 0 14 66 14 stale+active+clean 2024-09-18T14:07:15.417490+0000 32'80 44:118 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:23.994 INFO:tasks.workunit.client.0.smithi047.stdout:1.17 2 0 0 0 0 1207 0 0 14 66 14 stale+active+clean 2024-09-18T14:07:09.762065+0000 32'80 43:120 [1,2,0] 1 [1,2,0] 1 32'80 2024-09-18T14:07:05.418931+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:23.994 INFO:tasks.workunit.client.0.smithi047.stdout:1.18 3 0 0 0 0 1690 0 0 15 66 15 stale+active+clean 2024-09-18T14:06:53.871248+0000 34'81 43:115 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:23.995 INFO:tasks.workunit.client.0.smithi047.stdout:1.19 2 0 0 0 0 966 0 0 14 66 14 stale+active+clean 2024-09-18T14:07:07.546465+0000 32'80 46:124 [2,0,1] 2 [2,0,1] 2 32'80 2024-09-18T14:07:07.546410+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T15:57:14.042002+0000 2 0 2024-09-18T14:07:23.995 INFO:tasks.workunit.client.0.smithi047.stdout:1.1a 3 0 0 0 0 1690 0 0 15 66 15 stale+active+clean 2024-09-18T14:07:07.767321+0000 34'81 44:123 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:23.995 INFO:tasks.workunit.client.0.smithi047.stdout:1.1b 4 0 0 0 0 1361 0 0 17 66 17 stale+active+clean 2024-09-18T14:07:09.762009+0000 40'83 43:123 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:07:23.995 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:07:23.995 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:07:23.995 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:07:24.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: for i in $non_divergent 2024-09-18T14:07:24.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: activate_osd td/divergent-priors 0 2024-09-18T14:07:24.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-18T14:07:24.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:07:24.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T14:07:24.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:07:24.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-09-18T14:07:24.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T14:07:24.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:07:24.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:07:24.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:07:24.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-18T14:07:24.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-18T14:07:24.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:07:24.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:07:24.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-18T14:07:24.006 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:07:24.006 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:07:24.006 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:07:24.006 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:07:24.006 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:07:24.006 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:07:24.006 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:07:24.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:07:24.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:07:24.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-18T14:07:24.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-18T14:07:24.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:07:24.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:07:24.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:07:24.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:07:24.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:07:24.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:07:24.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-09-18T14:07:24.007 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T14:07:24.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T14:07:24.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:07:24.008 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-09-18T14:07:24.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T14:07:24.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:07:24.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:07:24.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:07:24.035 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:07:24.034+0000 7f3f26a1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:07:24.037 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:07:24.037+0000 7f3f26a1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:07:24.038 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:07:24.038+0000 7f3f26a1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:07:24.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T14:07:24.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:07:24.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:07:24.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:07:24.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:07:24.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:07:24.377 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:07:24.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:07:24.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:07:24.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:07:24.553 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:07:24.553+0000 7f3f26a1d440 -1 Falling back to public interface 2024-09-18T14:07:24.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:07:25.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:07:25.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:07:25.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:07:25.803 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:07:25.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:07:25.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:07:26.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:07:27.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:07:27.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:07:27.192 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:07:27.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:07:27.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:07:27.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:07:27.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:07:27.871 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:07:27.871+0000 7f3f26a1d440 -1 osd.0 40 log_to_monitors true 2024-09-18T14:07:28.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:07:28.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:07:28.578 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:07:28.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:07:28.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:07:28.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:07:29.068 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up out weight 0 up_from 57 up_thru 37 down_at 49 last_clean_interval [5,40) [v2:127.0.0.1:6802/68576,v1:127.0.0.1:6803/68576] [v2:127.0.0.1:6804/68576,v1:127.0.0.1:6805/68576] exists,up 2d333a34-fa14-4e6f-aca9-f14a57e36c76 2024-09-18T14:07:29.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:07:29.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:07:29.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:07:29.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: for i in $non_divergent 2024-09-18T14:07:29.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: activate_osd td/divergent-priors 2 2024-09-18T14:07:29.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-18T14:07:29.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:07:29.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T14:07:29.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:07:29.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-09-18T14:07:29.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T14:07:29.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:07:29.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:07:29.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:07:29.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-18T14:07:29.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-18T14:07:29.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:07:29.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:07:29.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-18T14:07:29.072 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:07:29.072 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:07:29.072 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:07:29.073 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:07:29.073 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:07:29.073 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:07:29.073 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:07:29.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:07:29.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:07:29.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-18T14:07:29.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-18T14:07:29.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:07:29.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:07:29.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:07:29.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:07:29.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:07:29.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:07:29.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-09-18T14:07:29.074 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T14:07:29.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T14:07:29.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:07:29.075 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-09-18T14:07:29.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T14:07:29.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:07:29.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:07:29.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:07:29.101 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:07:29.100+0000 7fe8f5b21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:07:29.104 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:07:29.104+0000 7fe8f5b21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:07:29.105 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:07:29.105+0000 7fe8f5b21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:07:29.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T14:07:29.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:07:29.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T14:07:29.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:07:29.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:07:29.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:07:29.462 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:07:29.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:07:29.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:07:29.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:07:29.620 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:07:29.620+0000 7fe8f5b21440 -1 Falling back to public interface 2024-09-18T14:07:29.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:07:30.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:07:30.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:07:30.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:07:30.870 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:07:30.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:07:30.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:07:31.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:07:32.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:07:32.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:07:32.340 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:07:32.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:07:32.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:07:32.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:07:32.502 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:07:32.502+0000 7fe8f5b21440 -1 osd.2 40 log_to_monitors true 2024-09-18T14:07:32.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:07:33.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:07:33.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:07:33.854 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:07:33.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:07:33.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:07:33.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:07:34.272 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up out weight 0 up_from 61 up_thru 35 down_at 51 last_clean_interval [15,40) [v2:127.0.0.1:6810/69154,v1:127.0.0.1:6811/69154] [v2:127.0.0.1:6812/69154,v1:127.0.0.1:6813/69154] exists,up 20ba0a43-7c90-449e-bd64-6203deaebae3 2024-09-18T14:07:34.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:07:34.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:07:34.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:07:34.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:745: TEST_divergent_3: for i in $non_divergent 2024-09-18T14:07:34.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph osd in osd.0 2024-09-18T14:07:34.991 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already in. 2024-09-18T14:07:35.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:745: TEST_divergent_3: for i in $non_divergent 2024-09-18T14:07:35.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph osd in osd.2 2024-09-18T14:07:36.140 INFO:tasks.workunit.client.0.smithi047.stderr:osd.2 is already in. 2024-09-18T14:07:36.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-18T14:07:36.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: wait_for_clean 2024-09-18T14:07:36.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:07:36.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:07:36.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:07:36.152 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-18T14:07:36.152 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:07:36.152 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:07:36.152 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:07:36.152 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:07:36.152 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:07:36.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-18T14:07:36.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:07:36.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:07:36.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:07:36.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:07:36.208 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:07:36.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:07:36.653 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T14:07:36.653 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T14:07:36.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:07:36.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:07:36.653 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:07:36.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=244813135876 2024-09-18T14:07:36.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 244813135876 2024-09-18T14:07:36.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-244813135876' 2024-09-18T14:07:36.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:07:36.853 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:07:36.938 INFO:tasks.workunit.client.0.smithi047.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-18T14:07:36.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq= 2024-09-18T14:07:36.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z '' 2024-09-18T14:07:36.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: flush_pg_stats: continue 2024-09-18T14:07:36.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:07:36.940 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:07:37.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=261993005059 2024-09-18T14:07:37.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 261993005059 2024-09-18T14:07:37.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-244813135876 2-261993005059' 2024-09-18T14:07:37.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:07:37.173 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-244813135876 2024-09-18T14:07:37.174 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:07:37.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:07:37.175 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-244813135876 2024-09-18T14:07:37.175 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:07:37.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=244813135876 2024-09-18T14:07:37.176 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 244813135876 2024-09-18T14:07:37.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 244813135876' 2024-09-18T14:07:37.176 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:07:37.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 244813135874 -lt 244813135876 2024-09-18T14:07:37.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:07:38.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:07:38.554 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:07:38.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 244813135876 -lt 244813135876 2024-09-18T14:07:38.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:07:38.956 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-261993005059 2024-09-18T14:07:38.956 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:07:38.956 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:07:38.957 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-261993005059 2024-09-18T14:07:38.957 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:07:38.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=261993005059 2024-09-18T14:07:38.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 261993005059' 2024-09-18T14:07:38.959 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 261993005059 2024-09-18T14:07:38.959 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:07:39.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 261993005059 -lt 261993005059 2024-09-18T14:07:39.327 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:07:39.327 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:07:39.328 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:07:39.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-09-18T14:07:39.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:07:39.833 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:07:39.833 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:07:39.833 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:07:39.833 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:07:39.833 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:07:39.834 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:07:40.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:07:40.235 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:07:40.235 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:07:40.235 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:07:40.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-18T14:07:40.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-18T14:07:40.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T14:07:40.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-18T14:07:40.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:07:40.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:07:40.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:07:40.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:07:40.851 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:07:40.851 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:07:40.851 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:07:40.851 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:07:40.851 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:07:40.851 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:07:41.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:07:41.219 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:07:41.219 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:07:41.219 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:07:41.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-18T14:07:41.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:07:41.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:07:41.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:07:41.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:07:41.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:07:41.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:07:41.728 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:07:41.728 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:07:42.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:07:42.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:07:42.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-09-18T14:07:42.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:07:42.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-18T14:07:42.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:07:42.469 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:07:42.469 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:07:42.469 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:07:42.469 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:07:42.470 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:07:42.470 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:07:42.470 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:07:42.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:07:42.845 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:07:42.845 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:07:42.846 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:07:43.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-18T14:07:43.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:07:43.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:07:43.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:07:43.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:07:43.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:07:43.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:07:43.345 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:07:43.345 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:07:43.875 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:07:43.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:07:43.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-09-18T14:07:43.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:07:43.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-18T14:07:44.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:07:44.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:07:44.278 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:07:44.278 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:07:44.278 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:07:44.278 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:07:44.278 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:07:44.278 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:07:44.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:07:44.650 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:07:44.650 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:07:44.651 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:07:45.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-18T14:07:45.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:07:45.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:07:45.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:07:45.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:07:45.157 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:07:45.157 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:07:45.157 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:07:45.157 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:07:45.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:07:45.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:07:45.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-09-18T14:07:45.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:07:45.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-09-18T14:07:46.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:07:46.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:07:46.466 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:07:46.466 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:07:46.466 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:07:46.466 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:07:46.466 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:07:46.466 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:07:46.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:07:46.845 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:07:46.845 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:07:46.845 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:07:47.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-18T14:07:47.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:07:47.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:07:47.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:07:47.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:07:47.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:07:47.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:07:47.347 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:07:47.347 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:07:47.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:07:47.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:07:47.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-09-18T14:07:47.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:07:47.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-09-18T14:07:49.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:07:49.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:07:49.489 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:07:49.489 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:07:49.489 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:07:49.489 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:07:49.489 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:07:49.490 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:07:49.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:07:49.867 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:07:49.867 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:07:49.867 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:07:50.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-18T14:07:50.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:07:50.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:07:50.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:07:50.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:07:50.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:07:50.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:07:50.388 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:07:50.388 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:07:50.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:07:50.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:07:50.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-09-18T14:07:50.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:07:50.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-09-18T14:07:54.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:07:54.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:07:54.124 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:07:54.124 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:07:54.125 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:07:54.125 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:07:54.125 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:07:54.125 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:07:54.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:07:54.504 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:07:54.504 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:07:54.504 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:07:55.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-18T14:07:55.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:07:55.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:07:55.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:07:55.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:07:55.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:07:55.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:07:55.015 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:07:55.015 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:07:55.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:07:55.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:07:55.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-09-18T14:07:55.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:07:55.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-09-18T14:08:01.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:08:01.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:08:01.945 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:08:01.945 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:08:01.945 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:08:01.945 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:08:01.945 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:08:01.945 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:08:02.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:08:02.327 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:08:02.328 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:08:02.328 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:08:02.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-18T14:08:02.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:08:02.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:08:02.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:08:02.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:08:02.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:08:02.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:08:02.840 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:08:02.840 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:08:03.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:08:03.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:08:03.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-09-18T14:08:03.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:08:03.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-09-18T14:08:10.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:08:10.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:08:10.678 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:08:10.678 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:08:10.678 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:08:10.678 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:08:10.678 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:08:10.678 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:08:11.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:08:11.067 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:08:11.068 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:08:11.068 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:08:11.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-18T14:08:11.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:08:11.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:08:11.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:08:11.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:08:11.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:08:11.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:08:11.572 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:08:11.572 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:08:12.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:08:12.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:08:12.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-09-18T14:08:12.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-18T14:08:12.657 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T14:08:12.657 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_fingerprint": "295f4427-4a7c-4940-a333-e06d3d18f172", 2024-09-18T14:08:12.658 INFO:tasks.workunit.client.0.smithi047.stdout: "version": "17.2.7-1645-g7061ea6c", 2024-09-18T14:08:12.658 INFO:tasks.workunit.client.0.smithi047.stdout: "commit": "7061ea6ca50908d104e9f3979dc615e118d9e153", 2024-09-18T14:08:12.658 INFO:tasks.workunit.client.0.smithi047.stdout: "timestamp": "2024-09-18T14:08:12.656887+0000", 2024-09-18T14:08:12.658 INFO:tasks.workunit.client.0.smithi047.stdout: "tag": "", 2024-09-18T14:08:12.658 INFO:tasks.workunit.client.0.smithi047.stdout: "health": { 2024-09-18T14:08:12.658 INFO:tasks.workunit.client.0.smithi047.stdout: "status": "HEALTH_WARN", 2024-09-18T14:08:12.658 INFO:tasks.workunit.client.0.smithi047.stdout: "checks": { 2024-09-18T14:08:12.658 INFO:tasks.workunit.client.0.smithi047.stdout: "OSDMAP_FLAGS": { 2024-09-18T14:08:12.658 INFO:tasks.workunit.client.0.smithi047.stdout: "severity": "HEALTH_WARN", 2024-09-18T14:08:12.658 INFO:tasks.workunit.client.0.smithi047.stdout: "summary": { 2024-09-18T14:08:12.658 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-18T14:08:12.658 INFO:tasks.workunit.client.0.smithi047.stdout: "count": 17 2024-09-18T14:08:12.658 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.659 INFO:tasks.workunit.client.0.smithi047.stdout: "detail": [], 2024-09-18T14:08:12.659 INFO:tasks.workunit.client.0.smithi047.stdout: "muted": false 2024-09-18T14:08:12.659 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.659 INFO:tasks.workunit.client.0.smithi047.stdout: "OSD_DOWN": { 2024-09-18T14:08:12.659 INFO:tasks.workunit.client.0.smithi047.stdout: "severity": "HEALTH_WARN", 2024-09-18T14:08:12.659 INFO:tasks.workunit.client.0.smithi047.stdout: "summary": { 2024-09-18T14:08:12.659 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "1 osds down", 2024-09-18T14:08:12.659 INFO:tasks.workunit.client.0.smithi047.stdout: "count": 1 2024-09-18T14:08:12.659 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.659 INFO:tasks.workunit.client.0.smithi047.stdout: "detail": [ 2024-09-18T14:08:12.659 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.659 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "osd.1 (root=default,host=smithi047) is down" 2024-09-18T14:08:12.659 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.659 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:12.660 INFO:tasks.workunit.client.0.smithi047.stdout: "muted": false 2024-09-18T14:08:12.660 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.660 INFO:tasks.workunit.client.0.smithi047.stdout: "PG_DEGRADED": { 2024-09-18T14:08:12.660 INFO:tasks.workunit.client.0.smithi047.stdout: "severity": "HEALTH_WARN", 2024-09-18T14:08:12.660 INFO:tasks.workunit.client.0.smithi047.stdout: "summary": { 2024-09-18T14:08:12.660 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded", 2024-09-18T14:08:12.660 INFO:tasks.workunit.client.0.smithi047.stdout: "count": 32 2024-09-18T14:08:12.660 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.660 INFO:tasks.workunit.client.0.smithi047.stdout: "detail": [ 2024-09-18T14:08:12.660 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.660 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-09-18T14:08:12.660 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.660 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.661 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.1 is active+undersized+degraded, acting [2,0]" 2024-09-18T14:08:12.661 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.661 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.661 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0,2]" 2024-09-18T14:08:12.661 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.661 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.661 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [2,0]" 2024-09-18T14:08:12.661 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.661 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.661 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.4 is active+undersized+degraded, acting [0,2]" 2024-09-18T14:08:12.661 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.661 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.661 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.5 is active+undersized+degraded, acting [2,0]" 2024-09-18T14:08:12.661 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.662 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.662 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.6 is active+undersized+degraded, acting [0,2]" 2024-09-18T14:08:12.662 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.662 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.662 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.7 is active+undersized+degraded, acting [2,0]" 2024-09-18T14:08:12.662 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.662 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.662 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.8 is active+undersized+degraded, acting [2,0]" 2024-09-18T14:08:12.662 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.662 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.662 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.9 is active+undersized+degraded, acting [2,0]" 2024-09-18T14:08:12.662 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.662 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.662 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.a is active+undersized+degraded, acting [2,0]" 2024-09-18T14:08:12.663 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.663 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.663 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.b is active+undersized+degraded, acting [0,2]" 2024-09-18T14:08:12.663 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.663 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.663 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.c is active+undersized+degraded, acting [0,2]" 2024-09-18T14:08:12.663 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.663 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.663 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.d is active+undersized+degraded, acting [2,0]" 2024-09-18T14:08:12.663 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.663 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.663 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.e is active+undersized+degraded, acting [2,0]" 2024-09-18T14:08:12.663 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.663 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.664 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.f is active+undersized+degraded, acting [2,0]" 2024-09-18T14:08:12.664 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.664 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.664 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.10 is active+undersized+degraded, acting [0,2]" 2024-09-18T14:08:12.664 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.664 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.664 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.11 is active+undersized+degraded, acting [0,2]" 2024-09-18T14:08:12.664 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.664 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.664 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.12 is active+undersized+degraded, acting [0,2]" 2024-09-18T14:08:12.664 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.664 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.664 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.13 is active+undersized+degraded, acting [2,0]" 2024-09-18T14:08:12.665 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.665 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.665 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.14 is active+undersized+degraded, acting [0,2]" 2024-09-18T14:08:12.665 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.665 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.665 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.15 is active+undersized+degraded, acting [2,0]" 2024-09-18T14:08:12.665 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.665 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.665 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.16 is active+undersized+degraded, acting [0,2]" 2024-09-18T14:08:12.665 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.665 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.665 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.17 is active+undersized+degraded, acting [2,0]" 2024-09-18T14:08:12.665 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.665 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.666 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.18 is active+undersized+degraded, acting [2,0]" 2024-09-18T14:08:12.666 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.666 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.666 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.19 is active+undersized+degraded, acting [2,0]" 2024-09-18T14:08:12.666 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.666 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.666 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.1a is active+undersized+degraded, acting [2,0]" 2024-09-18T14:08:12.666 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.666 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.666 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.1b is active+undersized+degraded, acting [0,2]" 2024-09-18T14:08:12.666 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.666 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.666 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.1c is active+undersized+degraded, acting [0,2]" 2024-09-18T14:08:12.666 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.667 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.667 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.1d is active+undersized+degraded, acting [2,0]" 2024-09-18T14:08:12.667 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.667 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.667 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.1e is active+undersized+degraded, acting [2,0]" 2024-09-18T14:08:12.667 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.667 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.667 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.1f is active+undersized+degraded, acting [2,0]" 2024-09-18T14:08:12.667 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.667 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:12.667 INFO:tasks.workunit.client.0.smithi047.stdout: "muted": false 2024-09-18T14:08:12.667 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.667 INFO:tasks.workunit.client.0.smithi047.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-18T14:08:12.667 INFO:tasks.workunit.client.0.smithi047.stdout: "severity": "HEALTH_WARN", 2024-09-18T14:08:12.668 INFO:tasks.workunit.client.0.smithi047.stdout: "summary": { 2024-09-18T14:08:12.668 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-18T14:08:12.668 INFO:tasks.workunit.client.0.smithi047.stdout: "count": 1 2024-09-18T14:08:12.668 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.668 INFO:tasks.workunit.client.0.smithi047.stdout: "detail": [ 2024-09-18T14:08:12.668 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.668 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "application not enabled on pool 'test'" 2024-09-18T14:08:12.668 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.668 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.668 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-18T14:08:12.668 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.668 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:12.668 INFO:tasks.workunit.client.0.smithi047.stdout: "muted": false 2024-09-18T14:08:12.668 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.668 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.669 INFO:tasks.workunit.client.0.smithi047.stdout: "mutes": [] 2024-09-18T14:08:12.669 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.669 INFO:tasks.workunit.client.0.smithi047.stdout: "monmap_first_committed": 1, 2024-09-18T14:08:12.669 INFO:tasks.workunit.client.0.smithi047.stdout: "monmap_last_committed": 1, 2024-09-18T14:08:12.669 INFO:tasks.workunit.client.0.smithi047.stdout: "monmap": { 2024-09-18T14:08:12.669 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 1, 2024-09-18T14:08:12.669 INFO:tasks.workunit.client.0.smithi047.stdout: "fsid": "d7330bd7-8062-424a-bbd8-8c587903c456", 2024-09-18T14:08:12.669 INFO:tasks.workunit.client.0.smithi047.stdout: "modified": "2024-09-18T14:04:45.277080Z", 2024-09-18T14:08:12.669 INFO:tasks.workunit.client.0.smithi047.stdout: "created": "2024-09-18T14:04:45.277080Z", 2024-09-18T14:08:12.669 INFO:tasks.workunit.client.0.smithi047.stdout: "min_mon_release": 17, 2024-09-18T14:08:12.669 INFO:tasks.workunit.client.0.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-18T14:08:12.669 INFO:tasks.workunit.client.0.smithi047.stdout: "election_strategy": 1, 2024-09-18T14:08:12.669 INFO:tasks.workunit.client.0.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-18T14:08:12.670 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_mode": false, 2024-09-18T14:08:12.670 INFO:tasks.workunit.client.0.smithi047.stdout: "tiebreaker_mon": "", 2024-09-18T14:08:12.670 INFO:tasks.workunit.client.0.smithi047.stdout: "removed_ranks: ": "", 2024-09-18T14:08:12.670 INFO:tasks.workunit.client.0.smithi047.stdout: "features": { 2024-09-18T14:08:12.670 INFO:tasks.workunit.client.0.smithi047.stdout: "persistent": [ 2024-09-18T14:08:12.670 INFO:tasks.workunit.client.0.smithi047.stdout: "kraken", 2024-09-18T14:08:12.670 INFO:tasks.workunit.client.0.smithi047.stdout: "luminous", 2024-09-18T14:08:12.670 INFO:tasks.workunit.client.0.smithi047.stdout: "mimic", 2024-09-18T14:08:12.670 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap-prune", 2024-09-18T14:08:12.670 INFO:tasks.workunit.client.0.smithi047.stdout: "nautilus", 2024-09-18T14:08:12.670 INFO:tasks.workunit.client.0.smithi047.stdout: "octopus", 2024-09-18T14:08:12.670 INFO:tasks.workunit.client.0.smithi047.stdout: "pacific", 2024-09-18T14:08:12.670 INFO:tasks.workunit.client.0.smithi047.stdout: "elector-pinging", 2024-09-18T14:08:12.670 INFO:tasks.workunit.client.0.smithi047.stdout: "quincy" 2024-09-18T14:08:12.671 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:12.671 INFO:tasks.workunit.client.0.smithi047.stdout: "optional": [] 2024-09-18T14:08:12.671 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.671 INFO:tasks.workunit.client.0.smithi047.stdout: "mons": [ 2024-09-18T14:08:12.671 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.671 INFO:tasks.workunit.client.0.smithi047.stdout: "rank": 0, 2024-09-18T14:08:12.671 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "a", 2024-09-18T14:08:12.671 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addrs": { 2024-09-18T14:08:12.671 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:08:12.671 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.671 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:08:12.671 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:7115", 2024-09-18T14:08:12.671 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 0 2024-09-18T14:08:12.672 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.672 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:12.672 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.672 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:7115/0", 2024-09-18T14:08:12.672 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-18T14:08:12.672 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 0, 2024-09-18T14:08:12.672 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 0, 2024-09-18T14:08:12.672 INFO:tasks.workunit.client.0.smithi047.stdout: "crush_location": "{}" 2024-09-18T14:08:12.672 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.672 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:12.672 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.672 INFO:tasks.workunit.client.0.smithi047.stdout: "quorum": [ 2024-09-18T14:08:12.672 INFO:tasks.workunit.client.0.smithi047.stdout: 0 2024-09-18T14:08:12.672 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:12.673 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap": { 2024-09-18T14:08:12.673 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 70, 2024-09-18T14:08:12.673 INFO:tasks.workunit.client.0.smithi047.stdout: "fsid": "d7330bd7-8062-424a-bbd8-8c587903c456", 2024-09-18T14:08:12.673 INFO:tasks.workunit.client.0.smithi047.stdout: "created": "2024-09-18T14:04:46.122705+0000", 2024-09-18T14:08:12.673 INFO:tasks.workunit.client.0.smithi047.stdout: "modified": "2024-09-18T14:07:37.245559+0000", 2024-09-18T14:08:12.673 INFO:tasks.workunit.client.0.smithi047.stdout: "last_up_change": "2024-09-18T14:07:32.707559+0000", 2024-09-18T14:08:12.673 INFO:tasks.workunit.client.0.smithi047.stdout: "last_in_change": "2024-09-18T14:07:35.610542+0000", 2024-09-18T14:08:12.673 INFO:tasks.workunit.client.0.smithi047.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-18T14:08:12.673 INFO:tasks.workunit.client.0.smithi047.stdout: "flags_num": 5800384, 2024-09-18T14:08:12.673 INFO:tasks.workunit.client.0.smithi047.stdout: "flags_set": [ 2024-09-18T14:08:12.673 INFO:tasks.workunit.client.0.smithi047.stdout: "nodown", 2024-09-18T14:08:12.673 INFO:tasks.workunit.client.0.smithi047.stdout: "noin", 2024-09-18T14:08:12.673 INFO:tasks.workunit.client.0.smithi047.stdout: "noout", 2024-09-18T14:08:12.673 INFO:tasks.workunit.client.0.smithi047.stdout: "pglog_hardlimit", 2024-09-18T14:08:12.674 INFO:tasks.workunit.client.0.smithi047.stdout: "purged_snapdirs", 2024-09-18T14:08:12.674 INFO:tasks.workunit.client.0.smithi047.stdout: "recovery_deletes", 2024-09-18T14:08:12.674 INFO:tasks.workunit.client.0.smithi047.stdout: "sortbitwise" 2024-09-18T14:08:12.674 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:12.674 INFO:tasks.workunit.client.0.smithi047.stdout: "crush_version": 9, 2024-09-18T14:08:12.674 INFO:tasks.workunit.client.0.smithi047.stdout: "full_ratio": 0.99000000953674316, 2024-09-18T14:08:12.674 INFO:tasks.workunit.client.0.smithi047.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-18T14:08:12.674 INFO:tasks.workunit.client.0.smithi047.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-18T14:08:12.674 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_snapshot": "", 2024-09-18T14:08:12.674 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_max": 1, 2024-09-18T14:08:12.674 INFO:tasks.workunit.client.0.smithi047.stdout: "max_osd": 3, 2024-09-18T14:08:12.674 INFO:tasks.workunit.client.0.smithi047.stdout: "require_min_compat_client": "luminous", 2024-09-18T14:08:12.674 INFO:tasks.workunit.client.0.smithi047.stdout: "min_compat_client": "jewel", 2024-09-18T14:08:12.674 INFO:tasks.workunit.client.0.smithi047.stdout: "require_osd_release": "quincy", 2024-09-18T14:08:12.675 INFO:tasks.workunit.client.0.smithi047.stdout: "pools": [ 2024-09-18T14:08:12.675 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.675 INFO:tasks.workunit.client.0.smithi047.stdout: "pool": 1, 2024-09-18T14:08:12.675 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_name": "test", 2024-09-18T14:08:12.675 INFO:tasks.workunit.client.0.smithi047.stdout: "create_time": "2024-09-18T14:06:27.251699+0000", 2024-09-18T14:08:12.675 INFO:tasks.workunit.client.0.smithi047.stdout: "flags": 1, 2024-09-18T14:08:12.675 INFO:tasks.workunit.client.0.smithi047.stdout: "flags_names": "hashpspool", 2024-09-18T14:08:12.675 INFO:tasks.workunit.client.0.smithi047.stdout: "type": 1, 2024-09-18T14:08:12.675 INFO:tasks.workunit.client.0.smithi047.stdout: "size": 3, 2024-09-18T14:08:12.675 INFO:tasks.workunit.client.0.smithi047.stdout: "min_size": 2, 2024-09-18T14:08:12.675 INFO:tasks.workunit.client.0.smithi047.stdout: "crush_rule": 0, 2024-09-18T14:08:12.675 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_count": 0, 2024-09-18T14:08:12.675 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_target": 0, 2024-09-18T14:08:12.675 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T14:08:12.676 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T14:08:12.676 INFO:tasks.workunit.client.0.smithi047.stdout: "object_hash": 2, 2024-09-18T14:08:12.676 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_autoscale_mode": "on", 2024-09-18T14:08:12.676 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_num": 32, 2024-09-18T14:08:12.676 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_placement_num": 18, 2024-09-18T14:08:12.676 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_placement_num_target": 32, 2024-09-18T14:08:12.676 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_num_target": 32, 2024-09-18T14:08:12.676 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_num_pending": 32, 2024-09-18T14:08:12.677 INFO:tasks.workunit.client.0.smithi047.stdout: "last_pg_merge_meta": { 2024-09-18T14:08:12.677 INFO:tasks.workunit.client.0.smithi047.stdout: "source_pgid": "0.0", 2024-09-18T14:08:12.677 INFO:tasks.workunit.client.0.smithi047.stdout: "ready_epoch": 0, 2024-09-18T14:08:12.677 INFO:tasks.workunit.client.0.smithi047.stdout: "last_epoch_started": 0, 2024-09-18T14:08:12.677 INFO:tasks.workunit.client.0.smithi047.stdout: "last_epoch_clean": 0, 2024-09-18T14:08:12.677 INFO:tasks.workunit.client.0.smithi047.stdout: "source_version": "0'0", 2024-09-18T14:08:12.677 INFO:tasks.workunit.client.0.smithi047.stdout: "target_version": "0'0" 2024-09-18T14:08:12.677 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.677 INFO:tasks.workunit.client.0.smithi047.stdout: "last_change": "69", 2024-09-18T14:08:12.677 INFO:tasks.workunit.client.0.smithi047.stdout: "last_force_op_resend": "0", 2024-09-18T14:08:12.677 INFO:tasks.workunit.client.0.smithi047.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T14:08:12.677 INFO:tasks.workunit.client.0.smithi047.stdout: "last_force_op_resend_preluminous": "33", 2024-09-18T14:08:12.677 INFO:tasks.workunit.client.0.smithi047.stdout: "auid": 0, 2024-09-18T14:08:12.677 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_mode": "selfmanaged", 2024-09-18T14:08:12.678 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_seq": 0, 2024-09-18T14:08:12.678 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_epoch": 0, 2024-09-18T14:08:12.678 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_snaps": [], 2024-09-18T14:08:12.678 INFO:tasks.workunit.client.0.smithi047.stdout: "removed_snaps": "[]", 2024-09-18T14:08:12.678 INFO:tasks.workunit.client.0.smithi047.stdout: "quota_max_bytes": 0, 2024-09-18T14:08:12.678 INFO:tasks.workunit.client.0.smithi047.stdout: "quota_max_objects": 0, 2024-09-18T14:08:12.678 INFO:tasks.workunit.client.0.smithi047.stdout: "tiers": [], 2024-09-18T14:08:12.678 INFO:tasks.workunit.client.0.smithi047.stdout: "tier_of": -1, 2024-09-18T14:08:12.678 INFO:tasks.workunit.client.0.smithi047.stdout: "read_tier": -1, 2024-09-18T14:08:12.678 INFO:tasks.workunit.client.0.smithi047.stdout: "write_tier": -1, 2024-09-18T14:08:12.678 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_mode": "none", 2024-09-18T14:08:12.678 INFO:tasks.workunit.client.0.smithi047.stdout: "target_max_bytes": 0, 2024-09-18T14:08:12.678 INFO:tasks.workunit.client.0.smithi047.stdout: "target_max_objects": 0, 2024-09-18T14:08:12.678 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T14:08:12.679 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T14:08:12.679 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T14:08:12.679 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_min_flush_age": 0, 2024-09-18T14:08:12.679 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_min_evict_age": 0, 2024-09-18T14:08:12.679 INFO:tasks.workunit.client.0.smithi047.stdout: "erasure_code_profile": "", 2024-09-18T14:08:12.679 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_params": { 2024-09-18T14:08:12.679 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "none" 2024-09-18T14:08:12.679 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.679 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_period": 0, 2024-09-18T14:08:12.679 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_count": 0, 2024-09-18T14:08:12.679 INFO:tasks.workunit.client.0.smithi047.stdout: "use_gmt_hitset": true, 2024-09-18T14:08:12.679 INFO:tasks.workunit.client.0.smithi047.stdout: "min_read_recency_for_promote": 0, 2024-09-18T14:08:12.679 INFO:tasks.workunit.client.0.smithi047.stdout: "min_write_recency_for_promote": 0, 2024-09-18T14:08:12.679 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T14:08:12.679 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_search_last_n": 0, 2024-09-18T14:08:12.680 INFO:tasks.workunit.client.0.smithi047.stdout: "grade_table": [], 2024-09-18T14:08:12.680 INFO:tasks.workunit.client.0.smithi047.stdout: "stripe_width": 0, 2024-09-18T14:08:12.680 INFO:tasks.workunit.client.0.smithi047.stdout: "expected_num_objects": 0, 2024-09-18T14:08:12.680 INFO:tasks.workunit.client.0.smithi047.stdout: "fast_read": false, 2024-09-18T14:08:12.680 INFO:tasks.workunit.client.0.smithi047.stdout: "options": {}, 2024-09-18T14:08:12.680 INFO:tasks.workunit.client.0.smithi047.stdout: "application_metadata": {} 2024-09-18T14:08:12.680 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.680 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:12.680 INFO:tasks.workunit.client.0.smithi047.stdout: "osds": [ 2024-09-18T14:08:12.680 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.680 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 0, 2024-09-18T14:08:12.680 INFO:tasks.workunit.client.0.smithi047.stdout: "uuid": "2d333a34-fa14-4e6f-aca9-f14a57e36c76", 2024-09-18T14:08:12.680 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 1, 2024-09-18T14:08:12.680 INFO:tasks.workunit.client.0.smithi047.stdout: "in": 1, 2024-09-18T14:08:12.680 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 1, 2024-09-18T14:08:12.681 INFO:tasks.workunit.client.0.smithi047.stdout: "primary_affinity": 1, 2024-09-18T14:08:12.681 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_begin": 5, 2024-09-18T14:08:12.681 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_end": 40, 2024-09-18T14:08:12.681 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 57, 2024-09-18T14:08:12.681 INFO:tasks.workunit.client.0.smithi047.stdout: "up_thru": 69, 2024-09-18T14:08:12.681 INFO:tasks.workunit.client.0.smithi047.stdout: "down_at": 49, 2024-09-18T14:08:12.681 INFO:tasks.workunit.client.0.smithi047.stdout: "lost_at": 0, 2024-09-18T14:08:12.681 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addrs": { 2024-09-18T14:08:12.681 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:08:12.681 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.681 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:08:12.681 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6802", 2024-09-18T14:08:12.681 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 68576 2024-09-18T14:08:12.681 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.682 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.682 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:08:12.682 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6803", 2024-09-18T14:08:12.682 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 68576 2024-09-18T14:08:12.682 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.682 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:12.682 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.682 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addrs": { 2024-09-18T14:08:12.682 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:08:12.682 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.682 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:08:12.682 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6804", 2024-09-18T14:08:12.682 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 68576 2024-09-18T14:08:12.682 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.682 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.683 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:08:12.683 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6805", 2024-09-18T14:08:12.683 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 68576 2024-09-18T14:08:12.683 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.683 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:12.683 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.683 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addrs": { 2024-09-18T14:08:12.683 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:08:12.683 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.683 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:08:12.683 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6808", 2024-09-18T14:08:12.683 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 68576 2024-09-18T14:08:12.683 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.683 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.683 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:08:12.684 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6809", 2024-09-18T14:08:12.684 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 68576 2024-09-18T14:08:12.684 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.684 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:12.684 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.684 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addrs": { 2024-09-18T14:08:12.684 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:08:12.684 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.684 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:08:12.684 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6806", 2024-09-18T14:08:12.684 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 68576 2024-09-18T14:08:12.684 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.684 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.684 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:08:12.684 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6807", 2024-09-18T14:08:12.685 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 68576 2024-09-18T14:08:12.685 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.685 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:12.685 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.685 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addr": "127.0.0.1:6803/68576", 2024-09-18T14:08:12.685 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addr": "127.0.0.1:6805/68576", 2024-09-18T14:08:12.685 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addr": "127.0.0.1:6809/68576", 2024-09-18T14:08:12.685 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addr": "127.0.0.1:6807/68576", 2024-09-18T14:08:12.685 INFO:tasks.workunit.client.0.smithi047.stdout: "state": [ 2024-09-18T14:08:12.685 INFO:tasks.workunit.client.0.smithi047.stdout: "exists", 2024-09-18T14:08:12.685 INFO:tasks.workunit.client.0.smithi047.stdout: "up" 2024-09-18T14:08:12.685 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:12.685 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.685 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.685 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 1, 2024-09-18T14:08:12.686 INFO:tasks.workunit.client.0.smithi047.stdout: "uuid": "d16b04c1-1e01-47e1-87e5-e8058bd968e4", 2024-09-18T14:08:12.686 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 0, 2024-09-18T14:08:12.686 INFO:tasks.workunit.client.0.smithi047.stdout: "in": 1, 2024-09-18T14:08:12.686 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 1, 2024-09-18T14:08:12.686 INFO:tasks.workunit.client.0.smithi047.stdout: "primary_affinity": 1, 2024-09-18T14:08:12.686 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_begin": 0, 2024-09-18T14:08:12.686 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_end": 0, 2024-09-18T14:08:12.686 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 10, 2024-09-18T14:08:12.686 INFO:tasks.workunit.client.0.smithi047.stdout: "up_thru": 46, 2024-09-18T14:08:12.686 INFO:tasks.workunit.client.0.smithi047.stdout: "down_at": 50, 2024-09-18T14:08:12.686 INFO:tasks.workunit.client.0.smithi047.stdout: "lost_at": 0, 2024-09-18T14:08:12.686 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addrs": { 2024-09-18T14:08:12.686 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:08:12.686 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.686 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:08:12.687 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6810", 2024-09-18T14:08:12.687 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 62217 2024-09-18T14:08:12.687 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.687 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.687 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:08:12.687 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6811", 2024-09-18T14:08:12.687 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 62217 2024-09-18T14:08:12.687 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.687 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:12.687 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.687 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addrs": { 2024-09-18T14:08:12.687 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:08:12.687 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.687 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:08:12.687 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6812", 2024-09-18T14:08:12.688 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 62217 2024-09-18T14:08:12.688 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.688 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.688 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:08:12.688 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6813", 2024-09-18T14:08:12.688 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 62217 2024-09-18T14:08:12.688 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.688 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:12.688 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.688 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addrs": { 2024-09-18T14:08:12.688 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:08:12.688 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.688 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:08:12.688 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6816", 2024-09-18T14:08:12.689 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 62217 2024-09-18T14:08:12.689 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.689 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.689 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:08:12.689 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6817", 2024-09-18T14:08:12.689 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 62217 2024-09-18T14:08:12.689 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.689 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:12.689 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.689 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addrs": { 2024-09-18T14:08:12.689 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:08:12.689 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.689 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:08:12.689 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6814", 2024-09-18T14:08:12.689 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 62217 2024-09-18T14:08:12.690 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.690 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.690 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:08:12.690 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6815", 2024-09-18T14:08:12.690 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 62217 2024-09-18T14:08:12.690 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.690 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:12.690 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.690 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addr": "127.0.0.1:6811/62217", 2024-09-18T14:08:12.690 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addr": "127.0.0.1:6813/62217", 2024-09-18T14:08:12.690 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addr": "127.0.0.1:6817/62217", 2024-09-18T14:08:12.690 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addr": "127.0.0.1:6815/62217", 2024-09-18T14:08:12.690 INFO:tasks.workunit.client.0.smithi047.stdout: "state": [ 2024-09-18T14:08:12.690 INFO:tasks.workunit.client.0.smithi047.stdout: "exists" 2024-09-18T14:08:12.690 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:12.691 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.691 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.691 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 2, 2024-09-18T14:08:12.691 INFO:tasks.workunit.client.0.smithi047.stdout: "uuid": "20ba0a43-7c90-449e-bd64-6203deaebae3", 2024-09-18T14:08:12.691 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 1, 2024-09-18T14:08:12.691 INFO:tasks.workunit.client.0.smithi047.stdout: "in": 1, 2024-09-18T14:08:12.691 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 1, 2024-09-18T14:08:12.691 INFO:tasks.workunit.client.0.smithi047.stdout: "primary_affinity": 1, 2024-09-18T14:08:12.691 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_begin": 15, 2024-09-18T14:08:12.691 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_end": 40, 2024-09-18T14:08:12.691 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 61, 2024-09-18T14:08:12.691 INFO:tasks.workunit.client.0.smithi047.stdout: "up_thru": 67, 2024-09-18T14:08:12.691 INFO:tasks.workunit.client.0.smithi047.stdout: "down_at": 51, 2024-09-18T14:08:12.691 INFO:tasks.workunit.client.0.smithi047.stdout: "lost_at": 0, 2024-09-18T14:08:12.691 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addrs": { 2024-09-18T14:08:12.692 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:08:12.692 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.692 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:08:12.692 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6810", 2024-09-18T14:08:12.692 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 69154 2024-09-18T14:08:12.692 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.692 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.692 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:08:12.692 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6811", 2024-09-18T14:08:12.692 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 69154 2024-09-18T14:08:12.692 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.692 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:12.692 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.692 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addrs": { 2024-09-18T14:08:12.692 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:08:12.693 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.693 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:08:12.693 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6812", 2024-09-18T14:08:12.693 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 69154 2024-09-18T14:08:12.693 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.693 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.693 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:08:12.693 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6813", 2024-09-18T14:08:12.693 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 69154 2024-09-18T14:08:12.693 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.693 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:12.693 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.693 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addrs": { 2024-09-18T14:08:12.693 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:08:12.694 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.694 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:08:12.694 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6816", 2024-09-18T14:08:12.694 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 69154 2024-09-18T14:08:12.694 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.694 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.694 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:08:12.694 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6817", 2024-09-18T14:08:12.694 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 69154 2024-09-18T14:08:12.694 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.694 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:12.694 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.694 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addrs": { 2024-09-18T14:08:12.694 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:08:12.694 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.695 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:08:12.695 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6814", 2024-09-18T14:08:12.695 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 69154 2024-09-18T14:08:12.695 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.695 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.695 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:08:12.695 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6815", 2024-09-18T14:08:12.695 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 69154 2024-09-18T14:08:12.695 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.695 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:12.695 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.695 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addr": "127.0.0.1:6811/69154", 2024-09-18T14:08:12.695 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addr": "127.0.0.1:6813/69154", 2024-09-18T14:08:12.695 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addr": "127.0.0.1:6817/69154", 2024-09-18T14:08:12.696 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addr": "127.0.0.1:6815/69154", 2024-09-18T14:08:12.696 INFO:tasks.workunit.client.0.smithi047.stdout: "state": [ 2024-09-18T14:08:12.696 INFO:tasks.workunit.client.0.smithi047.stdout: "exists", 2024-09-18T14:08:12.696 INFO:tasks.workunit.client.0.smithi047.stdout: "up" 2024-09-18T14:08:12.696 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:12.696 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.696 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:12.696 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_xinfo": [ 2024-09-18T14:08:12.696 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.696 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 0, 2024-09-18T14:08:12.696 INFO:tasks.workunit.client.0.smithi047.stdout: "down_stamp": "2024-09-18T14:07:20.887595+0000", 2024-09-18T14:08:12.697 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_probability": 0, 2024-09-18T14:08:12.697 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_interval": 0, 2024-09-18T14:08:12.697 INFO:tasks.workunit.client.0.smithi047.stdout: "features": 4540138320759226367, 2024-09-18T14:08:12.697 INFO:tasks.workunit.client.0.smithi047.stdout: "old_weight": 0, 2024-09-18T14:08:12.697 INFO:tasks.workunit.client.0.smithi047.stdout: "last_purged_snaps_scrub": "2024-09-18T14:05:17.323724+0000", 2024-09-18T14:08:12.697 INFO:tasks.workunit.client.0.smithi047.stdout: "dead_epoch": 0 2024-09-18T14:08:12.697 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.697 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.697 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 1, 2024-09-18T14:08:12.697 INFO:tasks.workunit.client.0.smithi047.stdout: "down_stamp": "2024-09-18T14:07:21.512618+0000", 2024-09-18T14:08:12.697 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_probability": 0, 2024-09-18T14:08:12.697 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_interval": 0, 2024-09-18T14:08:12.697 INFO:tasks.workunit.client.0.smithi047.stdout: "features": 4540138320759226367, 2024-09-18T14:08:12.697 INFO:tasks.workunit.client.0.smithi047.stdout: "old_weight": 0, 2024-09-18T14:08:12.698 INFO:tasks.workunit.client.0.smithi047.stdout: "last_purged_snaps_scrub": "2024-09-18T14:05:50.556491+0000", 2024-09-18T14:08:12.698 INFO:tasks.workunit.client.0.smithi047.stdout: "dead_epoch": 0 2024-09-18T14:08:12.698 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.698 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.698 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 2, 2024-09-18T14:08:12.698 INFO:tasks.workunit.client.0.smithi047.stdout: "down_stamp": "2024-09-18T14:07:22.088585+0000", 2024-09-18T14:08:12.698 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_probability": 0, 2024-09-18T14:08:12.698 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_interval": 0, 2024-09-18T14:08:12.698 INFO:tasks.workunit.client.0.smithi047.stdout: "features": 4540138320759226367, 2024-09-18T14:08:12.698 INFO:tasks.workunit.client.0.smithi047.stdout: "old_weight": 0, 2024-09-18T14:08:12.698 INFO:tasks.workunit.client.0.smithi047.stdout: "last_purged_snaps_scrub": "2024-09-18T14:06:23.474574+0000", 2024-09-18T14:08:12.698 INFO:tasks.workunit.client.0.smithi047.stdout: "dead_epoch": 0 2024-09-18T14:08:12.698 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.698 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:12.698 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_upmap": [], 2024-09-18T14:08:12.699 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_upmap_items": [], 2024-09-18T14:08:12.699 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_temp": [], 2024-09-18T14:08:12.699 INFO:tasks.workunit.client.0.smithi047.stdout: "primary_temp": [], 2024-09-18T14:08:12.699 INFO:tasks.workunit.client.0.smithi047.stdout: "blocklist": {}, 2024-09-18T14:08:12.699 INFO:tasks.workunit.client.0.smithi047.stdout: "range_blocklist": {}, 2024-09-18T14:08:12.699 INFO:tasks.workunit.client.0.smithi047.stdout: "erasure_code_profiles": { 2024-09-18T14:08:12.699 INFO:tasks.workunit.client.0.smithi047.stdout: "default": { 2024-09-18T14:08:12.699 INFO:tasks.workunit.client.0.smithi047.stdout: "k": "2", 2024-09-18T14:08:12.699 INFO:tasks.workunit.client.0.smithi047.stdout: "m": "2", 2024-09-18T14:08:12.699 INFO:tasks.workunit.client.0.smithi047.stdout: "plugin": "jerasure", 2024-09-18T14:08:12.699 INFO:tasks.workunit.client.0.smithi047.stdout: "technique": "reed_sol_van" 2024-09-18T14:08:12.699 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.699 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.699 INFO:tasks.workunit.client.0.smithi047.stdout: "removed_snaps_queue": [], 2024-09-18T14:08:12.699 INFO:tasks.workunit.client.0.smithi047.stdout: "new_removed_snaps": [], 2024-09-18T14:08:12.700 INFO:tasks.workunit.client.0.smithi047.stdout: "new_purged_snaps": [], 2024-09-18T14:08:12.700 INFO:tasks.workunit.client.0.smithi047.stdout: "crush_node_flags": {}, 2024-09-18T14:08:12.700 INFO:tasks.workunit.client.0.smithi047.stdout: "device_class_flags": {}, 2024-09-18T14:08:12.700 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_mode": { 2024-09-18T14:08:12.700 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_mode_enabled": false, 2024-09-18T14:08:12.700 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_bucket_count": 0, 2024-09-18T14:08:12.700 INFO:tasks.workunit.client.0.smithi047.stdout: "degraded_stretch_mode": 0, 2024-09-18T14:08:12.700 INFO:tasks.workunit.client.0.smithi047.stdout: "recovering_stretch_mode": 0, 2024-09-18T14:08:12.700 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_mode_bucket": 0 2024-09-18T14:08:12.701 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.701 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.701 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_metadata": [ 2024-09-18T14:08:12.701 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.701 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T14:08:12.701 INFO:tasks.workunit.client.0.smithi047.stdout: "arch": "x86_64", 2024-09-18T14:08:12.701 INFO:tasks.workunit.client.0.smithi047.stdout: "back_addr": "[v2:127.0.0.1:6804/68576,v1:127.0.0.1:6805/68576]", 2024-09-18T14:08:12.701 INFO:tasks.workunit.client.0.smithi047.stdout: "back_iface": "", 2024-09-18T14:08:12.701 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs": "1", 2024-09-18T14:08:12.701 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_db": "0", 2024-09-18T14:08:12.701 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_wal": "0", 2024-09-18T14:08:12.702 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_single_shared_device": "1", 2024-09-18T14:08:12.702 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_allocation_from_file": "0", 2024-09-18T14:08:12.702 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_access_mode": "file", 2024-09-18T14:08:12.702 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_block_size": "4096", 2024-09-18T14:08:12.702 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_devices": "sda", 2024-09-18T14:08:12.702 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-18T14:08:12.702 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-18T14:08:12.702 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-18T14:08:12.702 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_rotational": "1", 2024-09-18T14:08:12.702 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_size": "107374182400", 2024-09-18T14:08:12.703 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_support_discard": "1", 2024-09-18T14:08:12.703 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_type": "hdd", 2024-09-18T14:08:12.703 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_min_alloc_size": "4096", 2024-09-18T14:08:12.703 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_release": "quincy", 2024-09-18T14:08:12.703 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:08:12.703 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_short": "17.2.7-1645-g7061ea6c", 2024-09-18T14:08:12.703 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_when_created": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:08:12.703 INFO:tasks.workunit.client.0.smithi047.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-18T14:08:12.703 INFO:tasks.workunit.client.0.smithi047.stdout: "created_at": "2024-09-18T14:04:51.996055Z", 2024-09-18T14:08:12.703 INFO:tasks.workunit.client.0.smithi047.stdout: "default_device_class": "hdd", 2024-09-18T14:08:12.703 INFO:tasks.workunit.client.0.smithi047.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4HN80", 2024-09-18T14:08:12.703 INFO:tasks.workunit.client.0.smithi047.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-18T14:08:12.703 INFO:tasks.workunit.client.0.smithi047.stdout: "devices": "sda", 2024-09-18T14:08:12.703 INFO:tasks.workunit.client.0.smithi047.stdout: "distro": "centos", 2024-09-18T14:08:12.703 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_description": "CentOS Stream 9", 2024-09-18T14:08:12.704 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_version": "9", 2024-09-18T14:08:12.704 INFO:tasks.workunit.client.0.smithi047.stdout: "front_addr": "[v2:127.0.0.1:6802/68576,v1:127.0.0.1:6803/68576]", 2024-09-18T14:08:12.704 INFO:tasks.workunit.client.0.smithi047.stdout: "front_iface": "", 2024-09-18T14:08:12.704 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/68576,v1:127.0.0.1:6809/68576]", 2024-09-18T14:08:12.704 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/68576,v1:127.0.0.1:6807/68576]", 2024-09-18T14:08:12.704 INFO:tasks.workunit.client.0.smithi047.stdout: "hostname": "smithi047", 2024-09-18T14:08:12.704 INFO:tasks.workunit.client.0.smithi047.stdout: "journal_rotational": "1", 2024-09-18T14:08:12.704 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-18T14:08:12.704 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-18T14:08:12.704 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_swap_kb": "0", 2024-09-18T14:08:12.704 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_total_kb": "32493492", 2024-09-18T14:08:12.704 INFO:tasks.workunit.client.0.smithi047.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-18T14:08:12.704 INFO:tasks.workunit.client.0.smithi047.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-18T14:08:12.704 INFO:tasks.workunit.client.0.smithi047.stdout: "os": "Linux", 2024-09-18T14:08:12.705 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_data": "td/divergent-priors/0", 2024-09-18T14:08:12.705 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_objectstore": "bluestore", 2024-09-18T14:08:12.705 INFO:tasks.workunit.client.0.smithi047.stdout: "osdspec_affinity": "", 2024-09-18T14:08:12.705 INFO:tasks.workunit.client.0.smithi047.stdout: "rotational": "1" 2024-09-18T14:08:12.705 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.705 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.705 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 1, 2024-09-18T14:08:12.705 INFO:tasks.workunit.client.0.smithi047.stdout: "arch": "x86_64", 2024-09-18T14:08:12.705 INFO:tasks.workunit.client.0.smithi047.stdout: "back_addr": "[v2:127.0.0.1:6812/62217,v1:127.0.0.1:6813/62217]", 2024-09-18T14:08:12.705 INFO:tasks.workunit.client.0.smithi047.stdout: "back_iface": "", 2024-09-18T14:08:12.705 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs": "1", 2024-09-18T14:08:12.705 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_db": "0", 2024-09-18T14:08:12.705 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_wal": "0", 2024-09-18T14:08:12.705 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_single_shared_device": "1", 2024-09-18T14:08:12.706 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_allocation_from_file": "0", 2024-09-18T14:08:12.706 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_access_mode": "file", 2024-09-18T14:08:12.706 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_block_size": "4096", 2024-09-18T14:08:12.706 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_devices": "sda", 2024-09-18T14:08:12.706 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-18T14:08:12.706 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-18T14:08:12.706 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-18T14:08:12.706 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_rotational": "1", 2024-09-18T14:08:12.706 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_size": "107374182400", 2024-09-18T14:08:12.706 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_support_discard": "1", 2024-09-18T14:08:12.706 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_type": "hdd", 2024-09-18T14:08:12.706 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_min_alloc_size": "4096", 2024-09-18T14:08:12.706 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_release": "quincy", 2024-09-18T14:08:12.706 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:08:12.707 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_short": "17.2.7-1645-g7061ea6c", 2024-09-18T14:08:12.707 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_when_created": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:08:12.707 INFO:tasks.workunit.client.0.smithi047.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-18T14:08:12.707 INFO:tasks.workunit.client.0.smithi047.stdout: "created_at": "2024-09-18T14:05:24.241405Z", 2024-09-18T14:08:12.707 INFO:tasks.workunit.client.0.smithi047.stdout: "default_device_class": "hdd", 2024-09-18T14:08:12.707 INFO:tasks.workunit.client.0.smithi047.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4HN80", 2024-09-18T14:08:12.707 INFO:tasks.workunit.client.0.smithi047.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-18T14:08:12.707 INFO:tasks.workunit.client.0.smithi047.stdout: "devices": "sda", 2024-09-18T14:08:12.707 INFO:tasks.workunit.client.0.smithi047.stdout: "distro": "centos", 2024-09-18T14:08:12.707 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_description": "CentOS Stream 9", 2024-09-18T14:08:12.707 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_version": "9", 2024-09-18T14:08:12.707 INFO:tasks.workunit.client.0.smithi047.stdout: "front_addr": "[v2:127.0.0.1:6810/62217,v1:127.0.0.1:6811/62217]", 2024-09-18T14:08:12.707 INFO:tasks.workunit.client.0.smithi047.stdout: "front_iface": "", 2024-09-18T14:08:12.707 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/62217,v1:127.0.0.1:6817/62217]", 2024-09-18T14:08:12.708 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/62217,v1:127.0.0.1:6815/62217]", 2024-09-18T14:08:12.708 INFO:tasks.workunit.client.0.smithi047.stdout: "hostname": "smithi047", 2024-09-18T14:08:12.708 INFO:tasks.workunit.client.0.smithi047.stdout: "journal_rotational": "1", 2024-09-18T14:08:12.708 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-18T14:08:12.708 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-18T14:08:12.708 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_swap_kb": "0", 2024-09-18T14:08:12.708 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_total_kb": "32493492", 2024-09-18T14:08:12.708 INFO:tasks.workunit.client.0.smithi047.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-18T14:08:12.708 INFO:tasks.workunit.client.0.smithi047.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-18T14:08:12.708 INFO:tasks.workunit.client.0.smithi047.stdout: "os": "Linux", 2024-09-18T14:08:12.708 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_data": "td/divergent-priors/1", 2024-09-18T14:08:12.708 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_objectstore": "bluestore", 2024-09-18T14:08:12.708 INFO:tasks.workunit.client.0.smithi047.stdout: "osdspec_affinity": "", 2024-09-18T14:08:12.708 INFO:tasks.workunit.client.0.smithi047.stdout: "rotational": "1" 2024-09-18T14:08:12.708 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.709 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.709 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T14:08:12.709 INFO:tasks.workunit.client.0.smithi047.stdout: "arch": "x86_64", 2024-09-18T14:08:12.709 INFO:tasks.workunit.client.0.smithi047.stdout: "back_addr": "[v2:127.0.0.1:6812/69154,v1:127.0.0.1:6813/69154]", 2024-09-18T14:08:12.709 INFO:tasks.workunit.client.0.smithi047.stdout: "back_iface": "", 2024-09-18T14:08:12.709 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs": "1", 2024-09-18T14:08:12.709 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_db": "0", 2024-09-18T14:08:12.709 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_wal": "0", 2024-09-18T14:08:12.709 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_single_shared_device": "1", 2024-09-18T14:08:12.709 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_allocation_from_file": "0", 2024-09-18T14:08:12.709 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_access_mode": "file", 2024-09-18T14:08:12.709 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_block_size": "4096", 2024-09-18T14:08:12.709 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_devices": "sda", 2024-09-18T14:08:12.709 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-18T14:08:12.710 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-18T14:08:12.710 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-18T14:08:12.710 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_rotational": "1", 2024-09-18T14:08:12.710 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_size": "107374182400", 2024-09-18T14:08:12.710 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_support_discard": "1", 2024-09-18T14:08:12.710 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_type": "hdd", 2024-09-18T14:08:12.710 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_min_alloc_size": "4096", 2024-09-18T14:08:12.710 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_release": "quincy", 2024-09-18T14:08:12.710 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:08:12.710 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_short": "17.2.7-1645-g7061ea6c", 2024-09-18T14:08:12.710 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_when_created": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:08:12.711 INFO:tasks.workunit.client.0.smithi047.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-18T14:08:12.711 INFO:tasks.workunit.client.0.smithi047.stdout: "created_at": "2024-09-18T14:05:56.801581Z", 2024-09-18T14:08:12.711 INFO:tasks.workunit.client.0.smithi047.stdout: "default_device_class": "hdd", 2024-09-18T14:08:12.711 INFO:tasks.workunit.client.0.smithi047.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4HN80", 2024-09-18T14:08:12.711 INFO:tasks.workunit.client.0.smithi047.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-18T14:08:12.711 INFO:tasks.workunit.client.0.smithi047.stdout: "devices": "sda", 2024-09-18T14:08:12.711 INFO:tasks.workunit.client.0.smithi047.stdout: "distro": "centos", 2024-09-18T14:08:12.711 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_description": "CentOS Stream 9", 2024-09-18T14:08:12.711 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_version": "9", 2024-09-18T14:08:12.711 INFO:tasks.workunit.client.0.smithi047.stdout: "front_addr": "[v2:127.0.0.1:6810/69154,v1:127.0.0.1:6811/69154]", 2024-09-18T14:08:12.711 INFO:tasks.workunit.client.0.smithi047.stdout: "front_iface": "", 2024-09-18T14:08:12.711 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/69154,v1:127.0.0.1:6817/69154]", 2024-09-18T14:08:12.711 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/69154,v1:127.0.0.1:6815/69154]", 2024-09-18T14:08:12.711 INFO:tasks.workunit.client.0.smithi047.stdout: "hostname": "smithi047", 2024-09-18T14:08:12.711 INFO:tasks.workunit.client.0.smithi047.stdout: "journal_rotational": "1", 2024-09-18T14:08:12.712 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-18T14:08:12.712 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-18T14:08:12.712 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_swap_kb": "0", 2024-09-18T14:08:12.712 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_total_kb": "32493492", 2024-09-18T14:08:12.712 INFO:tasks.workunit.client.0.smithi047.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-18T14:08:12.712 INFO:tasks.workunit.client.0.smithi047.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-18T14:08:12.712 INFO:tasks.workunit.client.0.smithi047.stdout: "os": "Linux", 2024-09-18T14:08:12.712 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_data": "td/divergent-priors/2", 2024-09-18T14:08:12.712 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_objectstore": "bluestore", 2024-09-18T14:08:12.712 INFO:tasks.workunit.client.0.smithi047.stdout: "osdspec_affinity": "", 2024-09-18T14:08:12.712 INFO:tasks.workunit.client.0.smithi047.stdout: "rotational": "1" 2024-09-18T14:08:12.712 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.712 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:12.713 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap_clean_epochs": { 2024-09-18T14:08:12.713 INFO:tasks.workunit.client.0.smithi047.stdout: "min_last_epoch_clean": 0, 2024-09-18T14:08:12.713 INFO:tasks.workunit.client.0.smithi047.stdout: "last_epoch_clean": { 2024-09-18T14:08:12.713 INFO:tasks.workunit.client.0.smithi047.stdout: "per_pool": [] 2024-09-18T14:08:12.713 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.713 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_epochs": [ 2024-09-18T14:08:12.713 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.713 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T14:08:12.713 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 57 2024-09-18T14:08:12.713 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.713 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.713 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T14:08:12.713 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 61 2024-09-18T14:08:12.713 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.714 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:12.714 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.714 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap_first_committed": 1, 2024-09-18T14:08:12.714 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap_last_committed": 70, 2024-09-18T14:08:12.714 INFO:tasks.workunit.client.0.smithi047.stdout: "crushmap": { 2024-09-18T14:08:12.714 INFO:tasks.workunit.client.0.smithi047.stdout: "devices": [ 2024-09-18T14:08:12.714 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.714 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T14:08:12.714 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "osd.0", 2024-09-18T14:08:12.714 INFO:tasks.workunit.client.0.smithi047.stdout: "class": "hdd" 2024-09-18T14:08:12.714 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.714 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.714 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 1, 2024-09-18T14:08:12.714 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "osd.1", 2024-09-18T14:08:12.714 INFO:tasks.workunit.client.0.smithi047.stdout: "class": "hdd" 2024-09-18T14:08:12.715 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.715 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.715 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T14:08:12.715 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "osd.2", 2024-09-18T14:08:12.715 INFO:tasks.workunit.client.0.smithi047.stdout: "class": "hdd" 2024-09-18T14:08:12.715 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.715 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:12.715 INFO:tasks.workunit.client.0.smithi047.stdout: "types": [ 2024-09-18T14:08:12.715 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.715 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 0, 2024-09-18T14:08:12.715 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "osd" 2024-09-18T14:08:12.715 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.715 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.715 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 1, 2024-09-18T14:08:12.716 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "host" 2024-09-18T14:08:12.716 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.716 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.716 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 2, 2024-09-18T14:08:12.716 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "chassis" 2024-09-18T14:08:12.716 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.716 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.716 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 3, 2024-09-18T14:08:12.716 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "rack" 2024-09-18T14:08:12.716 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.717 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.717 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 4, 2024-09-18T14:08:12.717 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "row" 2024-09-18T14:08:12.717 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.717 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.717 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 5, 2024-09-18T14:08:12.717 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "pdu" 2024-09-18T14:08:12.717 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.717 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.717 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 6, 2024-09-18T14:08:12.717 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "pod" 2024-09-18T14:08:12.717 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.717 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.717 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 7, 2024-09-18T14:08:12.717 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "room" 2024-09-18T14:08:12.718 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.718 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.718 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 8, 2024-09-18T14:08:12.718 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "datacenter" 2024-09-18T14:08:12.718 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.718 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.718 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 9, 2024-09-18T14:08:12.718 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "zone" 2024-09-18T14:08:12.718 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.718 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.718 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 10, 2024-09-18T14:08:12.718 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "region" 2024-09-18T14:08:12.719 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.719 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.719 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 11, 2024-09-18T14:08:12.719 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "root" 2024-09-18T14:08:12.719 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.719 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:12.719 INFO:tasks.workunit.client.0.smithi047.stdout: "buckets": [ 2024-09-18T14:08:12.719 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.719 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -1, 2024-09-18T14:08:12.719 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "default", 2024-09-18T14:08:12.719 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 11, 2024-09-18T14:08:12.719 INFO:tasks.workunit.client.0.smithi047.stdout: "type_name": "root", 2024-09-18T14:08:12.719 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:08:12.719 INFO:tasks.workunit.client.0.smithi047.stdout: "alg": "straw2", 2024-09-18T14:08:12.720 INFO:tasks.workunit.client.0.smithi047.stdout: "hash": "rjenkins1", 2024-09-18T14:08:12.720 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T14:08:12.720 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.720 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -3, 2024-09-18T14:08:12.720 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:08:12.720 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 0 2024-09-18T14:08:12.720 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.720 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:12.720 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.720 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.720 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -2, 2024-09-18T14:08:12.720 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "default~hdd", 2024-09-18T14:08:12.720 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 11, 2024-09-18T14:08:12.720 INFO:tasks.workunit.client.0.smithi047.stdout: "type_name": "root", 2024-09-18T14:08:12.720 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:08:12.721 INFO:tasks.workunit.client.0.smithi047.stdout: "alg": "straw2", 2024-09-18T14:08:12.721 INFO:tasks.workunit.client.0.smithi047.stdout: "hash": "rjenkins1", 2024-09-18T14:08:12.721 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T14:08:12.721 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.721 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -4, 2024-09-18T14:08:12.721 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:08:12.721 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 0 2024-09-18T14:08:12.721 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.721 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:12.721 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.722 INFO:tasks.workunit.client.0.smithi047.stderr:report 2788040041 2024-09-18T14:08:12.722 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.722 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -3, 2024-09-18T14:08:12.722 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "smithi047", 2024-09-18T14:08:12.722 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 1, 2024-09-18T14:08:12.722 INFO:tasks.workunit.client.0.smithi047.stdout: "type_name": "host", 2024-09-18T14:08:12.722 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:08:12.722 INFO:tasks.workunit.client.0.smithi047.stdout: "alg": "straw2", 2024-09-18T14:08:12.722 INFO:tasks.workunit.client.0.smithi047.stdout: "hash": "rjenkins1", 2024-09-18T14:08:12.722 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T14:08:12.723 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.723 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T14:08:12.723 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:08:12.723 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 0 2024-09-18T14:08:12.723 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.723 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.723 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 1, 2024-09-18T14:08:12.723 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:08:12.723 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 1 2024-09-18T14:08:12.723 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.723 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.723 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T14:08:12.723 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:08:12.723 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 2 2024-09-18T14:08:12.723 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.724 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:12.724 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.724 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.724 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -4, 2024-09-18T14:08:12.724 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "smithi047~hdd", 2024-09-18T14:08:12.724 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 1, 2024-09-18T14:08:12.724 INFO:tasks.workunit.client.0.smithi047.stdout: "type_name": "host", 2024-09-18T14:08:12.724 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:08:12.724 INFO:tasks.workunit.client.0.smithi047.stdout: "alg": "straw2", 2024-09-18T14:08:12.724 INFO:tasks.workunit.client.0.smithi047.stdout: "hash": "rjenkins1", 2024-09-18T14:08:12.724 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T14:08:12.724 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.724 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T14:08:12.724 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:08:12.725 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 0 2024-09-18T14:08:12.725 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.725 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.725 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 1, 2024-09-18T14:08:12.725 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:08:12.725 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 1 2024-09-18T14:08:12.725 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.725 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.725 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T14:08:12.725 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:08:12.725 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 2 2024-09-18T14:08:12.725 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.725 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:12.725 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.725 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:12.726 INFO:tasks.workunit.client.0.smithi047.stdout: "rules": [ 2024-09-18T14:08:12.726 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.726 INFO:tasks.workunit.client.0.smithi047.stdout: "rule_id": 0, 2024-09-18T14:08:12.726 INFO:tasks.workunit.client.0.smithi047.stdout: "rule_name": "replicated_rule", 2024-09-18T14:08:12.726 INFO:tasks.workunit.client.0.smithi047.stdout: "type": 1, 2024-09-18T14:08:12.726 INFO:tasks.workunit.client.0.smithi047.stdout: "steps": [ 2024-09-18T14:08:12.726 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.726 INFO:tasks.workunit.client.0.smithi047.stdout: "op": "take", 2024-09-18T14:08:12.726 INFO:tasks.workunit.client.0.smithi047.stdout: "item": -1, 2024-09-18T14:08:12.726 INFO:tasks.workunit.client.0.smithi047.stdout: "item_name": "default" 2024-09-18T14:08:12.726 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.726 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.726 INFO:tasks.workunit.client.0.smithi047.stdout: "op": "choose_firstn", 2024-09-18T14:08:12.727 INFO:tasks.workunit.client.0.smithi047.stdout: "num": 0, 2024-09-18T14:08:12.727 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "osd" 2024-09-18T14:08:12.727 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.727 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.727 INFO:tasks.workunit.client.0.smithi047.stdout: "op": "emit" 2024-09-18T14:08:12.727 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.727 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:12.727 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.727 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:12.727 INFO:tasks.workunit.client.0.smithi047.stdout: "tunables": { 2024-09-18T14:08:12.727 INFO:tasks.workunit.client.0.smithi047.stdout: "choose_local_tries": 0, 2024-09-18T14:08:12.727 INFO:tasks.workunit.client.0.smithi047.stdout: "choose_local_fallback_tries": 0, 2024-09-18T14:08:12.727 INFO:tasks.workunit.client.0.smithi047.stdout: "choose_total_tries": 50, 2024-09-18T14:08:12.727 INFO:tasks.workunit.client.0.smithi047.stdout: "chooseleaf_descend_once": 1, 2024-09-18T14:08:12.728 INFO:tasks.workunit.client.0.smithi047.stdout: "chooseleaf_vary_r": 1, 2024-09-18T14:08:12.728 INFO:tasks.workunit.client.0.smithi047.stdout: "chooseleaf_stable": 1, 2024-09-18T14:08:12.728 INFO:tasks.workunit.client.0.smithi047.stdout: "straw_calc_version": 1, 2024-09-18T14:08:12.728 INFO:tasks.workunit.client.0.smithi047.stdout: "allowed_bucket_algs": 54, 2024-09-18T14:08:12.728 INFO:tasks.workunit.client.0.smithi047.stdout: "profile": "jewel", 2024-09-18T14:08:12.728 INFO:tasks.workunit.client.0.smithi047.stdout: "optimal_tunables": 1, 2024-09-18T14:08:12.728 INFO:tasks.workunit.client.0.smithi047.stdout: "legacy_tunables": 0, 2024-09-18T14:08:12.728 INFO:tasks.workunit.client.0.smithi047.stdout: "minimum_required_version": "jewel", 2024-09-18T14:08:12.728 INFO:tasks.workunit.client.0.smithi047.stdout: "require_feature_tunables": 1, 2024-09-18T14:08:12.728 INFO:tasks.workunit.client.0.smithi047.stdout: "require_feature_tunables2": 1, 2024-09-18T14:08:12.728 INFO:tasks.workunit.client.0.smithi047.stdout: "has_v2_rules": 0, 2024-09-18T14:08:12.728 INFO:tasks.workunit.client.0.smithi047.stdout: "require_feature_tunables3": 1, 2024-09-18T14:08:12.728 INFO:tasks.workunit.client.0.smithi047.stdout: "has_v3_rules": 0, 2024-09-18T14:08:12.728 INFO:tasks.workunit.client.0.smithi047.stdout: "has_v4_buckets": 1, 2024-09-18T14:08:12.729 INFO:tasks.workunit.client.0.smithi047.stdout: "require_feature_tunables5": 1, 2024-09-18T14:08:12.729 INFO:tasks.workunit.client.0.smithi047.stdout: "has_v5_rules": 0 2024-09-18T14:08:12.729 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.729 INFO:tasks.workunit.client.0.smithi047.stdout: "choose_args": {} 2024-09-18T14:08:12.729 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.729 INFO:tasks.workunit.client.0.smithi047.stdout: "fsmap": { 2024-09-18T14:08:12.729 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 1, 2024-09-18T14:08:12.729 INFO:tasks.workunit.client.0.smithi047.stdout: "default_fscid": -1, 2024-09-18T14:08:12.729 INFO:tasks.workunit.client.0.smithi047.stdout: "compat": { 2024-09-18T14:08:12.729 INFO:tasks.workunit.client.0.smithi047.stdout: "compat": {}, 2024-09-18T14:08:12.729 INFO:tasks.workunit.client.0.smithi047.stdout: "ro_compat": {}, 2024-09-18T14:08:12.729 INFO:tasks.workunit.client.0.smithi047.stdout: "incompat": { 2024-09-18T14:08:12.729 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_1": "base v0.20", 2024-09-18T14:08:12.729 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_2": "client writeable ranges", 2024-09-18T14:08:12.729 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_3": "default file layouts on dirs", 2024-09-18T14:08:12.730 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_4": "dir inode in separate object", 2024-09-18T14:08:12.730 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_5": "mds uses versioned encoding", 2024-09-18T14:08:12.730 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-18T14:08:12.730 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_8": "no anchor table", 2024-09-18T14:08:12.730 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_9": "file layout v2", 2024-09-18T14:08:12.730 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_10": "snaprealm v2" 2024-09-18T14:08:12.730 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.730 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.730 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_flags": { 2024-09-18T14:08:12.730 INFO:tasks.workunit.client.0.smithi047.stdout: "enable_multiple": true, 2024-09-18T14:08:12.730 INFO:tasks.workunit.client.0.smithi047.stdout: "ever_enabled_multiple": true 2024-09-18T14:08:12.730 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.730 INFO:tasks.workunit.client.0.smithi047.stdout: "standbys": [], 2024-09-18T14:08:12.730 INFO:tasks.workunit.client.0.smithi047.stdout: "filesystems": [] 2024-09-18T14:08:12.730 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.731 INFO:tasks.workunit.client.0.smithi047.stdout: "mdsmap_first_committed": 1, 2024-09-18T14:08:12.731 INFO:tasks.workunit.client.0.smithi047.stdout: "mdsmap_last_committed": 1, 2024-09-18T14:08:12.731 INFO:tasks.workunit.client.0.smithi047.stdout: "auth": { 2024-09-18T14:08:12.731 INFO:tasks.workunit.client.0.smithi047.stdout: "first_committed": 1, 2024-09-18T14:08:12.731 INFO:tasks.workunit.client.0.smithi047.stdout: "last_committed": 5, 2024-09-18T14:08:12.731 INFO:tasks.workunit.client.0.smithi047.stdout: "num_secrets": 3 2024-09-18T14:08:12.731 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.731 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg": 32, 2024-09-18T14:08:12.731 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg_active": 32, 2024-09-18T14:08:12.731 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg_unknown": 0, 2024-09-18T14:08:12.731 INFO:tasks.workunit.client.0.smithi047.stdout: "num_osd": 3, 2024-09-18T14:08:12.731 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_sum": { 2024-09-18T14:08:12.731 INFO:tasks.workunit.client.0.smithi047.stdout: "stat_sum": { 2024-09-18T14:08:12.731 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes": 48059, 2024-09-18T14:08:12.732 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects": 100, 2024-09-18T14:08:12.732 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_clones": 0, 2024-09-18T14:08:12.732 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_copies": 300, 2024-09-18T14:08:12.732 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing_on_primary": 0, 2024-09-18T14:08:12.732 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing": 0, 2024-09-18T14:08:12.732 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_degraded": 100, 2024-09-18T14:08:12.732 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_misplaced": 0, 2024-09-18T14:08:12.732 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_unfound": 0, 2024-09-18T14:08:12.732 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_dirty": 100, 2024-09-18T14:08:12.732 INFO:tasks.workunit.client.0.smithi047.stdout: "num_whiteouts": 0, 2024-09-18T14:08:12.732 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read": 0, 2024-09-18T14:08:12.732 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read_kb": 0, 2024-09-18T14:08:12.732 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write": 81, 2024-09-18T14:08:12.732 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write_kb": 81, 2024-09-18T14:08:12.733 INFO:tasks.workunit.client.0.smithi047.stdout: "num_scrub_errors": 0, 2024-09-18T14:08:12.733 INFO:tasks.workunit.client.0.smithi047.stdout: "num_shallow_scrub_errors": 0, 2024-09-18T14:08:12.733 INFO:tasks.workunit.client.0.smithi047.stdout: "num_deep_scrub_errors": 0, 2024-09-18T14:08:12.733 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_recovered": 0, 2024-09-18T14:08:12.733 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_recovered": 0, 2024-09-18T14:08:12.733 INFO:tasks.workunit.client.0.smithi047.stdout: "num_keys_recovered": 0, 2024-09-18T14:08:12.733 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_omap": 0, 2024-09-18T14:08:12.733 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_hit_set_archive": 0, 2024-09-18T14:08:12.733 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_hit_set_archive": 0, 2024-09-18T14:08:12.733 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush": 0, 2024-09-18T14:08:12.733 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_kb": 0, 2024-09-18T14:08:12.733 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict": 0, 2024-09-18T14:08:12.733 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_kb": 0, 2024-09-18T14:08:12.733 INFO:tasks.workunit.client.0.smithi047.stdout: "num_promote": 0, 2024-09-18T14:08:12.733 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_high": 0, 2024-09-18T14:08:12.734 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_low": 0, 2024-09-18T14:08:12.734 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_some": 0, 2024-09-18T14:08:12.734 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_full": 0, 2024-09-18T14:08:12.734 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_pinned": 0, 2024-09-18T14:08:12.734 INFO:tasks.workunit.client.0.smithi047.stdout: "num_legacy_snapsets": 0, 2024-09-18T14:08:12.734 INFO:tasks.workunit.client.0.smithi047.stdout: "num_large_omap_objects": 0, 2024-09-18T14:08:12.734 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_manifest": 0, 2024-09-18T14:08:12.734 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_bytes": 0, 2024-09-18T14:08:12.734 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_keys": 0, 2024-09-18T14:08:12.734 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_repaired": 0 2024-09-18T14:08:12.734 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.734 INFO:tasks.workunit.client.0.smithi047.stdout: "store_stats": { 2024-09-18T14:08:12.734 INFO:tasks.workunit.client.0.smithi047.stdout: "total": 0, 2024-09-18T14:08:12.734 INFO:tasks.workunit.client.0.smithi047.stdout: "available": 0, 2024-09-18T14:08:12.735 INFO:tasks.workunit.client.0.smithi047.stdout: "internally_reserved": 0, 2024-09-18T14:08:12.735 INFO:tasks.workunit.client.0.smithi047.stdout: "allocated": 0, 2024-09-18T14:08:12.735 INFO:tasks.workunit.client.0.smithi047.stdout: "data_stored": 0, 2024-09-18T14:08:12.735 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed": 0, 2024-09-18T14:08:12.735 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_allocated": 0, 2024-09-18T14:08:12.735 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_original": 0, 2024-09-18T14:08:12.735 INFO:tasks.workunit.client.0.smithi047.stdout: "omap_allocated": 0, 2024-09-18T14:08:12.735 INFO:tasks.workunit.client.0.smithi047.stdout: "internal_metadata": 0 2024-09-18T14:08:12.735 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.735 INFO:tasks.workunit.client.0.smithi047.stdout: "log_size": 468, 2024-09-18T14:08:12.735 INFO:tasks.workunit.client.0.smithi047.stdout: "ondisk_log_size": 468, 2024-09-18T14:08:12.735 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 64, 2024-09-18T14:08:12.735 INFO:tasks.workunit.client.0.smithi047.stdout: "acting": 64, 2024-09-18T14:08:12.735 INFO:tasks.workunit.client.0.smithi047.stdout: "num_store_stats": 0 2024-09-18T14:08:12.735 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.736 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_sum": { 2024-09-18T14:08:12.736 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 0, 2024-09-18T14:08:12.736 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 0, 2024-09-18T14:08:12.736 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pgs": 96, 2024-09-18T14:08:12.736 INFO:tasks.workunit.client.0.smithi047.stdout: "num_osds": 3, 2024-09-18T14:08:12.736 INFO:tasks.workunit.client.0.smithi047.stdout: "num_per_pool_osds": 3, 2024-09-18T14:08:12.736 INFO:tasks.workunit.client.0.smithi047.stdout: "num_per_pool_omap_osds": 3, 2024-09-18T14:08:12.736 INFO:tasks.workunit.client.0.smithi047.stdout: "kb": 314572800, 2024-09-18T14:08:12.736 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used": 895620, 2024-09-18T14:08:12.736 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_data": 2688, 2024-09-18T14:08:12.736 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_omap": 10, 2024-09-18T14:08:12.736 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_meta": 892853, 2024-09-18T14:08:12.736 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_avail": 313677180, 2024-09-18T14:08:12.736 INFO:tasks.workunit.client.0.smithi047.stdout: "statfs": { 2024-09-18T14:08:12.737 INFO:tasks.workunit.client.0.smithi047.stdout: "total": 322122547200, 2024-09-18T14:08:12.737 INFO:tasks.workunit.client.0.smithi047.stdout: "available": 321205432320, 2024-09-18T14:08:12.737 INFO:tasks.workunit.client.0.smithi047.stdout: "internally_reserved": 0, 2024-09-18T14:08:12.737 INFO:tasks.workunit.client.0.smithi047.stdout: "allocated": 2752512, 2024-09-18T14:08:12.737 INFO:tasks.workunit.client.0.smithi047.stdout: "data_stored": 641813, 2024-09-18T14:08:12.737 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed": 0, 2024-09-18T14:08:12.737 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_allocated": 0, 2024-09-18T14:08:12.737 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_original": 0, 2024-09-18T14:08:12.737 INFO:tasks.workunit.client.0.smithi047.stdout: "omap_allocated": 10790, 2024-09-18T14:08:12.737 INFO:tasks.workunit.client.0.smithi047.stdout: "internal_metadata": 914281946 2024-09-18T14:08:12.737 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.737 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_peers": [], 2024-09-18T14:08:12.737 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_trim_queue_len": 0, 2024-09-18T14:08:12.737 INFO:tasks.workunit.client.0.smithi047.stdout: "num_snap_trimming": 0, 2024-09-18T14:08:12.737 INFO:tasks.workunit.client.0.smithi047.stdout: "num_shards_repaired": 0, 2024-09-18T14:08:12.738 INFO:tasks.workunit.client.0.smithi047.stdout: "op_queue_age_hist": { 2024-09-18T14:08:12.738 INFO:tasks.workunit.client.0.smithi047.stdout: "histogram": [], 2024-09-18T14:08:12.738 INFO:tasks.workunit.client.0.smithi047.stdout: "upper_bound": 1 2024-09-18T14:08:12.738 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.738 INFO:tasks.workunit.client.0.smithi047.stdout: "perf_stat": { 2024-09-18T14:08:12.738 INFO:tasks.workunit.client.0.smithi047.stdout: "commit_latency_ms": 140, 2024-09-18T14:08:12.738 INFO:tasks.workunit.client.0.smithi047.stdout: "apply_latency_ms": 140, 2024-09-18T14:08:12.738 INFO:tasks.workunit.client.0.smithi047.stdout: "commit_latency_ns": 140000000, 2024-09-18T14:08:12.738 INFO:tasks.workunit.client.0.smithi047.stdout: "apply_latency_ns": 140000000 2024-09-18T14:08:12.738 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.738 INFO:tasks.workunit.client.0.smithi047.stdout: "alerts": [], 2024-09-18T14:08:12.738 INFO:tasks.workunit.client.0.smithi047.stdout: "network_ping_times": [] 2024-09-18T14:08:12.738 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.738 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_sum_by_class": { 2024-09-18T14:08:12.738 INFO:tasks.workunit.client.0.smithi047.stdout: "hdd": { 2024-09-18T14:08:12.739 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 0, 2024-09-18T14:08:12.739 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 0, 2024-09-18T14:08:12.739 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pgs": 96, 2024-09-18T14:08:12.739 INFO:tasks.workunit.client.0.smithi047.stdout: "num_osds": 3, 2024-09-18T14:08:12.739 INFO:tasks.workunit.client.0.smithi047.stdout: "num_per_pool_osds": 3, 2024-09-18T14:08:12.739 INFO:tasks.workunit.client.0.smithi047.stdout: "num_per_pool_omap_osds": 3, 2024-09-18T14:08:12.739 INFO:tasks.workunit.client.0.smithi047.stdout: "kb": 314572800, 2024-09-18T14:08:12.739 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used": 895620, 2024-09-18T14:08:12.739 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_data": 2688, 2024-09-18T14:08:12.739 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_omap": 10, 2024-09-18T14:08:12.739 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_meta": 892853, 2024-09-18T14:08:12.739 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_avail": 313677180, 2024-09-18T14:08:12.739 INFO:tasks.workunit.client.0.smithi047.stdout: "statfs": { 2024-09-18T14:08:12.739 INFO:tasks.workunit.client.0.smithi047.stdout: "total": 322122547200, 2024-09-18T14:08:12.739 INFO:tasks.workunit.client.0.smithi047.stdout: "available": 321205432320, 2024-09-18T14:08:12.740 INFO:tasks.workunit.client.0.smithi047.stdout: "internally_reserved": 0, 2024-09-18T14:08:12.740 INFO:tasks.workunit.client.0.smithi047.stdout: "allocated": 2752512, 2024-09-18T14:08:12.740 INFO:tasks.workunit.client.0.smithi047.stdout: "data_stored": 641813, 2024-09-18T14:08:12.740 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed": 0, 2024-09-18T14:08:12.740 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_allocated": 0, 2024-09-18T14:08:12.740 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_original": 0, 2024-09-18T14:08:12.740 INFO:tasks.workunit.client.0.smithi047.stdout: "omap_allocated": 10790, 2024-09-18T14:08:12.740 INFO:tasks.workunit.client.0.smithi047.stdout: "internal_metadata": 914281946 2024-09-18T14:08:12.740 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.740 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_peers": [], 2024-09-18T14:08:12.740 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_trim_queue_len": 0, 2024-09-18T14:08:12.740 INFO:tasks.workunit.client.0.smithi047.stdout: "num_snap_trimming": 0, 2024-09-18T14:08:12.740 INFO:tasks.workunit.client.0.smithi047.stdout: "num_shards_repaired": 0, 2024-09-18T14:08:12.740 INFO:tasks.workunit.client.0.smithi047.stdout: "op_queue_age_hist": { 2024-09-18T14:08:12.741 INFO:tasks.workunit.client.0.smithi047.stdout: "histogram": [], 2024-09-18T14:08:12.741 INFO:tasks.workunit.client.0.smithi047.stdout: "upper_bound": 1 2024-09-18T14:08:12.741 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.741 INFO:tasks.workunit.client.0.smithi047.stdout: "perf_stat": { 2024-09-18T14:08:12.741 INFO:tasks.workunit.client.0.smithi047.stdout: "commit_latency_ms": 140, 2024-09-18T14:08:12.741 INFO:tasks.workunit.client.0.smithi047.stdout: "apply_latency_ms": 140, 2024-09-18T14:08:12.741 INFO:tasks.workunit.client.0.smithi047.stdout: "commit_latency_ns": 140000000, 2024-09-18T14:08:12.741 INFO:tasks.workunit.client.0.smithi047.stdout: "apply_latency_ns": 140000000 2024-09-18T14:08:12.741 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.741 INFO:tasks.workunit.client.0.smithi047.stdout: "alerts": [], 2024-09-18T14:08:12.741 INFO:tasks.workunit.client.0.smithi047.stdout: "network_ping_times": [] 2024-09-18T14:08:12.741 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.741 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.741 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_stats": [ 2024-09-18T14:08:12.742 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.742 INFO:tasks.workunit.client.0.smithi047.stdout: "poolid": 1, 2024-09-18T14:08:12.742 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg": 32, 2024-09-18T14:08:12.742 INFO:tasks.workunit.client.0.smithi047.stdout: "stat_sum": { 2024-09-18T14:08:12.742 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes": 48059, 2024-09-18T14:08:12.742 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects": 100, 2024-09-18T14:08:12.742 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_clones": 0, 2024-09-18T14:08:12.742 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_copies": 300, 2024-09-18T14:08:12.742 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing_on_primary": 0, 2024-09-18T14:08:12.742 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing": 0, 2024-09-18T14:08:12.742 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_degraded": 100, 2024-09-18T14:08:12.742 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_misplaced": 0, 2024-09-18T14:08:12.742 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_unfound": 0, 2024-09-18T14:08:12.742 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_dirty": 100, 2024-09-18T14:08:12.743 INFO:tasks.workunit.client.0.smithi047.stdout: "num_whiteouts": 0, 2024-09-18T14:08:12.743 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read": 0, 2024-09-18T14:08:12.743 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read_kb": 0, 2024-09-18T14:08:12.743 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write": 81, 2024-09-18T14:08:12.743 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write_kb": 81, 2024-09-18T14:08:12.743 INFO:tasks.workunit.client.0.smithi047.stdout: "num_scrub_errors": 0, 2024-09-18T14:08:12.743 INFO:tasks.workunit.client.0.smithi047.stdout: "num_shallow_scrub_errors": 0, 2024-09-18T14:08:12.743 INFO:tasks.workunit.client.0.smithi047.stdout: "num_deep_scrub_errors": 0, 2024-09-18T14:08:12.743 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_recovered": 0, 2024-09-18T14:08:12.743 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_recovered": 0, 2024-09-18T14:08:12.743 INFO:tasks.workunit.client.0.smithi047.stdout: "num_keys_recovered": 0, 2024-09-18T14:08:12.743 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_omap": 0, 2024-09-18T14:08:12.743 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_hit_set_archive": 0, 2024-09-18T14:08:12.743 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_hit_set_archive": 0, 2024-09-18T14:08:12.743 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush": 0, 2024-09-18T14:08:12.744 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_kb": 0, 2024-09-18T14:08:12.744 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict": 0, 2024-09-18T14:08:12.744 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_kb": 0, 2024-09-18T14:08:12.744 INFO:tasks.workunit.client.0.smithi047.stdout: "num_promote": 0, 2024-09-18T14:08:12.744 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_high": 0, 2024-09-18T14:08:12.744 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_low": 0, 2024-09-18T14:08:12.744 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_some": 0, 2024-09-18T14:08:12.744 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_full": 0, 2024-09-18T14:08:12.744 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_pinned": 0, 2024-09-18T14:08:12.745 INFO:tasks.workunit.client.0.smithi047.stdout: "num_legacy_snapsets": 0, 2024-09-18T14:08:12.745 INFO:tasks.workunit.client.0.smithi047.stdout: "num_large_omap_objects": 0, 2024-09-18T14:08:12.745 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_manifest": 0, 2024-09-18T14:08:12.745 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_bytes": 0, 2024-09-18T14:08:12.745 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_keys": 0, 2024-09-18T14:08:12.745 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_repaired": 0 2024-09-18T14:08:12.745 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.745 INFO:tasks.workunit.client.0.smithi047.stdout: "store_stats": { 2024-09-18T14:08:12.745 INFO:tasks.workunit.client.0.smithi047.stdout: "total": 0, 2024-09-18T14:08:12.745 INFO:tasks.workunit.client.0.smithi047.stdout: "available": 0, 2024-09-18T14:08:12.745 INFO:tasks.workunit.client.0.smithi047.stdout: "internally_reserved": 0, 2024-09-18T14:08:12.745 INFO:tasks.workunit.client.0.smithi047.stdout: "allocated": 1220608, 2024-09-18T14:08:12.745 INFO:tasks.workunit.client.0.smithi047.stdout: "data_stored": 143276, 2024-09-18T14:08:12.745 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed": 0, 2024-09-18T14:08:12.745 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_allocated": 0, 2024-09-18T14:08:12.746 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_original": 0, 2024-09-18T14:08:12.746 INFO:tasks.workunit.client.0.smithi047.stdout: "omap_allocated": 0, 2024-09-18T14:08:12.746 INFO:tasks.workunit.client.0.smithi047.stdout: "internal_metadata": 0 2024-09-18T14:08:12.746 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.746 INFO:tasks.workunit.client.0.smithi047.stdout: "log_size": 468, 2024-09-18T14:08:12.746 INFO:tasks.workunit.client.0.smithi047.stdout: "ondisk_log_size": 468, 2024-09-18T14:08:12.746 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 64, 2024-09-18T14:08:12.746 INFO:tasks.workunit.client.0.smithi047.stdout: "acting": 64, 2024-09-18T14:08:12.746 INFO:tasks.workunit.client.0.smithi047.stdout: "num_store_stats": 3 2024-09-18T14:08:12.746 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.746 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:12.746 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_stats": [ 2024-09-18T14:08:12.746 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.746 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 0, 2024-09-18T14:08:12.746 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 244813135883 2024-09-18T14:08:12.746 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.747 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.747 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 1, 2024-09-18T14:08:12.747 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 42949672989 2024-09-18T14:08:12.747 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.747 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.747 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 2, 2024-09-18T14:08:12.747 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 261993005066 2024-09-18T14:08:12.747 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.747 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:12.747 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg_by_state": [ 2024-09-18T14:08:12.747 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.747 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "active+undersized+degraded", 2024-09-18T14:08:12.747 INFO:tasks.workunit.client.0.smithi047.stdout: "num": 32 2024-09-18T14:08:12.747 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.748 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:12.748 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg_by_osd": [ 2024-09-18T14:08:12.748 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.748 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 2, 2024-09-18T14:08:12.748 INFO:tasks.workunit.client.0.smithi047.stdout: "num_primary_pg": 19, 2024-09-18T14:08:12.748 INFO:tasks.workunit.client.0.smithi047.stdout: "num_acting_pg": 32, 2024-09-18T14:08:12.748 INFO:tasks.workunit.client.0.smithi047.stdout: "num_up_not_acting_pg": 0 2024-09-18T14:08:12.748 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.748 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.748 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 0, 2024-09-18T14:08:12.748 INFO:tasks.workunit.client.0.smithi047.stdout: "num_primary_pg": 13, 2024-09-18T14:08:12.748 INFO:tasks.workunit.client.0.smithi047.stdout: "num_acting_pg": 32, 2024-09-18T14:08:12.748 INFO:tasks.workunit.client.0.smithi047.stdout: "num_up_not_acting_pg": 0 2024-09-18T14:08:12.748 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.748 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.749 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 1, 2024-09-18T14:08:12.749 INFO:tasks.workunit.client.0.smithi047.stdout: "num_primary_pg": 0, 2024-09-18T14:08:12.749 INFO:tasks.workunit.client.0.smithi047.stdout: "num_acting_pg": 0, 2024-09-18T14:08:12.749 INFO:tasks.workunit.client.0.smithi047.stdout: "num_up_not_acting_pg": 0 2024-09-18T14:08:12.749 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.749 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:12.749 INFO:tasks.workunit.client.0.smithi047.stdout: "purged_snaps": [ 2024-09-18T14:08:12.749 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:12.749 INFO:tasks.workunit.client.0.smithi047.stdout: "pool": 1, 2024-09-18T14:08:12.749 INFO:tasks.workunit.client.0.smithi047.stdout: "purged_snaps": {} 2024-09-18T14:08:12.749 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.749 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:12.749 INFO:tasks.workunit.client.0.smithi047.stdout: "servicemap": { 2024-09-18T14:08:12.749 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 5, 2024-09-18T14:08:12.749 INFO:tasks.workunit.client.0.smithi047.stdout: "modified": "2024-09-18T14:07:24.609344+0000", 2024-09-18T14:08:12.750 INFO:tasks.workunit.client.0.smithi047.stdout: "services": {} 2024-09-18T14:08:12.750 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:12.750 INFO:tasks.workunit.client.0.smithi047.stdout: "paxos": { 2024-09-18T14:08:12.750 INFO:tasks.workunit.client.0.smithi047.stdout: "first_committed": 1, 2024-09-18T14:08:12.750 INFO:tasks.workunit.client.0.smithi047.stdout: "last_committed": 574, 2024-09-18T14:08:12.750 INFO:tasks.workunit.client.0.smithi047.stdout: "last_pn": 0, 2024-09-18T14:08:12.750 INFO:tasks.workunit.client.0.smithi047.stdout: "accepted_pn": 0 2024-09-18T14:08:12.750 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:12.750 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T14:08:12.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-18T14:08:12.751 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:753: TEST_divergent_3: expr 2 + 2 2024-09-18T14:08:12.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:753: TEST_divergent_3: objname=existing_4 2024-09-18T14:08:12.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:754: TEST_divergent_3: echo 'writing non-divergent object existing_4' 2024-09-18T14:08:12.751 INFO:tasks.workunit.client.0.smithi047.stdout:writing non-divergent object existing_4 2024-09-18T14:08:12.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:755: TEST_divergent_3: ceph pg dump pgs 2024-09-18T14:08:13.037 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:08:13.037 INFO:tasks.workunit.client.0.smithi047.stdout:1.1f 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814303+0000 32'80 70:119 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:13.037 INFO:tasks.workunit.client.0.smithi047.stdout:1.1e 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.814587+0000 34'81 70:118 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:13.037 INFO:tasks.workunit.client.0.smithi047.stdout:1.1d 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814571+0000 32'80 70:119 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:13.037 INFO:tasks.workunit.client.0.smithi047.stdout:1.b 4 0 4 0 0 1691 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.787168+0000 34'81 70:131 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:13.037 INFO:tasks.workunit.client.0.smithi047.stdout:1.a 3 0 3 0 0 1208 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.813808+0000 32'80 70:105 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:13.037 INFO:tasks.workunit.client.0.smithi047.stdout:1.9 5 0 5 0 0 2174 0 0 16 66 16 active+undersized+degraded 2024-09-18T14:07:51.812741+0000 34'82 70:131 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:13.037 INFO:tasks.workunit.client.0.smithi047.stdout:1.8 5 0 5 0 0 2174 0 0 16 66 16 active+undersized+degraded 2024-09-18T14:07:51.814091+0000 34'82 70:119 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:13.037 INFO:tasks.workunit.client.0.smithi047.stdout:1.7 3 0 3 0 0 1208 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.812691+0000 32'80 70:119 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:13.037 INFO:tasks.workunit.client.0.smithi047.stdout:1.6 3 0 3 0 0 1208 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814625+0000 32'80 70:129 [0,2] 0 [0,2] 0 32'80 2024-09-18T14:06:58.202394+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:13.037 INFO:tasks.workunit.client.0.smithi047.stdout:1.5 6 0 6 0 0 2898 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.814179+0000 34'81 70:140 [2,0] 2 [2,0] 2 34'81 2024-09-18T14:07:06.420533+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T15:28:19.740089+0000 6 0 2024-09-18T14:08:13.038 INFO:tasks.workunit.client.0.smithi047.stdout:1.4 3 0 3 0 0 1208 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.813528+0000 32'80 70:117 [0,2] 0 [0,2] 0 32'80 2024-09-18T14:07:08.170467+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:13.038 INFO:tasks.workunit.client.0.smithi047.stdout:1.3 4 0 4 0 0 1691 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.815161+0000 34'81 70:132 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:13.038 INFO:tasks.workunit.client.0.smithi047.stdout:1.2 3 0 3 0 0 1208 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814414+0000 32'80 70:129 [0,2] 0 [0,2] 0 32'80 2024-09-18T14:07:04.230275+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:13.038 INFO:tasks.workunit.client.0.smithi047.stdout:1.c 4 0 4 0 0 1691 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.812888+0000 34'81 70:130 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:13.038 INFO:tasks.workunit.client.0.smithi047.stdout:1.1 4 0 4 0 0 1691 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.813144+0000 34'81 70:132 [2,0] 2 [2,0] 2 34'81 2024-09-18T14:06:56.389936+0000 34'81 2024-09-18T14:06:56.389936+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:13.038 INFO:tasks.workunit.client.0.smithi047.stdout:1.1c 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.812871+0000 32'80 70:117 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:13.038 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 3 0 3 0 0 1449 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.815935+0000 34'81 70:141 [0,2] 0 [0,2] 0 32'80 2024-09-18T14:06:54.303691+0000 32'80 2024-09-18T14:06:54.303691+0000 0 1 periodic scrub scheduled @ 2024-09-19T21:42:35.244281+0000 2 0 2024-09-18T14:08:13.038 INFO:tasks.workunit.client.0.smithi047.stdout:1.d 3 0 3 0 0 1449 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.814765+0000 34'81 70:143 [2,0] 2 [2,0] 2 34'81 2024-09-18T14:07:09.554363+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T21:30:11.231921+0000 3 0 2024-09-18T14:08:13.038 INFO:tasks.workunit.client.0.smithi047.stdout:1.e 5 0 5 0 0 2174 0 0 16 66 16 active+undersized+degraded 2024-09-18T14:07:51.815577+0000 34'82 70:119 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:13.038 INFO:tasks.workunit.client.0.smithi047.stdout:1.f 4 0 4 0 0 1932 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814986+0000 32'80 70:129 [2,0] 2 [2,0] 2 32'80 2024-09-18T14:07:00.547567+0000 32'80 2024-09-18T14:07:00.547567+0000 0 1 periodic scrub scheduled @ 2024-09-19T22:57:43.008803+0000 4 0 2024-09-18T14:08:13.038 INFO:tasks.workunit.client.0.smithi047.stdout:1.10 4 0 4 0 0 1691 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.813572+0000 34'81 70:130 [0,2] 0 [0,2] 0 34'81 2024-09-18T14:07:07.204718+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:13.038 INFO:tasks.workunit.client.0.smithi047.stdout:1.11 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:52.902633+0000 32'80 70:117 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:13.038 INFO:tasks.workunit.client.0.smithi047.stdout:1.12 1 0 1 0 0 483 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.812635+0000 32'80 70:137 [0,2] 0 [0,2] 0 32'80 2024-09-18T14:07:08.487332+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T19:22:31.668808+0000 1 0 2024-09-18T14:08:13.039 INFO:tasks.workunit.client.0.smithi047.stdout:1.13 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.813086+0000 34'81 70:132 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:13.039 INFO:tasks.workunit.client.0.smithi047.stdout:1.14 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.816330+0000 34'81 70:130 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:13.039 INFO:tasks.workunit.client.0.smithi047.stdout:1.15 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.815134+0000 32'80 70:119 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:13.039 INFO:tasks.workunit.client.0.smithi047.stdout:1.16 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814810+0000 32'80 70:129 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:13.039 INFO:tasks.workunit.client.0.smithi047.stdout:1.17 1 0 1 0 0 483 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.813569+0000 32'80 70:129 [2,0] 2 [2,0] 2 32'80 2024-09-18T14:07:05.418931+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T19:59:06.890013+0000 1 0 2024-09-18T14:08:13.039 INFO:tasks.workunit.client.0.smithi047.stdout:1.18 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.814107+0000 34'81 70:118 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:13.039 INFO:tasks.workunit.client.0.smithi047.stdout:1.19 2 0 2 0 0 966 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.813376+0000 32'80 70:127 [2,0] 2 [2,0] 2 32'80 2024-09-18T14:07:07.546410+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T20:01:59.930990+0000 2 0 2024-09-18T14:08:13.039 INFO:tasks.workunit.client.0.smithi047.stdout:1.1a 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.812774+0000 34'81 70:118 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:13.039 INFO:tasks.workunit.client.0.smithi047.stdout:1.1b 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.813412+0000 34'81 70:131 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:13.039 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:08:13.039 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:08:13.039 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:08:13.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:756: TEST_divergent_3: rados -p test put existing_4 /etc/resolv.conf 2024-09-18T14:08:13.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-18T14:08:13.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: wait_for_clean 2024-09-18T14:08:13.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:08:13.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:08:13.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:08:13.169 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-18T14:08:13.169 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:08:13.169 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:08:13.170 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:08:13.170 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:08:13.170 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:08:13.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-18T14:08:13.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:08:13.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:08:13.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:08:13.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:08:13.228 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:08:13.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:08:13.598 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T14:08:13.598 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T14:08:13.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:08:13.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:08:13.598 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:08:13.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=244813135885 2024-09-18T14:08:13.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 244813135885 2024-09-18T14:08:13.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-244813135885' 2024-09-18T14:08:13.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:08:13.715 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:08:13.799 INFO:tasks.workunit.client.0.smithi047.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-18T14:08:13.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq= 2024-09-18T14:08:13.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z '' 2024-09-18T14:08:13.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: flush_pg_stats: continue 2024-09-18T14:08:13.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:08:13.801 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:08:13.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=261993005069 2024-09-18T14:08:13.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 261993005069 2024-09-18T14:08:13.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-244813135885 2-261993005069' 2024-09-18T14:08:13.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:08:13.918 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-244813135885 2024-09-18T14:08:13.918 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:08:13.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:08:13.919 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-244813135885 2024-09-18T14:08:13.919 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:08:13.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=244813135885 2024-09-18T14:08:13.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 244813135885' 2024-09-18T14:08:13.921 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 244813135885 2024-09-18T14:08:13.921 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:08:14.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 244813135883 -lt 244813135885 2024-09-18T14:08:14.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:08:15.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:08:15.366 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:08:15.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 244813135886 -lt 244813135885 2024-09-18T14:08:15.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:08:15.752 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-261993005069 2024-09-18T14:08:15.752 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:08:15.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:08:15.753 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-261993005069 2024-09-18T14:08:15.753 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:08:15.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=261993005069 2024-09-18T14:08:15.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 261993005069' 2024-09-18T14:08:15.755 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 261993005069 2024-09-18T14:08:15.755 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:08:16.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 261993005069 -lt 261993005069 2024-09-18T14:08:16.159 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:08:16.159 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:08:16.159 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:08:16.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-09-18T14:08:16.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:08:16.660 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:08:16.660 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:08:16.660 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:08:16.660 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:08:16.661 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:08:16.661 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:08:17.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:08:17.042 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:08:17.043 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:08:17.043 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:08:17.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-18T14:08:17.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-18T14:08:17.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T14:08:17.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-18T14:08:17.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:08:17.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:08:17.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:08:17.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:08:17.656 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:08:17.656 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:08:17.656 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:08:17.656 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:08:17.656 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:08:17.656 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:08:18.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:08:18.026 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:08:18.026 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:08:18.026 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:08:18.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-18T14:08:18.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:08:18.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:08:18.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:08:18.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:08:18.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:08:18.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:08:18.532 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:08:18.532 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:08:19.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:08:19.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:08:19.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-09-18T14:08:19.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:08:19.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-18T14:08:19.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:08:19.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:08:19.267 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:08:19.267 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:08:19.267 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:08:19.267 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:08:19.267 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:08:19.267 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:08:19.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:08:19.634 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:08:19.635 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:08:19.635 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:08:20.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-18T14:08:20.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:08:20.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:08:20.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:08:20.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:08:20.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:08:20.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:08:20.139 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:08:20.139 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:08:20.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:08:20.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:08:20.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-09-18T14:08:20.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:08:20.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-18T14:08:21.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:08:21.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:08:21.101 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:08:21.101 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:08:21.101 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:08:21.101 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:08:21.101 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:08:21.101 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:08:21.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:08:21.431 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:08:21.431 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:08:21.431 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:08:21.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-18T14:08:21.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:08:21.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:08:21.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:08:21.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:08:21.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:08:21.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:08:21.937 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:08:21.937 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:08:22.469 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:08:22.469 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:08:22.469 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-09-18T14:08:22.469 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:08:22.469 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-09-18T14:08:23.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:08:23.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:08:23.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:08:23.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:08:23.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:08:23.273 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:08:23.273 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:08:23.273 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:08:23.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:08:23.655 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:08:23.655 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:08:23.655 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:08:24.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-18T14:08:24.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:08:24.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:08:24.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:08:24.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:08:24.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:08:24.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:08:24.166 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:08:24.166 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:08:24.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:08:24.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:08:24.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-09-18T14:08:24.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:08:24.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-09-18T14:08:26.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:08:26.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:08:26.271 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:08:26.271 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:08:26.271 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:08:26.271 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:08:26.271 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:08:26.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:08:26.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:08:26.650 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:08:26.650 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:08:26.650 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:08:27.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-18T14:08:27.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:08:27.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:08:27.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:08:27.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:08:27.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:08:27.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:08:27.150 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:08:27.150 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:08:27.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:08:27.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:08:27.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-09-18T14:08:27.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:08:27.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-09-18T14:08:30.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:08:30.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:08:30.875 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:08:30.875 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:08:30.875 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:08:30.875 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:08:30.876 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:08:30.876 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:08:31.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:08:31.254 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:08:31.254 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:08:31.254 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:08:31.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-18T14:08:31.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:08:31.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:08:31.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:08:31.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:08:31.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:08:31.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:08:31.759 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:08:31.759 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:08:32.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:08:32.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:08:32.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-09-18T14:08:32.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:08:32.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-09-18T14:08:38.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:08:38.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:08:38.702 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:08:38.702 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:08:38.702 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:08:38.703 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:08:38.703 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:08:38.703 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:08:39.079 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:08:39.079 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:08:39.079 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:08:39.079 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:08:39.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-18T14:08:39.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:08:39.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:08:39.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:08:39.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:08:39.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:08:39.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:08:39.585 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:08:39.585 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:08:40.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:08:40.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:08:40.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-09-18T14:08:40.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:08:40.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-09-18T14:08:47.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:08:47.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:08:47.427 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:08:47.427 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:08:47.427 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:08:47.427 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:08:47.427 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:08:47.427 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:08:47.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:08:47.807 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:08:47.807 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:08:47.807 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:08:48.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-18T14:08:48.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:08:48.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:08:48.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:08:48.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:08:48.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:08:48.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:08:48.307 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:08:48.307 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:08:48.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:08:48.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:08:48.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-09-18T14:08:48.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-18T14:08:49.327 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T14:08:49.328 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_fingerprint": "295f4427-4a7c-4940-a333-e06d3d18f172", 2024-09-18T14:08:49.328 INFO:tasks.workunit.client.0.smithi047.stdout: "version": "17.2.7-1645-g7061ea6c", 2024-09-18T14:08:49.328 INFO:tasks.workunit.client.0.smithi047.stdout: "commit": "7061ea6ca50908d104e9f3979dc615e118d9e153", 2024-09-18T14:08:49.328 INFO:tasks.workunit.client.0.smithi047.stdout: "timestamp": "2024-09-18T14:08:49.327120+0000", 2024-09-18T14:08:49.328 INFO:tasks.workunit.client.0.smithi047.stdout: "tag": "", 2024-09-18T14:08:49.328 INFO:tasks.workunit.client.0.smithi047.stdout: "health": { 2024-09-18T14:08:49.328 INFO:tasks.workunit.client.0.smithi047.stdout: "status": "HEALTH_WARN", 2024-09-18T14:08:49.328 INFO:tasks.workunit.client.0.smithi047.stdout: "checks": { 2024-09-18T14:08:49.328 INFO:tasks.workunit.client.0.smithi047.stdout: "OSDMAP_FLAGS": { 2024-09-18T14:08:49.328 INFO:tasks.workunit.client.0.smithi047.stdout: "severity": "HEALTH_WARN", 2024-09-18T14:08:49.329 INFO:tasks.workunit.client.0.smithi047.stdout: "summary": { 2024-09-18T14:08:49.329 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-18T14:08:49.329 INFO:tasks.workunit.client.0.smithi047.stdout: "count": 17 2024-09-18T14:08:49.329 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.329 INFO:tasks.workunit.client.0.smithi047.stdout: "detail": [], 2024-09-18T14:08:49.329 INFO:tasks.workunit.client.0.smithi047.stdout: "muted": false 2024-09-18T14:08:49.329 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.329 INFO:tasks.workunit.client.0.smithi047.stdout: "OSD_DOWN": { 2024-09-18T14:08:49.329 INFO:tasks.workunit.client.0.smithi047.stdout: "severity": "HEALTH_WARN", 2024-09-18T14:08:49.329 INFO:tasks.workunit.client.0.smithi047.stdout: "summary": { 2024-09-18T14:08:49.329 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "1 osds down", 2024-09-18T14:08:49.329 INFO:tasks.workunit.client.0.smithi047.stdout: "count": 1 2024-09-18T14:08:49.329 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.330 INFO:tasks.workunit.client.0.smithi047.stdout: "detail": [ 2024-09-18T14:08:49.330 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.330 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "osd.1 (root=default,host=smithi047) is down" 2024-09-18T14:08:49.330 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.330 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:49.330 INFO:tasks.workunit.client.0.smithi047.stdout: "muted": false 2024-09-18T14:08:49.330 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.330 INFO:tasks.workunit.client.0.smithi047.stdout: "PG_DEGRADED": { 2024-09-18T14:08:49.330 INFO:tasks.workunit.client.0.smithi047.stdout: "severity": "HEALTH_WARN", 2024-09-18T14:08:49.330 INFO:tasks.workunit.client.0.smithi047.stdout: "summary": { 2024-09-18T14:08:49.330 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded, 32 pgs undersized", 2024-09-18T14:08:49.330 INFO:tasks.workunit.client.0.smithi047.stdout: "count": 64 2024-09-18T14:08:49.330 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.331 INFO:tasks.workunit.client.0.smithi047.stdout: "detail": [ 2024-09-18T14:08:49.331 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.331 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.0 is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2024-09-18T14:08:49.331 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.331 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.331 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.1 is stuck undersized for 71s, current state active+undersized+degraded, last acting [2,0]" 2024-09-18T14:08:49.331 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.331 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.331 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.2 is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2024-09-18T14:08:49.331 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.331 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.331 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.3 is stuck undersized for 71s, current state active+undersized+degraded, last acting [2,0]" 2024-09-18T14:08:49.331 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.331 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.332 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.4 is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2024-09-18T14:08:49.332 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.332 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.332 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.5 is stuck undersized for 71s, current state active+undersized+degraded, last acting [2,0]" 2024-09-18T14:08:49.332 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.332 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.332 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.6 is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2024-09-18T14:08:49.332 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.332 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.332 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.7 is stuck undersized for 71s, current state active+undersized+degraded, last acting [2,0]" 2024-09-18T14:08:49.332 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.332 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.332 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.8 is stuck undersized for 71s, current state active+undersized+degraded, last acting [2,0]" 2024-09-18T14:08:49.333 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.333 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.333 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.9 is stuck undersized for 71s, current state active+undersized+degraded, last acting [2,0]" 2024-09-18T14:08:49.333 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.333 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.333 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.a is stuck undersized for 71s, current state active+undersized+degraded, last acting [2,0]" 2024-09-18T14:08:49.333 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.333 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.333 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.b is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2024-09-18T14:08:49.333 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.333 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.333 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.c is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2024-09-18T14:08:49.334 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.334 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.334 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.d is stuck undersized for 71s, current state active+undersized+degraded, last acting [2,0]" 2024-09-18T14:08:49.334 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.334 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.334 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.e is stuck undersized for 71s, current state active+undersized+degraded, last acting [2,0]" 2024-09-18T14:08:49.334 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.334 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.334 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.f is stuck undersized for 71s, current state active+undersized+degraded, last acting [2,0]" 2024-09-18T14:08:49.334 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.334 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.334 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.10 is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2024-09-18T14:08:49.334 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.334 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.334 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.11 is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2024-09-18T14:08:49.335 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.335 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.335 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.12 is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2024-09-18T14:08:49.335 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.335 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.335 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.13 is stuck undersized for 71s, current state active+undersized+degraded, last acting [2,0]" 2024-09-18T14:08:49.335 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.335 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.335 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.14 is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2024-09-18T14:08:49.335 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.335 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.335 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.15 is stuck undersized for 71s, current state active+undersized+degraded, last acting [2,0]" 2024-09-18T14:08:49.335 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.335 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.335 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.16 is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2024-09-18T14:08:49.336 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.336 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.336 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.17 is stuck undersized for 71s, current state active+undersized+degraded, last acting [2,0]" 2024-09-18T14:08:49.336 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.336 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.336 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.18 is stuck undersized for 71s, current state active+undersized+degraded, last acting [2,0]" 2024-09-18T14:08:49.336 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.336 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.336 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.19 is stuck undersized for 71s, current state active+undersized+degraded, last acting [2,0]" 2024-09-18T14:08:49.336 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.336 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.337 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.1a is stuck undersized for 71s, current state active+undersized+degraded, last acting [2,0]" 2024-09-18T14:08:49.337 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.337 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.337 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.1b is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2024-09-18T14:08:49.337 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.337 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.337 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.1c is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2024-09-18T14:08:49.337 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.337 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.337 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.1d is stuck undersized for 71s, current state active+undersized+degraded, last acting [2,0]" 2024-09-18T14:08:49.337 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.337 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.337 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.1e is stuck undersized for 71s, current state active+undersized+degraded, last acting [2,0]" 2024-09-18T14:08:49.338 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.338 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.338 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.1f is stuck undersized for 71s, current state active+undersized+degraded, last acting [2,0]" 2024-09-18T14:08:49.338 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.338 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:49.338 INFO:tasks.workunit.client.0.smithi047.stdout: "muted": false 2024-09-18T14:08:49.338 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.338 INFO:tasks.workunit.client.0.smithi047.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-18T14:08:49.338 INFO:tasks.workunit.client.0.smithi047.stdout: "severity": "HEALTH_WARN", 2024-09-18T14:08:49.338 INFO:tasks.workunit.client.0.smithi047.stdout: "summary": { 2024-09-18T14:08:49.338 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-18T14:08:49.338 INFO:tasks.workunit.client.0.smithi047.stdout: "count": 1 2024-09-18T14:08:49.339 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.339 INFO:tasks.workunit.client.0.smithi047.stdout: "detail": [ 2024-09-18T14:08:49.339 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.339 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "application not enabled on pool 'test'" 2024-09-18T14:08:49.339 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.339 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.339 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-18T14:08:49.339 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.339 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:49.339 INFO:tasks.workunit.client.0.smithi047.stdout: "muted": false 2024-09-18T14:08:49.339 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.339 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.339 INFO:tasks.workunit.client.0.smithi047.stdout: "mutes": [] 2024-09-18T14:08:49.339 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.340 INFO:tasks.workunit.client.0.smithi047.stdout: "monmap_first_committed": 1, 2024-09-18T14:08:49.340 INFO:tasks.workunit.client.0.smithi047.stdout: "monmap_last_committed": 1, 2024-09-18T14:08:49.340 INFO:tasks.workunit.client.0.smithi047.stdout: "monmap": { 2024-09-18T14:08:49.340 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 1, 2024-09-18T14:08:49.340 INFO:tasks.workunit.client.0.smithi047.stdout: "fsid": "d7330bd7-8062-424a-bbd8-8c587903c456", 2024-09-18T14:08:49.340 INFO:tasks.workunit.client.0.smithi047.stdout: "modified": "2024-09-18T14:04:45.277080Z", 2024-09-18T14:08:49.340 INFO:tasks.workunit.client.0.smithi047.stdout: "created": "2024-09-18T14:04:45.277080Z", 2024-09-18T14:08:49.340 INFO:tasks.workunit.client.0.smithi047.stdout: "min_mon_release": 17, 2024-09-18T14:08:49.340 INFO:tasks.workunit.client.0.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-18T14:08:49.340 INFO:tasks.workunit.client.0.smithi047.stdout: "election_strategy": 1, 2024-09-18T14:08:49.340 INFO:tasks.workunit.client.0.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-18T14:08:49.340 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_mode": false, 2024-09-18T14:08:49.340 INFO:tasks.workunit.client.0.smithi047.stdout: "tiebreaker_mon": "", 2024-09-18T14:08:49.340 INFO:tasks.workunit.client.0.smithi047.stdout: "removed_ranks: ": "", 2024-09-18T14:08:49.341 INFO:tasks.workunit.client.0.smithi047.stdout: "features": { 2024-09-18T14:08:49.341 INFO:tasks.workunit.client.0.smithi047.stdout: "persistent": [ 2024-09-18T14:08:49.341 INFO:tasks.workunit.client.0.smithi047.stdout: "kraken", 2024-09-18T14:08:49.341 INFO:tasks.workunit.client.0.smithi047.stdout: "luminous", 2024-09-18T14:08:49.341 INFO:tasks.workunit.client.0.smithi047.stdout: "mimic", 2024-09-18T14:08:49.341 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap-prune", 2024-09-18T14:08:49.341 INFO:tasks.workunit.client.0.smithi047.stdout: "nautilus", 2024-09-18T14:08:49.341 INFO:tasks.workunit.client.0.smithi047.stdout: "octopus", 2024-09-18T14:08:49.341 INFO:tasks.workunit.client.0.smithi047.stdout: "pacific", 2024-09-18T14:08:49.341 INFO:tasks.workunit.client.0.smithi047.stdout: "elector-pinging", 2024-09-18T14:08:49.341 INFO:tasks.workunit.client.0.smithi047.stdout: "quincy" 2024-09-18T14:08:49.341 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:49.341 INFO:tasks.workunit.client.0.smithi047.stdout: "optional": [] 2024-09-18T14:08:49.341 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.342 INFO:tasks.workunit.client.0.smithi047.stdout: "mons": [ 2024-09-18T14:08:49.342 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.342 INFO:tasks.workunit.client.0.smithi047.stdout: "rank": 0, 2024-09-18T14:08:49.342 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "a", 2024-09-18T14:08:49.342 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addrs": { 2024-09-18T14:08:49.342 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:08:49.342 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.342 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:08:49.342 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:7115", 2024-09-18T14:08:49.342 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 0 2024-09-18T14:08:49.342 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.342 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:49.342 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.342 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:7115/0", 2024-09-18T14:08:49.342 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-18T14:08:49.343 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 0, 2024-09-18T14:08:49.343 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 0, 2024-09-18T14:08:49.343 INFO:tasks.workunit.client.0.smithi047.stdout: "crush_location": "{}" 2024-09-18T14:08:49.343 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.343 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:49.343 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.343 INFO:tasks.workunit.client.0.smithi047.stdout: "quorum": [ 2024-09-18T14:08:49.343 INFO:tasks.workunit.client.0.smithi047.stdout: 0 2024-09-18T14:08:49.343 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:49.343 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap": { 2024-09-18T14:08:49.343 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 70, 2024-09-18T14:08:49.343 INFO:tasks.workunit.client.0.smithi047.stdout: "fsid": "d7330bd7-8062-424a-bbd8-8c587903c456", 2024-09-18T14:08:49.343 INFO:tasks.workunit.client.0.smithi047.stdout: "created": "2024-09-18T14:04:46.122705+0000", 2024-09-18T14:08:49.343 INFO:tasks.workunit.client.0.smithi047.stdout: "modified": "2024-09-18T14:07:37.245559+0000", 2024-09-18T14:08:49.343 INFO:tasks.workunit.client.0.smithi047.stdout: "last_up_change": "2024-09-18T14:07:32.707559+0000", 2024-09-18T14:08:49.344 INFO:tasks.workunit.client.0.smithi047.stdout: "last_in_change": "2024-09-18T14:07:35.610542+0000", 2024-09-18T14:08:49.344 INFO:tasks.workunit.client.0.smithi047.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-18T14:08:49.344 INFO:tasks.workunit.client.0.smithi047.stdout: "flags_num": 5800384, 2024-09-18T14:08:49.344 INFO:tasks.workunit.client.0.smithi047.stdout: "flags_set": [ 2024-09-18T14:08:49.344 INFO:tasks.workunit.client.0.smithi047.stdout: "nodown", 2024-09-18T14:08:49.344 INFO:tasks.workunit.client.0.smithi047.stdout: "noin", 2024-09-18T14:08:49.344 INFO:tasks.workunit.client.0.smithi047.stdout: "noout", 2024-09-18T14:08:49.344 INFO:tasks.workunit.client.0.smithi047.stdout: "pglog_hardlimit", 2024-09-18T14:08:49.344 INFO:tasks.workunit.client.0.smithi047.stdout: "purged_snapdirs", 2024-09-18T14:08:49.344 INFO:tasks.workunit.client.0.smithi047.stdout: "recovery_deletes", 2024-09-18T14:08:49.344 INFO:tasks.workunit.client.0.smithi047.stdout: "sortbitwise" 2024-09-18T14:08:49.344 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:49.344 INFO:tasks.workunit.client.0.smithi047.stdout: "crush_version": 9, 2024-09-18T14:08:49.344 INFO:tasks.workunit.client.0.smithi047.stdout: "full_ratio": 0.99000000953674316, 2024-09-18T14:08:49.345 INFO:tasks.workunit.client.0.smithi047.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-18T14:08:49.345 INFO:tasks.workunit.client.0.smithi047.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-18T14:08:49.345 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_snapshot": "", 2024-09-18T14:08:49.345 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_max": 1, 2024-09-18T14:08:49.345 INFO:tasks.workunit.client.0.smithi047.stdout: "max_osd": 3, 2024-09-18T14:08:49.345 INFO:tasks.workunit.client.0.smithi047.stdout: "require_min_compat_client": "luminous", 2024-09-18T14:08:49.345 INFO:tasks.workunit.client.0.smithi047.stdout: "min_compat_client": "jewel", 2024-09-18T14:08:49.345 INFO:tasks.workunit.client.0.smithi047.stdout: "require_osd_release": "quincy", 2024-09-18T14:08:49.345 INFO:tasks.workunit.client.0.smithi047.stdout: "pools": [ 2024-09-18T14:08:49.345 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.345 INFO:tasks.workunit.client.0.smithi047.stdout: "pool": 1, 2024-09-18T14:08:49.345 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_name": "test", 2024-09-18T14:08:49.345 INFO:tasks.workunit.client.0.smithi047.stdout: "create_time": "2024-09-18T14:06:27.251699+0000", 2024-09-18T14:08:49.345 INFO:tasks.workunit.client.0.smithi047.stdout: "flags": 1, 2024-09-18T14:08:49.346 INFO:tasks.workunit.client.0.smithi047.stdout: "flags_names": "hashpspool", 2024-09-18T14:08:49.346 INFO:tasks.workunit.client.0.smithi047.stdout: "type": 1, 2024-09-18T14:08:49.346 INFO:tasks.workunit.client.0.smithi047.stdout: "size": 3, 2024-09-18T14:08:49.346 INFO:tasks.workunit.client.0.smithi047.stdout: "min_size": 2, 2024-09-18T14:08:49.346 INFO:tasks.workunit.client.0.smithi047.stdout: "crush_rule": 0, 2024-09-18T14:08:49.346 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_count": 0, 2024-09-18T14:08:49.346 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_target": 0, 2024-09-18T14:08:49.346 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T14:08:49.346 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T14:08:49.346 INFO:tasks.workunit.client.0.smithi047.stdout: "object_hash": 2, 2024-09-18T14:08:49.346 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_autoscale_mode": "on", 2024-09-18T14:08:49.346 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_num": 32, 2024-09-18T14:08:49.346 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_placement_num": 18, 2024-09-18T14:08:49.346 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_placement_num_target": 32, 2024-09-18T14:08:49.346 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_num_target": 32, 2024-09-18T14:08:49.347 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_num_pending": 32, 2024-09-18T14:08:49.347 INFO:tasks.workunit.client.0.smithi047.stdout: "last_pg_merge_meta": { 2024-09-18T14:08:49.347 INFO:tasks.workunit.client.0.smithi047.stdout: "source_pgid": "0.0", 2024-09-18T14:08:49.347 INFO:tasks.workunit.client.0.smithi047.stdout: "ready_epoch": 0, 2024-09-18T14:08:49.347 INFO:tasks.workunit.client.0.smithi047.stdout: "last_epoch_started": 0, 2024-09-18T14:08:49.347 INFO:tasks.workunit.client.0.smithi047.stdout: "last_epoch_clean": 0, 2024-09-18T14:08:49.347 INFO:tasks.workunit.client.0.smithi047.stdout: "source_version": "0'0", 2024-09-18T14:08:49.347 INFO:tasks.workunit.client.0.smithi047.stdout: "target_version": "0'0" 2024-09-18T14:08:49.347 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.347 INFO:tasks.workunit.client.0.smithi047.stdout: "last_change": "69", 2024-09-18T14:08:49.347 INFO:tasks.workunit.client.0.smithi047.stdout: "last_force_op_resend": "0", 2024-09-18T14:08:49.347 INFO:tasks.workunit.client.0.smithi047.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T14:08:49.347 INFO:tasks.workunit.client.0.smithi047.stdout: "last_force_op_resend_preluminous": "33", 2024-09-18T14:08:49.347 INFO:tasks.workunit.client.0.smithi047.stdout: "auid": 0, 2024-09-18T14:08:49.347 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_mode": "selfmanaged", 2024-09-18T14:08:49.348 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_seq": 0, 2024-09-18T14:08:49.348 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_epoch": 0, 2024-09-18T14:08:49.348 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_snaps": [], 2024-09-18T14:08:49.348 INFO:tasks.workunit.client.0.smithi047.stdout: "removed_snaps": "[]", 2024-09-18T14:08:49.348 INFO:tasks.workunit.client.0.smithi047.stdout: "quota_max_bytes": 0, 2024-09-18T14:08:49.348 INFO:tasks.workunit.client.0.smithi047.stdout: "quota_max_objects": 0, 2024-09-18T14:08:49.348 INFO:tasks.workunit.client.0.smithi047.stdout: "tiers": [], 2024-09-18T14:08:49.348 INFO:tasks.workunit.client.0.smithi047.stdout: "tier_of": -1, 2024-09-18T14:08:49.348 INFO:tasks.workunit.client.0.smithi047.stdout: "read_tier": -1, 2024-09-18T14:08:49.349 INFO:tasks.workunit.client.0.smithi047.stdout: "write_tier": -1, 2024-09-18T14:08:49.349 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_mode": "none", 2024-09-18T14:08:49.349 INFO:tasks.workunit.client.0.smithi047.stdout: "target_max_bytes": 0, 2024-09-18T14:08:49.349 INFO:tasks.workunit.client.0.smithi047.stdout: "target_max_objects": 0, 2024-09-18T14:08:49.349 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T14:08:49.349 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T14:08:49.349 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T14:08:49.349 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_min_flush_age": 0, 2024-09-18T14:08:49.349 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_min_evict_age": 0, 2024-09-18T14:08:49.349 INFO:tasks.workunit.client.0.smithi047.stdout: "erasure_code_profile": "", 2024-09-18T14:08:49.350 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_params": { 2024-09-18T14:08:49.350 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "none" 2024-09-18T14:08:49.350 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.350 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_period": 0, 2024-09-18T14:08:49.350 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_count": 0, 2024-09-18T14:08:49.350 INFO:tasks.workunit.client.0.smithi047.stdout: "use_gmt_hitset": true, 2024-09-18T14:08:49.350 INFO:tasks.workunit.client.0.smithi047.stdout: "min_read_recency_for_promote": 0, 2024-09-18T14:08:49.350 INFO:tasks.workunit.client.0.smithi047.stdout: "min_write_recency_for_promote": 0, 2024-09-18T14:08:49.350 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T14:08:49.350 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_search_last_n": 0, 2024-09-18T14:08:49.350 INFO:tasks.workunit.client.0.smithi047.stdout: "grade_table": [], 2024-09-18T14:08:49.350 INFO:tasks.workunit.client.0.smithi047.stdout: "stripe_width": 0, 2024-09-18T14:08:49.350 INFO:tasks.workunit.client.0.smithi047.stdout: "expected_num_objects": 0, 2024-09-18T14:08:49.350 INFO:tasks.workunit.client.0.smithi047.stdout: "fast_read": false, 2024-09-18T14:08:49.350 INFO:tasks.workunit.client.0.smithi047.stdout: "options": {}, 2024-09-18T14:08:49.351 INFO:tasks.workunit.client.0.smithi047.stdout: "application_metadata": {} 2024-09-18T14:08:49.351 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.351 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:49.351 INFO:tasks.workunit.client.0.smithi047.stdout: "osds": [ 2024-09-18T14:08:49.351 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.351 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 0, 2024-09-18T14:08:49.351 INFO:tasks.workunit.client.0.smithi047.stdout: "uuid": "2d333a34-fa14-4e6f-aca9-f14a57e36c76", 2024-09-18T14:08:49.351 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 1, 2024-09-18T14:08:49.351 INFO:tasks.workunit.client.0.smithi047.stdout: "in": 1, 2024-09-18T14:08:49.351 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 1, 2024-09-18T14:08:49.351 INFO:tasks.workunit.client.0.smithi047.stdout: "primary_affinity": 1, 2024-09-18T14:08:49.351 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_begin": 5, 2024-09-18T14:08:49.351 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_end": 40, 2024-09-18T14:08:49.351 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 57, 2024-09-18T14:08:49.351 INFO:tasks.workunit.client.0.smithi047.stdout: "up_thru": 69, 2024-09-18T14:08:49.352 INFO:tasks.workunit.client.0.smithi047.stdout: "down_at": 49, 2024-09-18T14:08:49.352 INFO:tasks.workunit.client.0.smithi047.stdout: "lost_at": 0, 2024-09-18T14:08:49.352 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addrs": { 2024-09-18T14:08:49.352 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:08:49.352 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.352 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:08:49.352 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6802", 2024-09-18T14:08:49.352 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 68576 2024-09-18T14:08:49.352 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.352 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.352 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:08:49.352 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6803", 2024-09-18T14:08:49.352 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 68576 2024-09-18T14:08:49.352 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.352 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:49.353 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.353 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addrs": { 2024-09-18T14:08:49.353 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:08:49.353 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.353 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:08:49.353 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6804", 2024-09-18T14:08:49.353 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 68576 2024-09-18T14:08:49.353 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.353 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.353 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:08:49.353 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6805", 2024-09-18T14:08:49.353 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 68576 2024-09-18T14:08:49.354 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.354 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:49.354 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.354 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addrs": { 2024-09-18T14:08:49.354 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:08:49.354 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.354 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:08:49.354 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6808", 2024-09-18T14:08:49.354 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 68576 2024-09-18T14:08:49.354 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.354 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.354 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:08:49.354 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6809", 2024-09-18T14:08:49.354 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 68576 2024-09-18T14:08:49.355 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.355 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:49.355 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.355 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addrs": { 2024-09-18T14:08:49.355 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:08:49.355 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.355 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:08:49.355 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6806", 2024-09-18T14:08:49.355 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 68576 2024-09-18T14:08:49.355 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.355 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.355 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:08:49.356 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6807", 2024-09-18T14:08:49.356 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 68576 2024-09-18T14:08:49.356 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.356 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:49.356 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.356 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addr": "127.0.0.1:6803/68576", 2024-09-18T14:08:49.356 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addr": "127.0.0.1:6805/68576", 2024-09-18T14:08:49.356 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addr": "127.0.0.1:6809/68576", 2024-09-18T14:08:49.356 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addr": "127.0.0.1:6807/68576", 2024-09-18T14:08:49.356 INFO:tasks.workunit.client.0.smithi047.stdout: "state": [ 2024-09-18T14:08:49.356 INFO:tasks.workunit.client.0.smithi047.stdout: "exists", 2024-09-18T14:08:49.356 INFO:tasks.workunit.client.0.smithi047.stdout: "up" 2024-09-18T14:08:49.356 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:49.356 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.356 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.357 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 1, 2024-09-18T14:08:49.357 INFO:tasks.workunit.client.0.smithi047.stdout: "uuid": "d16b04c1-1e01-47e1-87e5-e8058bd968e4", 2024-09-18T14:08:49.357 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 0, 2024-09-18T14:08:49.357 INFO:tasks.workunit.client.0.smithi047.stdout: "in": 1, 2024-09-18T14:08:49.357 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 1, 2024-09-18T14:08:49.357 INFO:tasks.workunit.client.0.smithi047.stdout: "primary_affinity": 1, 2024-09-18T14:08:49.357 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_begin": 0, 2024-09-18T14:08:49.357 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_end": 0, 2024-09-18T14:08:49.357 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 10, 2024-09-18T14:08:49.357 INFO:tasks.workunit.client.0.smithi047.stdout: "up_thru": 46, 2024-09-18T14:08:49.357 INFO:tasks.workunit.client.0.smithi047.stdout: "down_at": 50, 2024-09-18T14:08:49.357 INFO:tasks.workunit.client.0.smithi047.stdout: "lost_at": 0, 2024-09-18T14:08:49.358 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addrs": { 2024-09-18T14:08:49.358 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:08:49.358 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.358 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:08:49.358 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6810", 2024-09-18T14:08:49.358 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 62217 2024-09-18T14:08:49.358 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.358 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.358 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:08:49.358 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6811", 2024-09-18T14:08:49.358 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 62217 2024-09-18T14:08:49.358 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.358 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:49.359 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.359 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addrs": { 2024-09-18T14:08:49.359 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:08:49.359 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.359 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:08:49.359 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6812", 2024-09-18T14:08:49.359 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 62217 2024-09-18T14:08:49.359 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.359 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.359 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:08:49.359 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6813", 2024-09-18T14:08:49.359 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 62217 2024-09-18T14:08:49.359 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.359 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:49.360 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.360 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addrs": { 2024-09-18T14:08:49.360 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:08:49.360 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.360 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:08:49.360 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6816", 2024-09-18T14:08:49.360 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 62217 2024-09-18T14:08:49.360 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.360 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.360 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:08:49.360 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6817", 2024-09-18T14:08:49.360 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 62217 2024-09-18T14:08:49.360 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.360 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:49.361 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.361 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addrs": { 2024-09-18T14:08:49.361 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:08:49.361 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.361 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:08:49.361 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6814", 2024-09-18T14:08:49.361 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 62217 2024-09-18T14:08:49.361 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.361 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.361 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:08:49.361 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6815", 2024-09-18T14:08:49.362 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 62217 2024-09-18T14:08:49.362 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.362 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:49.362 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.362 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addr": "127.0.0.1:6811/62217", 2024-09-18T14:08:49.362 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addr": "127.0.0.1:6813/62217", 2024-09-18T14:08:49.362 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addr": "127.0.0.1:6817/62217", 2024-09-18T14:08:49.362 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addr": "127.0.0.1:6815/62217", 2024-09-18T14:08:49.362 INFO:tasks.workunit.client.0.smithi047.stdout: "state": [ 2024-09-18T14:08:49.362 INFO:tasks.workunit.client.0.smithi047.stdout: "exists" 2024-09-18T14:08:49.362 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:49.362 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.362 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.362 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 2, 2024-09-18T14:08:49.363 INFO:tasks.workunit.client.0.smithi047.stdout: "uuid": "20ba0a43-7c90-449e-bd64-6203deaebae3", 2024-09-18T14:08:49.363 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 1, 2024-09-18T14:08:49.363 INFO:tasks.workunit.client.0.smithi047.stdout: "in": 1, 2024-09-18T14:08:49.363 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 1, 2024-09-18T14:08:49.363 INFO:tasks.workunit.client.0.smithi047.stdout: "primary_affinity": 1, 2024-09-18T14:08:49.363 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_begin": 15, 2024-09-18T14:08:49.363 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_end": 40, 2024-09-18T14:08:49.363 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 61, 2024-09-18T14:08:49.363 INFO:tasks.workunit.client.0.smithi047.stdout: "up_thru": 67, 2024-09-18T14:08:49.363 INFO:tasks.workunit.client.0.smithi047.stdout: "down_at": 51, 2024-09-18T14:08:49.363 INFO:tasks.workunit.client.0.smithi047.stdout: "lost_at": 0, 2024-09-18T14:08:49.363 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addrs": { 2024-09-18T14:08:49.363 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:08:49.363 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.363 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:08:49.364 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6810", 2024-09-18T14:08:49.364 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 69154 2024-09-18T14:08:49.364 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.364 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.364 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:08:49.364 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6811", 2024-09-18T14:08:49.364 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 69154 2024-09-18T14:08:49.364 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.364 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:49.364 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.364 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addrs": { 2024-09-18T14:08:49.364 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:08:49.364 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.364 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:08:49.364 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6812", 2024-09-18T14:08:49.365 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 69154 2024-09-18T14:08:49.365 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.365 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.365 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:08:49.365 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6813", 2024-09-18T14:08:49.365 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 69154 2024-09-18T14:08:49.365 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.365 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:49.365 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.365 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addrs": { 2024-09-18T14:08:49.365 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:08:49.365 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.365 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:08:49.365 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6816", 2024-09-18T14:08:49.366 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 69154 2024-09-18T14:08:49.366 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.366 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.366 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:08:49.366 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6817", 2024-09-18T14:08:49.366 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 69154 2024-09-18T14:08:49.366 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.366 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:49.366 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.366 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addrs": { 2024-09-18T14:08:49.366 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:08:49.366 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.366 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:08:49.366 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6814", 2024-09-18T14:08:49.366 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 69154 2024-09-18T14:08:49.367 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.367 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.367 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:08:49.367 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6815", 2024-09-18T14:08:49.367 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 69154 2024-09-18T14:08:49.367 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.367 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:49.367 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.367 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addr": "127.0.0.1:6811/69154", 2024-09-18T14:08:49.367 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addr": "127.0.0.1:6813/69154", 2024-09-18T14:08:49.367 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addr": "127.0.0.1:6817/69154", 2024-09-18T14:08:49.368 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addr": "127.0.0.1:6815/69154", 2024-09-18T14:08:49.368 INFO:tasks.workunit.client.0.smithi047.stdout: "state": [ 2024-09-18T14:08:49.368 INFO:tasks.workunit.client.0.smithi047.stdout: "exists", 2024-09-18T14:08:49.368 INFO:tasks.workunit.client.0.smithi047.stdout: "up" 2024-09-18T14:08:49.368 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:49.368 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.368 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:49.368 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_xinfo": [ 2024-09-18T14:08:49.369 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.369 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 0, 2024-09-18T14:08:49.369 INFO:tasks.workunit.client.0.smithi047.stdout: "down_stamp": "2024-09-18T14:07:20.887595+0000", 2024-09-18T14:08:49.369 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_probability": 0, 2024-09-18T14:08:49.369 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_interval": 0, 2024-09-18T14:08:49.370 INFO:tasks.workunit.client.0.smithi047.stdout: "features": 4540138320759226367, 2024-09-18T14:08:49.370 INFO:tasks.workunit.client.0.smithi047.stdout: "old_weight": 0, 2024-09-18T14:08:49.370 INFO:tasks.workunit.client.0.smithi047.stdout: "last_purged_snaps_scrub": "2024-09-18T14:05:17.323724+0000", 2024-09-18T14:08:49.370 INFO:tasks.workunit.client.0.smithi047.stdout: "dead_epoch": 0 2024-09-18T14:08:49.370 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.371 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.371 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 1, 2024-09-18T14:08:49.371 INFO:tasks.workunit.client.0.smithi047.stdout: "down_stamp": "2024-09-18T14:07:21.512618+0000", 2024-09-18T14:08:49.371 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_probability": 0, 2024-09-18T14:08:49.371 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_interval": 0, 2024-09-18T14:08:49.371 INFO:tasks.workunit.client.0.smithi047.stdout: "features": 4540138320759226367, 2024-09-18T14:08:49.371 INFO:tasks.workunit.client.0.smithi047.stdout: "old_weight": 0, 2024-09-18T14:08:49.371 INFO:tasks.workunit.client.0.smithi047.stdout: "last_purged_snaps_scrub": "2024-09-18T14:05:50.556491+0000", 2024-09-18T14:08:49.372 INFO:tasks.workunit.client.0.smithi047.stdout: "dead_epoch": 0 2024-09-18T14:08:49.372 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.372 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.372 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 2, 2024-09-18T14:08:49.372 INFO:tasks.workunit.client.0.smithi047.stdout: "down_stamp": "2024-09-18T14:07:22.088585+0000", 2024-09-18T14:08:49.372 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_probability": 0, 2024-09-18T14:08:49.372 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_interval": 0, 2024-09-18T14:08:49.372 INFO:tasks.workunit.client.0.smithi047.stdout: "features": 4540138320759226367, 2024-09-18T14:08:49.372 INFO:tasks.workunit.client.0.smithi047.stdout: "old_weight": 0, 2024-09-18T14:08:49.372 INFO:tasks.workunit.client.0.smithi047.stdout: "last_purged_snaps_scrub": "2024-09-18T14:06:23.474574+0000", 2024-09-18T14:08:49.372 INFO:tasks.workunit.client.0.smithi047.stdout: "dead_epoch": 0 2024-09-18T14:08:49.373 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.373 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:49.373 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_upmap": [], 2024-09-18T14:08:49.373 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_upmap_items": [], 2024-09-18T14:08:49.373 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_temp": [], 2024-09-18T14:08:49.373 INFO:tasks.workunit.client.0.smithi047.stdout: "primary_temp": [], 2024-09-18T14:08:49.373 INFO:tasks.workunit.client.0.smithi047.stdout: "blocklist": {}, 2024-09-18T14:08:49.373 INFO:tasks.workunit.client.0.smithi047.stdout: "range_blocklist": {}, 2024-09-18T14:08:49.373 INFO:tasks.workunit.client.0.smithi047.stdout: "erasure_code_profiles": { 2024-09-18T14:08:49.373 INFO:tasks.workunit.client.0.smithi047.stdout: "default": { 2024-09-18T14:08:49.373 INFO:tasks.workunit.client.0.smithi047.stdout: "k": "2", 2024-09-18T14:08:49.373 INFO:tasks.workunit.client.0.smithi047.stdout: "m": "2", 2024-09-18T14:08:49.373 INFO:tasks.workunit.client.0.smithi047.stdout: "plugin": "jerasure", 2024-09-18T14:08:49.373 INFO:tasks.workunit.client.0.smithi047.stdout: "technique": "reed_sol_van" 2024-09-18T14:08:49.374 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.374 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.374 INFO:tasks.workunit.client.0.smithi047.stdout: "removed_snaps_queue": [], 2024-09-18T14:08:49.374 INFO:tasks.workunit.client.0.smithi047.stdout: "new_removed_snaps": [], 2024-09-18T14:08:49.374 INFO:tasks.workunit.client.0.smithi047.stdout: "new_purged_snaps": [], 2024-09-18T14:08:49.374 INFO:tasks.workunit.client.0.smithi047.stdout: "crush_node_flags": {}, 2024-09-18T14:08:49.374 INFO:tasks.workunit.client.0.smithi047.stdout: "device_class_flags": {}, 2024-09-18T14:08:49.374 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_mode": { 2024-09-18T14:08:49.374 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_mode_enabled": false, 2024-09-18T14:08:49.374 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_bucket_count": 0, 2024-09-18T14:08:49.374 INFO:tasks.workunit.client.0.smithi047.stdout: "degraded_stretch_mode": 0, 2024-09-18T14:08:49.374 INFO:tasks.workunit.client.0.smithi047.stdout: "recovering_stretch_mode": 0, 2024-09-18T14:08:49.374 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_mode_bucket": 0 2024-09-18T14:08:49.374 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.375 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.375 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_metadata": [ 2024-09-18T14:08:49.375 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.375 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T14:08:49.375 INFO:tasks.workunit.client.0.smithi047.stdout: "arch": "x86_64", 2024-09-18T14:08:49.375 INFO:tasks.workunit.client.0.smithi047.stdout: "back_addr": "[v2:127.0.0.1:6804/68576,v1:127.0.0.1:6805/68576]", 2024-09-18T14:08:49.375 INFO:tasks.workunit.client.0.smithi047.stdout: "back_iface": "", 2024-09-18T14:08:49.375 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs": "1", 2024-09-18T14:08:49.376 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_db": "0", 2024-09-18T14:08:49.376 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_wal": "0", 2024-09-18T14:08:49.376 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_single_shared_device": "1", 2024-09-18T14:08:49.376 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_allocation_from_file": "0", 2024-09-18T14:08:49.376 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_access_mode": "file", 2024-09-18T14:08:49.376 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_block_size": "4096", 2024-09-18T14:08:49.376 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_devices": "sda", 2024-09-18T14:08:49.376 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-18T14:08:49.376 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-18T14:08:49.376 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-18T14:08:49.376 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_rotational": "1", 2024-09-18T14:08:49.376 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_size": "107374182400", 2024-09-18T14:08:49.376 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_support_discard": "1", 2024-09-18T14:08:49.376 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_type": "hdd", 2024-09-18T14:08:49.377 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_min_alloc_size": "4096", 2024-09-18T14:08:49.377 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_release": "quincy", 2024-09-18T14:08:49.377 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:08:49.377 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_short": "17.2.7-1645-g7061ea6c", 2024-09-18T14:08:49.377 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_when_created": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:08:49.377 INFO:tasks.workunit.client.0.smithi047.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-18T14:08:49.377 INFO:tasks.workunit.client.0.smithi047.stdout: "created_at": "2024-09-18T14:04:51.996055Z", 2024-09-18T14:08:49.377 INFO:tasks.workunit.client.0.smithi047.stdout: "default_device_class": "hdd", 2024-09-18T14:08:49.377 INFO:tasks.workunit.client.0.smithi047.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4HN80", 2024-09-18T14:08:49.377 INFO:tasks.workunit.client.0.smithi047.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-18T14:08:49.377 INFO:tasks.workunit.client.0.smithi047.stdout: "devices": "sda", 2024-09-18T14:08:49.378 INFO:tasks.workunit.client.0.smithi047.stdout: "distro": "centos", 2024-09-18T14:08:49.378 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_description": "CentOS Stream 9", 2024-09-18T14:08:49.378 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_version": "9", 2024-09-18T14:08:49.378 INFO:tasks.workunit.client.0.smithi047.stdout: "front_addr": "[v2:127.0.0.1:6802/68576,v1:127.0.0.1:6803/68576]", 2024-09-18T14:08:49.378 INFO:tasks.workunit.client.0.smithi047.stdout: "front_iface": "", 2024-09-18T14:08:49.378 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/68576,v1:127.0.0.1:6809/68576]", 2024-09-18T14:08:49.378 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/68576,v1:127.0.0.1:6807/68576]", 2024-09-18T14:08:49.378 INFO:tasks.workunit.client.0.smithi047.stdout: "hostname": "smithi047", 2024-09-18T14:08:49.378 INFO:tasks.workunit.client.0.smithi047.stdout: "journal_rotational": "1", 2024-09-18T14:08:49.378 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-18T14:08:49.378 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-18T14:08:49.378 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_swap_kb": "0", 2024-09-18T14:08:49.378 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_total_kb": "32493492", 2024-09-18T14:08:49.379 INFO:tasks.workunit.client.0.smithi047.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-18T14:08:49.379 INFO:tasks.workunit.client.0.smithi047.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-18T14:08:49.379 INFO:tasks.workunit.client.0.smithi047.stdout: "os": "Linux", 2024-09-18T14:08:49.379 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_data": "td/divergent-priors/0", 2024-09-18T14:08:49.379 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_objectstore": "bluestore", 2024-09-18T14:08:49.379 INFO:tasks.workunit.client.0.smithi047.stdout: "osdspec_affinity": "", 2024-09-18T14:08:49.379 INFO:tasks.workunit.client.0.smithi047.stdout: "rotational": "1" 2024-09-18T14:08:49.379 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.379 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.379 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 1, 2024-09-18T14:08:49.379 INFO:tasks.workunit.client.0.smithi047.stdout: "arch": "x86_64", 2024-09-18T14:08:49.379 INFO:tasks.workunit.client.0.smithi047.stdout: "back_addr": "[v2:127.0.0.1:6812/62217,v1:127.0.0.1:6813/62217]", 2024-09-18T14:08:49.379 INFO:tasks.workunit.client.0.smithi047.stdout: "back_iface": "", 2024-09-18T14:08:49.379 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs": "1", 2024-09-18T14:08:49.380 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_db": "0", 2024-09-18T14:08:49.380 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_wal": "0", 2024-09-18T14:08:49.380 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_single_shared_device": "1", 2024-09-18T14:08:49.380 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_allocation_from_file": "0", 2024-09-18T14:08:49.380 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_access_mode": "file", 2024-09-18T14:08:49.380 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_block_size": "4096", 2024-09-18T14:08:49.380 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_devices": "sda", 2024-09-18T14:08:49.380 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-18T14:08:49.380 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-18T14:08:49.380 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-18T14:08:49.380 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_rotational": "1", 2024-09-18T14:08:49.380 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_size": "107374182400", 2024-09-18T14:08:49.380 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_support_discard": "1", 2024-09-18T14:08:49.380 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_type": "hdd", 2024-09-18T14:08:49.381 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_min_alloc_size": "4096", 2024-09-18T14:08:49.381 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_release": "quincy", 2024-09-18T14:08:49.381 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:08:49.381 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_short": "17.2.7-1645-g7061ea6c", 2024-09-18T14:08:49.381 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_when_created": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:08:49.381 INFO:tasks.workunit.client.0.smithi047.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-18T14:08:49.381 INFO:tasks.workunit.client.0.smithi047.stdout: "created_at": "2024-09-18T14:05:24.241405Z", 2024-09-18T14:08:49.381 INFO:tasks.workunit.client.0.smithi047.stdout: "default_device_class": "hdd", 2024-09-18T14:08:49.381 INFO:tasks.workunit.client.0.smithi047.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4HN80", 2024-09-18T14:08:49.381 INFO:tasks.workunit.client.0.smithi047.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-18T14:08:49.381 INFO:tasks.workunit.client.0.smithi047.stdout: "devices": "sda", 2024-09-18T14:08:49.381 INFO:tasks.workunit.client.0.smithi047.stdout: "distro": "centos", 2024-09-18T14:08:49.381 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_description": "CentOS Stream 9", 2024-09-18T14:08:49.381 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_version": "9", 2024-09-18T14:08:49.382 INFO:tasks.workunit.client.0.smithi047.stdout: "front_addr": "[v2:127.0.0.1:6810/62217,v1:127.0.0.1:6811/62217]", 2024-09-18T14:08:49.382 INFO:tasks.workunit.client.0.smithi047.stdout: "front_iface": "", 2024-09-18T14:08:49.382 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/62217,v1:127.0.0.1:6817/62217]", 2024-09-18T14:08:49.382 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/62217,v1:127.0.0.1:6815/62217]", 2024-09-18T14:08:49.382 INFO:tasks.workunit.client.0.smithi047.stdout: "hostname": "smithi047", 2024-09-18T14:08:49.382 INFO:tasks.workunit.client.0.smithi047.stdout: "journal_rotational": "1", 2024-09-18T14:08:49.382 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-18T14:08:49.382 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-18T14:08:49.382 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_swap_kb": "0", 2024-09-18T14:08:49.382 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_total_kb": "32493492", 2024-09-18T14:08:49.383 INFO:tasks.workunit.client.0.smithi047.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-18T14:08:49.383 INFO:tasks.workunit.client.0.smithi047.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-18T14:08:49.383 INFO:tasks.workunit.client.0.smithi047.stdout: "os": "Linux", 2024-09-18T14:08:49.383 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_data": "td/divergent-priors/1", 2024-09-18T14:08:49.383 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_objectstore": "bluestore", 2024-09-18T14:08:49.383 INFO:tasks.workunit.client.0.smithi047.stdout: "osdspec_affinity": "", 2024-09-18T14:08:49.384 INFO:tasks.workunit.client.0.smithi047.stdout: "rotational": "1" 2024-09-18T14:08:49.384 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.384 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.384 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T14:08:49.384 INFO:tasks.workunit.client.0.smithi047.stdout: "arch": "x86_64", 2024-09-18T14:08:49.385 INFO:tasks.workunit.client.0.smithi047.stdout: "back_addr": "[v2:127.0.0.1:6812/69154,v1:127.0.0.1:6813/69154]", 2024-09-18T14:08:49.385 INFO:tasks.workunit.client.0.smithi047.stdout: "back_iface": "", 2024-09-18T14:08:49.385 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs": "1", 2024-09-18T14:08:49.385 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_db": "0", 2024-09-18T14:08:49.385 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_wal": "0", 2024-09-18T14:08:49.385 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_single_shared_device": "1", 2024-09-18T14:08:49.385 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_allocation_from_file": "0", 2024-09-18T14:08:49.385 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_access_mode": "file", 2024-09-18T14:08:49.385 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_block_size": "4096", 2024-09-18T14:08:49.385 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_devices": "sda", 2024-09-18T14:08:49.385 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-18T14:08:49.386 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-18T14:08:49.386 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-18T14:08:49.386 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_rotational": "1", 2024-09-18T14:08:49.386 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_size": "107374182400", 2024-09-18T14:08:49.386 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_support_discard": "1", 2024-09-18T14:08:49.386 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_type": "hdd", 2024-09-18T14:08:49.386 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_min_alloc_size": "4096", 2024-09-18T14:08:49.386 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_release": "quincy", 2024-09-18T14:08:49.386 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:08:49.386 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_short": "17.2.7-1645-g7061ea6c", 2024-09-18T14:08:49.386 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_when_created": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:08:49.387 INFO:tasks.workunit.client.0.smithi047.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-18T14:08:49.387 INFO:tasks.workunit.client.0.smithi047.stdout: "created_at": "2024-09-18T14:05:56.801581Z", 2024-09-18T14:08:49.387 INFO:tasks.workunit.client.0.smithi047.stdout: "default_device_class": "hdd", 2024-09-18T14:08:49.387 INFO:tasks.workunit.client.0.smithi047.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4HN80", 2024-09-18T14:08:49.387 INFO:tasks.workunit.client.0.smithi047.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-18T14:08:49.387 INFO:tasks.workunit.client.0.smithi047.stdout: "devices": "sda", 2024-09-18T14:08:49.387 INFO:tasks.workunit.client.0.smithi047.stdout: "distro": "centos", 2024-09-18T14:08:49.387 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_description": "CentOS Stream 9", 2024-09-18T14:08:49.387 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_version": "9", 2024-09-18T14:08:49.387 INFO:tasks.workunit.client.0.smithi047.stdout: "front_addr": "[v2:127.0.0.1:6810/69154,v1:127.0.0.1:6811/69154]", 2024-09-18T14:08:49.387 INFO:tasks.workunit.client.0.smithi047.stdout: "front_iface": "", 2024-09-18T14:08:49.387 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/69154,v1:127.0.0.1:6817/69154]", 2024-09-18T14:08:49.388 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/69154,v1:127.0.0.1:6815/69154]", 2024-09-18T14:08:49.388 INFO:tasks.workunit.client.0.smithi047.stdout: "hostname": "smithi047", 2024-09-18T14:08:49.388 INFO:tasks.workunit.client.0.smithi047.stdout: "journal_rotational": "1", 2024-09-18T14:08:49.388 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-18T14:08:49.388 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-18T14:08:49.388 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_swap_kb": "0", 2024-09-18T14:08:49.388 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_total_kb": "32493492", 2024-09-18T14:08:49.388 INFO:tasks.workunit.client.0.smithi047.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-18T14:08:49.388 INFO:tasks.workunit.client.0.smithi047.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-18T14:08:49.388 INFO:tasks.workunit.client.0.smithi047.stdout: "os": "Linux", 2024-09-18T14:08:49.388 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_data": "td/divergent-priors/2", 2024-09-18T14:08:49.389 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_objectstore": "bluestore", 2024-09-18T14:08:49.389 INFO:tasks.workunit.client.0.smithi047.stdout: "osdspec_affinity": "", 2024-09-18T14:08:49.389 INFO:tasks.workunit.client.0.smithi047.stdout: "rotational": "1" 2024-09-18T14:08:49.389 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.389 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:49.389 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap_clean_epochs": { 2024-09-18T14:08:49.389 INFO:tasks.workunit.client.0.smithi047.stdout: "min_last_epoch_clean": 0, 2024-09-18T14:08:49.389 INFO:tasks.workunit.client.0.smithi047.stdout: "last_epoch_clean": { 2024-09-18T14:08:49.389 INFO:tasks.workunit.client.0.smithi047.stdout: "per_pool": [] 2024-09-18T14:08:49.389 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.389 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_epochs": [ 2024-09-18T14:08:49.389 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.389 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T14:08:49.389 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 57 2024-09-18T14:08:49.390 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.390 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.390 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T14:08:49.390 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 61 2024-09-18T14:08:49.390 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.390 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:49.390 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.390 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap_first_committed": 1, 2024-09-18T14:08:49.390 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap_last_committed": 70, 2024-09-18T14:08:49.390 INFO:tasks.workunit.client.0.smithi047.stdout: "crushmap": { 2024-09-18T14:08:49.390 INFO:tasks.workunit.client.0.smithi047.stdout: "devices": [ 2024-09-18T14:08:49.390 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.390 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T14:08:49.390 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "osd.0", 2024-09-18T14:08:49.391 INFO:tasks.workunit.client.0.smithi047.stdout: "class": "hdd" 2024-09-18T14:08:49.391 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.391 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.391 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 1, 2024-09-18T14:08:49.391 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "osd.1", 2024-09-18T14:08:49.391 INFO:tasks.workunit.client.0.smithi047.stdout: "class": "hdd" 2024-09-18T14:08:49.391 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.391 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.391 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T14:08:49.391 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "osd.2", 2024-09-18T14:08:49.391 INFO:tasks.workunit.client.0.smithi047.stdout: "class": "hdd" 2024-09-18T14:08:49.391 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.391 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:49.391 INFO:tasks.workunit.client.0.smithi047.stdout: "types": [ 2024-09-18T14:08:49.391 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.392 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 0, 2024-09-18T14:08:49.392 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "osd" 2024-09-18T14:08:49.392 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.392 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.392 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 1, 2024-09-18T14:08:49.392 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "host" 2024-09-18T14:08:49.392 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.392 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.392 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 2, 2024-09-18T14:08:49.392 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "chassis" 2024-09-18T14:08:49.392 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.392 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.392 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 3, 2024-09-18T14:08:49.392 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "rack" 2024-09-18T14:08:49.392 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.393 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.393 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 4, 2024-09-18T14:08:49.393 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "row" 2024-09-18T14:08:49.393 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.393 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.393 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 5, 2024-09-18T14:08:49.393 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "pdu" 2024-09-18T14:08:49.393 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.393 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.393 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 6, 2024-09-18T14:08:49.394 INFO:tasks.workunit.client.0.smithi047.stderr:report 930964276 2024-09-18T14:08:49.395 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "pod" 2024-09-18T14:08:49.395 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.395 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.395 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 7, 2024-09-18T14:08:49.395 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "room" 2024-09-18T14:08:49.395 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.395 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.395 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 8, 2024-09-18T14:08:49.395 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "datacenter" 2024-09-18T14:08:49.395 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.396 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.396 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 9, 2024-09-18T14:08:49.396 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "zone" 2024-09-18T14:08:49.396 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.396 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.396 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 10, 2024-09-18T14:08:49.396 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "region" 2024-09-18T14:08:49.396 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.396 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.396 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 11, 2024-09-18T14:08:49.396 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "root" 2024-09-18T14:08:49.397 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.397 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:49.397 INFO:tasks.workunit.client.0.smithi047.stdout: "buckets": [ 2024-09-18T14:08:49.397 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.397 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -1, 2024-09-18T14:08:49.398 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "default", 2024-09-18T14:08:49.398 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 11, 2024-09-18T14:08:49.398 INFO:tasks.workunit.client.0.smithi047.stdout: "type_name": "root", 2024-09-18T14:08:49.398 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:08:49.398 INFO:tasks.workunit.client.0.smithi047.stdout: "alg": "straw2", 2024-09-18T14:08:49.398 INFO:tasks.workunit.client.0.smithi047.stdout: "hash": "rjenkins1", 2024-09-18T14:08:49.399 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T14:08:49.399 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.399 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -3, 2024-09-18T14:08:49.399 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:08:49.399 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 0 2024-09-18T14:08:49.399 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.399 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:49.400 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.400 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.400 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -2, 2024-09-18T14:08:49.400 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "default~hdd", 2024-09-18T14:08:49.400 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 11, 2024-09-18T14:08:49.400 INFO:tasks.workunit.client.0.smithi047.stdout: "type_name": "root", 2024-09-18T14:08:49.400 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:08:49.400 INFO:tasks.workunit.client.0.smithi047.stdout: "alg": "straw2", 2024-09-18T14:08:49.400 INFO:tasks.workunit.client.0.smithi047.stdout: "hash": "rjenkins1", 2024-09-18T14:08:49.400 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T14:08:49.400 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.401 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -4, 2024-09-18T14:08:49.401 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:08:49.401 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 0 2024-09-18T14:08:49.401 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.401 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:49.401 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.401 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.402 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -3, 2024-09-18T14:08:49.402 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "smithi047", 2024-09-18T14:08:49.402 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 1, 2024-09-18T14:08:49.402 INFO:tasks.workunit.client.0.smithi047.stdout: "type_name": "host", 2024-09-18T14:08:49.402 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:08:49.402 INFO:tasks.workunit.client.0.smithi047.stdout: "alg": "straw2", 2024-09-18T14:08:49.402 INFO:tasks.workunit.client.0.smithi047.stdout: "hash": "rjenkins1", 2024-09-18T14:08:49.402 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T14:08:49.402 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.402 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T14:08:49.403 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:08:49.403 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 0 2024-09-18T14:08:49.403 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.403 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.403 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 1, 2024-09-18T14:08:49.403 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:08:49.403 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 1 2024-09-18T14:08:49.403 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.404 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.404 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T14:08:49.404 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:08:49.404 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 2 2024-09-18T14:08:49.404 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.404 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:49.404 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.404 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.404 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -4, 2024-09-18T14:08:49.404 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "smithi047~hdd", 2024-09-18T14:08:49.404 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 1, 2024-09-18T14:08:49.404 INFO:tasks.workunit.client.0.smithi047.stdout: "type_name": "host", 2024-09-18T14:08:49.404 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:08:49.404 INFO:tasks.workunit.client.0.smithi047.stdout: "alg": "straw2", 2024-09-18T14:08:49.404 INFO:tasks.workunit.client.0.smithi047.stdout: "hash": "rjenkins1", 2024-09-18T14:08:49.405 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T14:08:49.405 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.405 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T14:08:49.405 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:08:49.405 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 0 2024-09-18T14:08:49.405 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.405 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.405 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 1, 2024-09-18T14:08:49.405 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:08:49.405 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 1 2024-09-18T14:08:49.405 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.405 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.405 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T14:08:49.405 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:08:49.406 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 2 2024-09-18T14:08:49.406 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.406 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:49.406 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.406 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:49.406 INFO:tasks.workunit.client.0.smithi047.stdout: "rules": [ 2024-09-18T14:08:49.406 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.406 INFO:tasks.workunit.client.0.smithi047.stdout: "rule_id": 0, 2024-09-18T14:08:49.406 INFO:tasks.workunit.client.0.smithi047.stdout: "rule_name": "replicated_rule", 2024-09-18T14:08:49.406 INFO:tasks.workunit.client.0.smithi047.stdout: "type": 1, 2024-09-18T14:08:49.406 INFO:tasks.workunit.client.0.smithi047.stdout: "steps": [ 2024-09-18T14:08:49.406 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.406 INFO:tasks.workunit.client.0.smithi047.stdout: "op": "take", 2024-09-18T14:08:49.406 INFO:tasks.workunit.client.0.smithi047.stdout: "item": -1, 2024-09-18T14:08:49.407 INFO:tasks.workunit.client.0.smithi047.stdout: "item_name": "default" 2024-09-18T14:08:49.407 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.407 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.407 INFO:tasks.workunit.client.0.smithi047.stdout: "op": "choose_firstn", 2024-09-18T14:08:49.407 INFO:tasks.workunit.client.0.smithi047.stdout: "num": 0, 2024-09-18T14:08:49.407 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "osd" 2024-09-18T14:08:49.407 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.407 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.407 INFO:tasks.workunit.client.0.smithi047.stdout: "op": "emit" 2024-09-18T14:08:49.407 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.407 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:08:49.407 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.407 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:49.407 INFO:tasks.workunit.client.0.smithi047.stdout: "tunables": { 2024-09-18T14:08:49.408 INFO:tasks.workunit.client.0.smithi047.stdout: "choose_local_tries": 0, 2024-09-18T14:08:49.408 INFO:tasks.workunit.client.0.smithi047.stdout: "choose_local_fallback_tries": 0, 2024-09-18T14:08:49.408 INFO:tasks.workunit.client.0.smithi047.stdout: "choose_total_tries": 50, 2024-09-18T14:08:49.408 INFO:tasks.workunit.client.0.smithi047.stdout: "chooseleaf_descend_once": 1, 2024-09-18T14:08:49.408 INFO:tasks.workunit.client.0.smithi047.stdout: "chooseleaf_vary_r": 1, 2024-09-18T14:08:49.408 INFO:tasks.workunit.client.0.smithi047.stdout: "chooseleaf_stable": 1, 2024-09-18T14:08:49.408 INFO:tasks.workunit.client.0.smithi047.stdout: "straw_calc_version": 1, 2024-09-18T14:08:49.408 INFO:tasks.workunit.client.0.smithi047.stdout: "allowed_bucket_algs": 54, 2024-09-18T14:08:49.408 INFO:tasks.workunit.client.0.smithi047.stdout: "profile": "jewel", 2024-09-18T14:08:49.408 INFO:tasks.workunit.client.0.smithi047.stdout: "optimal_tunables": 1, 2024-09-18T14:08:49.408 INFO:tasks.workunit.client.0.smithi047.stdout: "legacy_tunables": 0, 2024-09-18T14:08:49.408 INFO:tasks.workunit.client.0.smithi047.stdout: "minimum_required_version": "jewel", 2024-09-18T14:08:49.408 INFO:tasks.workunit.client.0.smithi047.stdout: "require_feature_tunables": 1, 2024-09-18T14:08:49.408 INFO:tasks.workunit.client.0.smithi047.stdout: "require_feature_tunables2": 1, 2024-09-18T14:08:49.409 INFO:tasks.workunit.client.0.smithi047.stdout: "has_v2_rules": 0, 2024-09-18T14:08:49.409 INFO:tasks.workunit.client.0.smithi047.stdout: "require_feature_tunables3": 1, 2024-09-18T14:08:49.409 INFO:tasks.workunit.client.0.smithi047.stdout: "has_v3_rules": 0, 2024-09-18T14:08:49.409 INFO:tasks.workunit.client.0.smithi047.stdout: "has_v4_buckets": 1, 2024-09-18T14:08:49.409 INFO:tasks.workunit.client.0.smithi047.stdout: "require_feature_tunables5": 1, 2024-09-18T14:08:49.409 INFO:tasks.workunit.client.0.smithi047.stdout: "has_v5_rules": 0 2024-09-18T14:08:49.409 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.409 INFO:tasks.workunit.client.0.smithi047.stdout: "choose_args": {} 2024-09-18T14:08:49.409 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.409 INFO:tasks.workunit.client.0.smithi047.stdout: "fsmap": { 2024-09-18T14:08:49.409 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 1, 2024-09-18T14:08:49.409 INFO:tasks.workunit.client.0.smithi047.stdout: "default_fscid": -1, 2024-09-18T14:08:49.409 INFO:tasks.workunit.client.0.smithi047.stdout: "compat": { 2024-09-18T14:08:49.409 INFO:tasks.workunit.client.0.smithi047.stdout: "compat": {}, 2024-09-18T14:08:49.410 INFO:tasks.workunit.client.0.smithi047.stdout: "ro_compat": {}, 2024-09-18T14:08:49.410 INFO:tasks.workunit.client.0.smithi047.stdout: "incompat": { 2024-09-18T14:08:49.410 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_1": "base v0.20", 2024-09-18T14:08:49.410 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_2": "client writeable ranges", 2024-09-18T14:08:49.410 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_3": "default file layouts on dirs", 2024-09-18T14:08:49.410 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_4": "dir inode in separate object", 2024-09-18T14:08:49.410 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_5": "mds uses versioned encoding", 2024-09-18T14:08:49.410 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-18T14:08:49.410 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_8": "no anchor table", 2024-09-18T14:08:49.410 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_9": "file layout v2", 2024-09-18T14:08:49.410 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_10": "snaprealm v2" 2024-09-18T14:08:49.410 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.410 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.410 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_flags": { 2024-09-18T14:08:49.411 INFO:tasks.workunit.client.0.smithi047.stdout: "enable_multiple": true, 2024-09-18T14:08:49.411 INFO:tasks.workunit.client.0.smithi047.stdout: "ever_enabled_multiple": true 2024-09-18T14:08:49.411 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.411 INFO:tasks.workunit.client.0.smithi047.stdout: "standbys": [], 2024-09-18T14:08:49.411 INFO:tasks.workunit.client.0.smithi047.stdout: "filesystems": [] 2024-09-18T14:08:49.411 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.411 INFO:tasks.workunit.client.0.smithi047.stdout: "mdsmap_first_committed": 1, 2024-09-18T14:08:49.411 INFO:tasks.workunit.client.0.smithi047.stdout: "mdsmap_last_committed": 1, 2024-09-18T14:08:49.411 INFO:tasks.workunit.client.0.smithi047.stdout: "auth": { 2024-09-18T14:08:49.411 INFO:tasks.workunit.client.0.smithi047.stdout: "first_committed": 1, 2024-09-18T14:08:49.411 INFO:tasks.workunit.client.0.smithi047.stdout: "last_committed": 5, 2024-09-18T14:08:49.411 INFO:tasks.workunit.client.0.smithi047.stdout: "num_secrets": 3 2024-09-18T14:08:49.411 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.411 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg": 32, 2024-09-18T14:08:49.412 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg_active": 32, 2024-09-18T14:08:49.412 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg_unknown": 0, 2024-09-18T14:08:49.412 INFO:tasks.workunit.client.0.smithi047.stdout: "num_osd": 3, 2024-09-18T14:08:49.412 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_sum": { 2024-09-18T14:08:49.412 INFO:tasks.workunit.client.0.smithi047.stdout: "stat_sum": { 2024-09-18T14:08:49.412 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes": 47730, 2024-09-18T14:08:49.412 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects": 100, 2024-09-18T14:08:49.412 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_clones": 0, 2024-09-18T14:08:49.412 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_copies": 300, 2024-09-18T14:08:49.412 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing_on_primary": 0, 2024-09-18T14:08:49.412 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing": 0, 2024-09-18T14:08:49.412 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_degraded": 100, 2024-09-18T14:08:49.412 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_misplaced": 0, 2024-09-18T14:08:49.412 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_unfound": 0, 2024-09-18T14:08:49.413 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_dirty": 100, 2024-09-18T14:08:49.413 INFO:tasks.workunit.client.0.smithi047.stdout: "num_whiteouts": 0, 2024-09-18T14:08:49.413 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read": 0, 2024-09-18T14:08:49.413 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read_kb": 0, 2024-09-18T14:08:49.413 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write": 82, 2024-09-18T14:08:49.413 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write_kb": 82, 2024-09-18T14:08:49.413 INFO:tasks.workunit.client.0.smithi047.stdout: "num_scrub_errors": 0, 2024-09-18T14:08:49.413 INFO:tasks.workunit.client.0.smithi047.stdout: "num_shallow_scrub_errors": 0, 2024-09-18T14:08:49.413 INFO:tasks.workunit.client.0.smithi047.stdout: "num_deep_scrub_errors": 0, 2024-09-18T14:08:49.413 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_recovered": 0, 2024-09-18T14:08:49.413 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_recovered": 0, 2024-09-18T14:08:49.413 INFO:tasks.workunit.client.0.smithi047.stdout: "num_keys_recovered": 0, 2024-09-18T14:08:49.413 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_omap": 0, 2024-09-18T14:08:49.414 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_hit_set_archive": 0, 2024-09-18T14:08:49.414 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_hit_set_archive": 0, 2024-09-18T14:08:49.414 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush": 0, 2024-09-18T14:08:49.414 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_kb": 0, 2024-09-18T14:08:49.414 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict": 0, 2024-09-18T14:08:49.414 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_kb": 0, 2024-09-18T14:08:49.414 INFO:tasks.workunit.client.0.smithi047.stdout: "num_promote": 0, 2024-09-18T14:08:49.414 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_high": 0, 2024-09-18T14:08:49.414 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_low": 0, 2024-09-18T14:08:49.414 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_some": 0, 2024-09-18T14:08:49.414 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_full": 0, 2024-09-18T14:08:49.415 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_pinned": 0, 2024-09-18T14:08:49.415 INFO:tasks.workunit.client.0.smithi047.stdout: "num_legacy_snapsets": 0, 2024-09-18T14:08:49.415 INFO:tasks.workunit.client.0.smithi047.stdout: "num_large_omap_objects": 0, 2024-09-18T14:08:49.415 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_manifest": 0, 2024-09-18T14:08:49.415 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_bytes": 0, 2024-09-18T14:08:49.415 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_keys": 0, 2024-09-18T14:08:49.415 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_repaired": 0 2024-09-18T14:08:49.415 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.415 INFO:tasks.workunit.client.0.smithi047.stdout: "store_stats": { 2024-09-18T14:08:49.415 INFO:tasks.workunit.client.0.smithi047.stdout: "total": 0, 2024-09-18T14:08:49.415 INFO:tasks.workunit.client.0.smithi047.stdout: "available": 0, 2024-09-18T14:08:49.415 INFO:tasks.workunit.client.0.smithi047.stdout: "internally_reserved": 0, 2024-09-18T14:08:49.415 INFO:tasks.workunit.client.0.smithi047.stdout: "allocated": 0, 2024-09-18T14:08:49.415 INFO:tasks.workunit.client.0.smithi047.stdout: "data_stored": 0, 2024-09-18T14:08:49.416 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed": 0, 2024-09-18T14:08:49.416 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_allocated": 0, 2024-09-18T14:08:49.416 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_original": 0, 2024-09-18T14:08:49.416 INFO:tasks.workunit.client.0.smithi047.stdout: "omap_allocated": 0, 2024-09-18T14:08:49.416 INFO:tasks.workunit.client.0.smithi047.stdout: "internal_metadata": 0 2024-09-18T14:08:49.416 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.416 INFO:tasks.workunit.client.0.smithi047.stdout: "log_size": 469, 2024-09-18T14:08:49.416 INFO:tasks.workunit.client.0.smithi047.stdout: "ondisk_log_size": 469, 2024-09-18T14:08:49.416 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 64, 2024-09-18T14:08:49.416 INFO:tasks.workunit.client.0.smithi047.stdout: "acting": 64, 2024-09-18T14:08:49.416 INFO:tasks.workunit.client.0.smithi047.stdout: "num_store_stats": 0 2024-09-18T14:08:49.416 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.416 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_sum": { 2024-09-18T14:08:49.416 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 0, 2024-09-18T14:08:49.417 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 0, 2024-09-18T14:08:49.417 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pgs": 96, 2024-09-18T14:08:49.417 INFO:tasks.workunit.client.0.smithi047.stdout: "num_osds": 3, 2024-09-18T14:08:49.417 INFO:tasks.workunit.client.0.smithi047.stdout: "num_per_pool_osds": 3, 2024-09-18T14:08:49.417 INFO:tasks.workunit.client.0.smithi047.stdout: "num_per_pool_omap_osds": 3, 2024-09-18T14:08:49.417 INFO:tasks.workunit.client.0.smithi047.stdout: "kb": 314572800, 2024-09-18T14:08:49.417 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used": 895620, 2024-09-18T14:08:49.417 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_data": 2688, 2024-09-18T14:08:49.417 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_omap": 10, 2024-09-18T14:08:49.417 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_meta": 892853, 2024-09-18T14:08:49.417 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_avail": 313677180, 2024-09-18T14:08:49.417 INFO:tasks.workunit.client.0.smithi047.stdout: "statfs": { 2024-09-18T14:08:49.417 INFO:tasks.workunit.client.0.smithi047.stdout: "total": 322122547200, 2024-09-18T14:08:49.417 INFO:tasks.workunit.client.0.smithi047.stdout: "available": 321205432320, 2024-09-18T14:08:49.418 INFO:tasks.workunit.client.0.smithi047.stdout: "internally_reserved": 0, 2024-09-18T14:08:49.418 INFO:tasks.workunit.client.0.smithi047.stdout: "allocated": 2752512, 2024-09-18T14:08:49.418 INFO:tasks.workunit.client.0.smithi047.stdout: "data_stored": 641155, 2024-09-18T14:08:49.418 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed": 0, 2024-09-18T14:08:49.418 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_allocated": 0, 2024-09-18T14:08:49.418 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_original": 0, 2024-09-18T14:08:49.418 INFO:tasks.workunit.client.0.smithi047.stdout: "omap_allocated": 10790, 2024-09-18T14:08:49.418 INFO:tasks.workunit.client.0.smithi047.stdout: "internal_metadata": 914281946 2024-09-18T14:08:49.418 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.418 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_peers": [], 2024-09-18T14:08:49.418 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_trim_queue_len": 0, 2024-09-18T14:08:49.418 INFO:tasks.workunit.client.0.smithi047.stdout: "num_snap_trimming": 0, 2024-09-18T14:08:49.419 INFO:tasks.workunit.client.0.smithi047.stdout: "num_shards_repaired": 0, 2024-09-18T14:08:49.419 INFO:tasks.workunit.client.0.smithi047.stdout: "op_queue_age_hist": { 2024-09-18T14:08:49.419 INFO:tasks.workunit.client.0.smithi047.stdout: "histogram": [], 2024-09-18T14:08:49.419 INFO:tasks.workunit.client.0.smithi047.stdout: "upper_bound": 1 2024-09-18T14:08:49.419 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.419 INFO:tasks.workunit.client.0.smithi047.stdout: "perf_stat": { 2024-09-18T14:08:49.419 INFO:tasks.workunit.client.0.smithi047.stdout: "commit_latency_ms": 140, 2024-09-18T14:08:49.419 INFO:tasks.workunit.client.0.smithi047.stdout: "apply_latency_ms": 140, 2024-09-18T14:08:49.419 INFO:tasks.workunit.client.0.smithi047.stdout: "commit_latency_ns": 140000000, 2024-09-18T14:08:49.419 INFO:tasks.workunit.client.0.smithi047.stdout: "apply_latency_ns": 140000000 2024-09-18T14:08:49.419 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.419 INFO:tasks.workunit.client.0.smithi047.stdout: "alerts": [], 2024-09-18T14:08:49.419 INFO:tasks.workunit.client.0.smithi047.stdout: "network_ping_times": [] 2024-09-18T14:08:49.419 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.419 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_sum_by_class": { 2024-09-18T14:08:49.420 INFO:tasks.workunit.client.0.smithi047.stdout: "hdd": { 2024-09-18T14:08:49.420 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 0, 2024-09-18T14:08:49.420 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 0, 2024-09-18T14:08:49.420 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pgs": 96, 2024-09-18T14:08:49.420 INFO:tasks.workunit.client.0.smithi047.stdout: "num_osds": 3, 2024-09-18T14:08:49.420 INFO:tasks.workunit.client.0.smithi047.stdout: "num_per_pool_osds": 3, 2024-09-18T14:08:49.420 INFO:tasks.workunit.client.0.smithi047.stdout: "num_per_pool_omap_osds": 3, 2024-09-18T14:08:49.420 INFO:tasks.workunit.client.0.smithi047.stdout: "kb": 314572800, 2024-09-18T14:08:49.420 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used": 895620, 2024-09-18T14:08:49.420 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_data": 2688, 2024-09-18T14:08:49.420 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_omap": 10, 2024-09-18T14:08:49.420 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_meta": 892853, 2024-09-18T14:08:49.420 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_avail": 313677180, 2024-09-18T14:08:49.421 INFO:tasks.workunit.client.0.smithi047.stdout: "statfs": { 2024-09-18T14:08:49.421 INFO:tasks.workunit.client.0.smithi047.stdout: "total": 322122547200, 2024-09-18T14:08:49.421 INFO:tasks.workunit.client.0.smithi047.stdout: "available": 321205432320, 2024-09-18T14:08:49.421 INFO:tasks.workunit.client.0.smithi047.stdout: "internally_reserved": 0, 2024-09-18T14:08:49.421 INFO:tasks.workunit.client.0.smithi047.stdout: "allocated": 2752512, 2024-09-18T14:08:49.421 INFO:tasks.workunit.client.0.smithi047.stdout: "data_stored": 641155, 2024-09-18T14:08:49.422 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed": 0, 2024-09-18T14:08:49.422 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_allocated": 0, 2024-09-18T14:08:49.422 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_original": 0, 2024-09-18T14:08:49.422 INFO:tasks.workunit.client.0.smithi047.stdout: "omap_allocated": 10790, 2024-09-18T14:08:49.422 INFO:tasks.workunit.client.0.smithi047.stdout: "internal_metadata": 914281946 2024-09-18T14:08:49.422 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.422 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_peers": [], 2024-09-18T14:08:49.422 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_trim_queue_len": 0, 2024-09-18T14:08:49.422 INFO:tasks.workunit.client.0.smithi047.stdout: "num_snap_trimming": 0, 2024-09-18T14:08:49.422 INFO:tasks.workunit.client.0.smithi047.stdout: "num_shards_repaired": 0, 2024-09-18T14:08:49.422 INFO:tasks.workunit.client.0.smithi047.stdout: "op_queue_age_hist": { 2024-09-18T14:08:49.423 INFO:tasks.workunit.client.0.smithi047.stdout: "histogram": [], 2024-09-18T14:08:49.423 INFO:tasks.workunit.client.0.smithi047.stdout: "upper_bound": 1 2024-09-18T14:08:49.423 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.423 INFO:tasks.workunit.client.0.smithi047.stdout: "perf_stat": { 2024-09-18T14:08:49.423 INFO:tasks.workunit.client.0.smithi047.stdout: "commit_latency_ms": 140, 2024-09-18T14:08:49.423 INFO:tasks.workunit.client.0.smithi047.stdout: "apply_latency_ms": 140, 2024-09-18T14:08:49.423 INFO:tasks.workunit.client.0.smithi047.stdout: "commit_latency_ns": 140000000, 2024-09-18T14:08:49.423 INFO:tasks.workunit.client.0.smithi047.stdout: "apply_latency_ns": 140000000 2024-09-18T14:08:49.423 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.423 INFO:tasks.workunit.client.0.smithi047.stdout: "alerts": [], 2024-09-18T14:08:49.423 INFO:tasks.workunit.client.0.smithi047.stdout: "network_ping_times": [] 2024-09-18T14:08:49.423 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.423 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.423 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_stats": [ 2024-09-18T14:08:49.423 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.424 INFO:tasks.workunit.client.0.smithi047.stdout: "poolid": 1, 2024-09-18T14:08:49.424 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg": 32, 2024-09-18T14:08:49.424 INFO:tasks.workunit.client.0.smithi047.stdout: "stat_sum": { 2024-09-18T14:08:49.424 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes": 47730, 2024-09-18T14:08:49.424 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects": 100, 2024-09-18T14:08:49.424 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_clones": 0, 2024-09-18T14:08:49.424 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_copies": 300, 2024-09-18T14:08:49.424 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing_on_primary": 0, 2024-09-18T14:08:49.424 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing": 0, 2024-09-18T14:08:49.424 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_degraded": 100, 2024-09-18T14:08:49.425 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_misplaced": 0, 2024-09-18T14:08:49.425 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_unfound": 0, 2024-09-18T14:08:49.425 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_dirty": 100, 2024-09-18T14:08:49.425 INFO:tasks.workunit.client.0.smithi047.stdout: "num_whiteouts": 0, 2024-09-18T14:08:49.425 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read": 0, 2024-09-18T14:08:49.425 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read_kb": 0, 2024-09-18T14:08:49.425 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write": 82, 2024-09-18T14:08:49.425 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write_kb": 82, 2024-09-18T14:08:49.425 INFO:tasks.workunit.client.0.smithi047.stdout: "num_scrub_errors": 0, 2024-09-18T14:08:49.425 INFO:tasks.workunit.client.0.smithi047.stdout: "num_shallow_scrub_errors": 0, 2024-09-18T14:08:49.425 INFO:tasks.workunit.client.0.smithi047.stdout: "num_deep_scrub_errors": 0, 2024-09-18T14:08:49.425 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_recovered": 0, 2024-09-18T14:08:49.425 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_recovered": 0, 2024-09-18T14:08:49.426 INFO:tasks.workunit.client.0.smithi047.stdout: "num_keys_recovered": 0, 2024-09-18T14:08:49.426 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_omap": 0, 2024-09-18T14:08:49.426 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_hit_set_archive": 0, 2024-09-18T14:08:49.426 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_hit_set_archive": 0, 2024-09-18T14:08:49.426 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush": 0, 2024-09-18T14:08:49.426 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_kb": 0, 2024-09-18T14:08:49.426 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict": 0, 2024-09-18T14:08:49.426 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_kb": 0, 2024-09-18T14:08:49.426 INFO:tasks.workunit.client.0.smithi047.stdout: "num_promote": 0, 2024-09-18T14:08:49.426 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_high": 0, 2024-09-18T14:08:49.426 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_low": 0, 2024-09-18T14:08:49.426 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_some": 0, 2024-09-18T14:08:49.427 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_full": 0, 2024-09-18T14:08:49.427 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_pinned": 0, 2024-09-18T14:08:49.427 INFO:tasks.workunit.client.0.smithi047.stdout: "num_legacy_snapsets": 0, 2024-09-18T14:08:49.427 INFO:tasks.workunit.client.0.smithi047.stdout: "num_large_omap_objects": 0, 2024-09-18T14:08:49.427 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_manifest": 0, 2024-09-18T14:08:49.427 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_bytes": 0, 2024-09-18T14:08:49.427 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_keys": 0, 2024-09-18T14:08:49.427 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_repaired": 0 2024-09-18T14:08:49.427 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.427 INFO:tasks.workunit.client.0.smithi047.stdout: "store_stats": { 2024-09-18T14:08:49.427 INFO:tasks.workunit.client.0.smithi047.stdout: "total": 0, 2024-09-18T14:08:49.427 INFO:tasks.workunit.client.0.smithi047.stdout: "available": 0, 2024-09-18T14:08:49.427 INFO:tasks.workunit.client.0.smithi047.stdout: "internally_reserved": 0, 2024-09-18T14:08:49.427 INFO:tasks.workunit.client.0.smithi047.stdout: "allocated": 1220608, 2024-09-18T14:08:49.428 INFO:tasks.workunit.client.0.smithi047.stdout: "data_stored": 142618, 2024-09-18T14:08:49.428 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed": 0, 2024-09-18T14:08:49.428 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_allocated": 0, 2024-09-18T14:08:49.428 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_original": 0, 2024-09-18T14:08:49.428 INFO:tasks.workunit.client.0.smithi047.stdout: "omap_allocated": 0, 2024-09-18T14:08:49.428 INFO:tasks.workunit.client.0.smithi047.stdout: "internal_metadata": 0 2024-09-18T14:08:49.428 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.428 INFO:tasks.workunit.client.0.smithi047.stdout: "log_size": 469, 2024-09-18T14:08:49.428 INFO:tasks.workunit.client.0.smithi047.stdout: "ondisk_log_size": 469, 2024-09-18T14:08:49.428 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 64, 2024-09-18T14:08:49.428 INFO:tasks.workunit.client.0.smithi047.stdout: "acting": 64, 2024-09-18T14:08:49.428 INFO:tasks.workunit.client.0.smithi047.stdout: "num_store_stats": 3 2024-09-18T14:08:49.428 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.428 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:49.428 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_stats": [ 2024-09-18T14:08:49.429 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.429 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 0, 2024-09-18T14:08:49.429 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 244813135892 2024-09-18T14:08:49.429 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.429 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.429 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 1, 2024-09-18T14:08:49.429 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 42949672989 2024-09-18T14:08:49.429 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.429 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.429 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 2, 2024-09-18T14:08:49.429 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 261993005075 2024-09-18T14:08:49.429 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.429 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:49.429 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg_by_state": [ 2024-09-18T14:08:49.429 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.430 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "active+undersized+degraded", 2024-09-18T14:08:49.430 INFO:tasks.workunit.client.0.smithi047.stdout: "num": 32 2024-09-18T14:08:49.430 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.430 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:49.430 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg_by_osd": [ 2024-09-18T14:08:49.430 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.430 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 2, 2024-09-18T14:08:49.430 INFO:tasks.workunit.client.0.smithi047.stdout: "num_primary_pg": 19, 2024-09-18T14:08:49.430 INFO:tasks.workunit.client.0.smithi047.stdout: "num_acting_pg": 32, 2024-09-18T14:08:49.430 INFO:tasks.workunit.client.0.smithi047.stdout: "num_up_not_acting_pg": 0 2024-09-18T14:08:49.430 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.430 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.430 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 0, 2024-09-18T14:08:49.431 INFO:tasks.workunit.client.0.smithi047.stdout: "num_primary_pg": 13, 2024-09-18T14:08:49.431 INFO:tasks.workunit.client.0.smithi047.stdout: "num_acting_pg": 32, 2024-09-18T14:08:49.431 INFO:tasks.workunit.client.0.smithi047.stdout: "num_up_not_acting_pg": 0 2024-09-18T14:08:49.431 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.431 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.431 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 1, 2024-09-18T14:08:49.431 INFO:tasks.workunit.client.0.smithi047.stdout: "num_primary_pg": 0, 2024-09-18T14:08:49.431 INFO:tasks.workunit.client.0.smithi047.stdout: "num_acting_pg": 0, 2024-09-18T14:08:49.431 INFO:tasks.workunit.client.0.smithi047.stdout: "num_up_not_acting_pg": 0 2024-09-18T14:08:49.431 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.431 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:49.431 INFO:tasks.workunit.client.0.smithi047.stdout: "purged_snaps": [ 2024-09-18T14:08:49.431 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:08:49.431 INFO:tasks.workunit.client.0.smithi047.stdout: "pool": 1, 2024-09-18T14:08:49.431 INFO:tasks.workunit.client.0.smithi047.stdout: "purged_snaps": {} 2024-09-18T14:08:49.432 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.432 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:08:49.432 INFO:tasks.workunit.client.0.smithi047.stdout: "servicemap": { 2024-09-18T14:08:49.432 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 7, 2024-09-18T14:08:49.432 INFO:tasks.workunit.client.0.smithi047.stdout: "modified": "2024-09-18T14:08:34.624949+0000", 2024-09-18T14:08:49.432 INFO:tasks.workunit.client.0.smithi047.stdout: "services": {} 2024-09-18T14:08:49.432 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:08:49.432 INFO:tasks.workunit.client.0.smithi047.stdout: "paxos": { 2024-09-18T14:08:49.432 INFO:tasks.workunit.client.0.smithi047.stdout: "first_committed": 1, 2024-09-18T14:08:49.432 INFO:tasks.workunit.client.0.smithi047.stdout: "last_committed": 662, 2024-09-18T14:08:49.432 INFO:tasks.workunit.client.0.smithi047.stdout: "last_pn": 0, 2024-09-18T14:08:49.432 INFO:tasks.workunit.client.0.smithi047.stdout: "accepted_pn": 0 2024-09-18T14:08:49.432 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:08:49.432 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T14:08:49.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-18T14:08:49.433 INFO:tasks.workunit.client.0.smithi047.stdout:delay recovery 2024-09-18T14:08:49.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:761: TEST_divergent_3: echo 'delay recovery' 2024-09-18T14:08:49.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: ceph pg dump pgs 2024-09-18T14:08:49.712 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:08:49.713 INFO:tasks.workunit.client.0.smithi047.stdout:1.1f 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814303+0000 32'80 70:119 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:49.713 INFO:tasks.workunit.client.0.smithi047.stdout:1.1e 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.814587+0000 34'81 70:118 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:49.713 INFO:tasks.workunit.client.0.smithi047.stdout:1.1d 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814571+0000 32'80 70:119 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:49.713 INFO:tasks.workunit.client.0.smithi047.stdout:1.b 4 0 4 0 0 1691 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.787168+0000 34'81 70:131 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:49.713 INFO:tasks.workunit.client.0.smithi047.stdout:1.a 3 0 3 0 0 1208 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.813808+0000 32'80 70:105 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:49.713 INFO:tasks.workunit.client.0.smithi047.stdout:1.9 5 0 5 0 0 2174 0 0 16 66 16 active+undersized+degraded 2024-09-18T14:07:51.812741+0000 34'82 70:131 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:49.713 INFO:tasks.workunit.client.0.smithi047.stdout:1.8 5 0 5 0 0 2174 0 0 16 66 16 active+undersized+degraded 2024-09-18T14:07:51.814091+0000 34'82 70:119 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:49.713 INFO:tasks.workunit.client.0.smithi047.stdout:1.7 3 0 3 0 0 1208 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.812691+0000 32'80 70:119 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:49.713 INFO:tasks.workunit.client.0.smithi047.stdout:1.6 3 0 3 0 0 1208 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814625+0000 32'80 70:129 [0,2] 0 [0,2] 0 32'80 2024-09-18T14:06:58.202394+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:49.713 INFO:tasks.workunit.client.0.smithi047.stdout:1.5 6 0 6 0 0 2898 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.814179+0000 34'81 70:140 [2,0] 2 [2,0] 2 34'81 2024-09-18T14:07:06.420533+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T15:28:19.740089+0000 6 0 2024-09-18T14:08:49.713 INFO:tasks.workunit.client.0.smithi047.stdout:1.4 3 0 3 0 0 1208 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.813528+0000 32'80 70:117 [0,2] 0 [0,2] 0 32'80 2024-09-18T14:07:08.170467+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:49.713 INFO:tasks.workunit.client.0.smithi047.stdout:1.3 4 0 4 0 0 1691 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.815161+0000 34'81 70:132 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:49.714 INFO:tasks.workunit.client.0.smithi047.stdout:1.2 3 0 3 0 0 1208 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814414+0000 32'80 70:129 [0,2] 0 [0,2] 0 32'80 2024-09-18T14:07:04.230275+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:49.714 INFO:tasks.workunit.client.0.smithi047.stdout:1.c 4 0 4 0 0 1691 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.812888+0000 34'81 70:130 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:49.714 INFO:tasks.workunit.client.0.smithi047.stdout:1.1 4 0 4 0 0 1691 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.813144+0000 34'81 70:132 [2,0] 2 [2,0] 2 34'81 2024-09-18T14:06:56.389936+0000 34'81 2024-09-18T14:06:56.389936+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:49.714 INFO:tasks.workunit.client.0.smithi047.stdout:1.1c 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.812871+0000 32'80 70:117 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:49.714 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 3 0 3 0 0 1449 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.815935+0000 34'81 70:141 [0,2] 0 [0,2] 0 32'80 2024-09-18T14:06:54.303691+0000 32'80 2024-09-18T14:06:54.303691+0000 0 1 periodic scrub scheduled @ 2024-09-19T21:42:35.244281+0000 2 0 2024-09-18T14:08:49.714 INFO:tasks.workunit.client.0.smithi047.stdout:1.d 3 0 3 0 0 1449 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.814765+0000 34'81 70:143 [2,0] 2 [2,0] 2 34'81 2024-09-18T14:07:09.554363+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T21:30:11.231921+0000 3 0 2024-09-18T14:08:49.714 INFO:tasks.workunit.client.0.smithi047.stdout:1.e 5 0 5 0 0 1845 0 0 17 66 17 active+undersized+degraded 2024-09-18T14:07:51.815577+0000 70'83 70:120 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:49.714 INFO:tasks.workunit.client.0.smithi047.stdout:1.f 4 0 4 0 0 1932 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814986+0000 32'80 70:129 [2,0] 2 [2,0] 2 32'80 2024-09-18T14:07:00.547567+0000 32'80 2024-09-18T14:07:00.547567+0000 0 1 periodic scrub scheduled @ 2024-09-19T22:57:43.008803+0000 4 0 2024-09-18T14:08:49.714 INFO:tasks.workunit.client.0.smithi047.stdout:1.10 4 0 4 0 0 1691 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.813572+0000 34'81 70:130 [0,2] 0 [0,2] 0 34'81 2024-09-18T14:07:07.204718+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:49.714 INFO:tasks.workunit.client.0.smithi047.stdout:1.11 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:52.902633+0000 32'80 70:117 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:49.714 INFO:tasks.workunit.client.0.smithi047.stdout:1.12 1 0 1 0 0 483 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.812635+0000 32'80 70:137 [0,2] 0 [0,2] 0 32'80 2024-09-18T14:07:08.487332+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T19:22:31.668808+0000 1 0 2024-09-18T14:08:49.714 INFO:tasks.workunit.client.0.smithi047.stdout:1.13 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.813086+0000 34'81 70:132 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:49.714 INFO:tasks.workunit.client.0.smithi047.stdout:1.14 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.816330+0000 34'81 70:130 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:49.715 INFO:tasks.workunit.client.0.smithi047.stdout:1.15 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.815134+0000 32'80 70:119 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:49.715 INFO:tasks.workunit.client.0.smithi047.stdout:1.16 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814810+0000 32'80 70:129 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:49.715 INFO:tasks.workunit.client.0.smithi047.stdout:1.17 1 0 1 0 0 483 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.813569+0000 32'80 70:129 [2,0] 2 [2,0] 2 32'80 2024-09-18T14:07:05.418931+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T19:59:06.890013+0000 1 0 2024-09-18T14:08:49.715 INFO:tasks.workunit.client.0.smithi047.stdout:1.18 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.814107+0000 34'81 70:118 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:49.715 INFO:tasks.workunit.client.0.smithi047.stdout:1.19 2 0 2 0 0 966 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.813376+0000 32'80 70:127 [2,0] 2 [2,0] 2 32'80 2024-09-18T14:07:07.546410+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T20:01:59.930990+0000 2 0 2024-09-18T14:08:49.715 INFO:tasks.workunit.client.0.smithi047.stdout:1.1a 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.812774+0000 34'81 70:118 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:49.715 INFO:tasks.workunit.client.0.smithi047.stdout:1.1b 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.813412+0000 34'81 70:131 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:49.715 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:08:49.715 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:08:49.715 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:08:49.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:763: TEST_divergent_3: for i in $non_divergent 2024-09-18T14:08:49.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: get_asok_path osd.0 2024-09-18T14:08:49.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:08:49.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:08:49.724 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:08:49.724 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:08:49.724 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:08:49.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32609/ceph-osd.0.asok 2024-09-18T14:08:49.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: CEPH_ARGS= 2024-09-18T14:08:49.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.32609/ceph-osd.0.asok set_recovery_delay 100000 2024-09-18T14:08:49.803 INFO:tasks.workunit.client.0.smithi047.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-18T14:08:49.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:763: TEST_divergent_3: for i in $non_divergent 2024-09-18T14:08:49.810 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: get_asok_path osd.2 2024-09-18T14:08:49.811 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:08:49.811 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:08:49.811 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:08:49.811 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:08:49.811 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:08:49.811 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32609/ceph-osd.2.asok 2024-09-18T14:08:49.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: CEPH_ARGS= 2024-09-18T14:08:49.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.32609/ceph-osd.2.asok set_recovery_delay 100000 2024-09-18T14:08:49.889 INFO:tasks.workunit.client.0.smithi047.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-18T14:08:49.896 INFO:tasks.workunit.client.0.smithi047.stdout:revive divergent 1 2024-09-18T14:08:49.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:769: TEST_divergent_3: echo 'revive divergent 1' 2024-09-18T14:08:49.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:770: TEST_divergent_3: ceph pg dump pgs 2024-09-18T14:08:50.253 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:08:50.253 INFO:tasks.workunit.client.0.smithi047.stdout:1.1f 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814303+0000 32'80 70:119 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:50.253 INFO:tasks.workunit.client.0.smithi047.stdout:1.1e 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.814587+0000 34'81 70:118 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:50.253 INFO:tasks.workunit.client.0.smithi047.stdout:1.1d 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814571+0000 32'80 70:119 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:50.253 INFO:tasks.workunit.client.0.smithi047.stdout:1.b 4 0 4 0 0 1691 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.787168+0000 34'81 70:131 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:50.254 INFO:tasks.workunit.client.0.smithi047.stdout:1.a 3 0 3 0 0 1208 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.813808+0000 32'80 70:105 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:50.254 INFO:tasks.workunit.client.0.smithi047.stdout:1.9 5 0 5 0 0 2174 0 0 16 66 16 active+undersized+degraded 2024-09-18T14:07:51.812741+0000 34'82 70:131 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:50.254 INFO:tasks.workunit.client.0.smithi047.stdout:1.8 5 0 5 0 0 2174 0 0 16 66 16 active+undersized+degraded 2024-09-18T14:07:51.814091+0000 34'82 70:119 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:50.254 INFO:tasks.workunit.client.0.smithi047.stdout:1.7 3 0 3 0 0 1208 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.812691+0000 32'80 70:119 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:50.254 INFO:tasks.workunit.client.0.smithi047.stdout:1.6 3 0 3 0 0 1208 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814625+0000 32'80 70:129 [0,2] 0 [0,2] 0 32'80 2024-09-18T14:06:58.202394+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:50.254 INFO:tasks.workunit.client.0.smithi047.stdout:1.5 6 0 6 0 0 2898 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.814179+0000 34'81 70:140 [2,0] 2 [2,0] 2 34'81 2024-09-18T14:07:06.420533+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T15:28:19.740089+0000 6 0 2024-09-18T14:08:50.254 INFO:tasks.workunit.client.0.smithi047.stdout:1.4 3 0 3 0 0 1208 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.813528+0000 32'80 70:117 [0,2] 0 [0,2] 0 32'80 2024-09-18T14:07:08.170467+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:50.254 INFO:tasks.workunit.client.0.smithi047.stdout:1.3 4 0 4 0 0 1691 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.815161+0000 34'81 70:132 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:50.254 INFO:tasks.workunit.client.0.smithi047.stdout:1.2 3 0 3 0 0 1208 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814414+0000 32'80 70:129 [0,2] 0 [0,2] 0 32'80 2024-09-18T14:07:04.230275+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:50.254 INFO:tasks.workunit.client.0.smithi047.stdout:1.c 4 0 4 0 0 1691 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.812888+0000 34'81 70:130 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:50.254 INFO:tasks.workunit.client.0.smithi047.stdout:1.1 4 0 4 0 0 1691 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.813144+0000 34'81 70:132 [2,0] 2 [2,0] 2 34'81 2024-09-18T14:06:56.389936+0000 34'81 2024-09-18T14:06:56.389936+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:50.254 INFO:tasks.workunit.client.0.smithi047.stdout:1.1c 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.812871+0000 32'80 70:117 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:50.254 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 3 0 3 0 0 1449 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.815935+0000 34'81 70:141 [0,2] 0 [0,2] 0 32'80 2024-09-18T14:06:54.303691+0000 32'80 2024-09-18T14:06:54.303691+0000 0 1 periodic scrub scheduled @ 2024-09-19T21:42:35.244281+0000 2 0 2024-09-18T14:08:50.255 INFO:tasks.workunit.client.0.smithi047.stdout:1.d 3 0 3 0 0 1449 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.814765+0000 34'81 70:143 [2,0] 2 [2,0] 2 34'81 2024-09-18T14:07:09.554363+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T21:30:11.231921+0000 3 0 2024-09-18T14:08:50.255 INFO:tasks.workunit.client.0.smithi047.stdout:1.e 5 0 5 0 0 1845 0 0 17 66 17 active+undersized+degraded 2024-09-18T14:07:51.815577+0000 70'83 70:120 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:50.255 INFO:tasks.workunit.client.0.smithi047.stdout:1.f 4 0 4 0 0 1932 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814986+0000 32'80 70:129 [2,0] 2 [2,0] 2 32'80 2024-09-18T14:07:00.547567+0000 32'80 2024-09-18T14:07:00.547567+0000 0 1 periodic scrub scheduled @ 2024-09-19T22:57:43.008803+0000 4 0 2024-09-18T14:08:50.255 INFO:tasks.workunit.client.0.smithi047.stdout:1.10 4 0 4 0 0 1691 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.813572+0000 34'81 70:130 [0,2] 0 [0,2] 0 34'81 2024-09-18T14:07:07.204718+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:50.255 INFO:tasks.workunit.client.0.smithi047.stdout:1.11 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:52.902633+0000 32'80 70:117 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:50.255 INFO:tasks.workunit.client.0.smithi047.stdout:1.12 1 0 1 0 0 483 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.812635+0000 32'80 70:137 [0,2] 0 [0,2] 0 32'80 2024-09-18T14:07:08.487332+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T19:22:31.668808+0000 1 0 2024-09-18T14:08:50.255 INFO:tasks.workunit.client.0.smithi047.stdout:1.13 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.813086+0000 34'81 70:132 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:50.255 INFO:tasks.workunit.client.0.smithi047.stdout:1.14 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.816330+0000 34'81 70:130 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:50.255 INFO:tasks.workunit.client.0.smithi047.stdout:1.15 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.815134+0000 32'80 70:119 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:50.255 INFO:tasks.workunit.client.0.smithi047.stdout:1.16 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814810+0000 32'80 70:129 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:50.255 INFO:tasks.workunit.client.0.smithi047.stdout:1.17 1 0 1 0 0 483 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.813569+0000 32'80 70:129 [2,0] 2 [2,0] 2 32'80 2024-09-18T14:07:05.418931+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T19:59:06.890013+0000 1 0 2024-09-18T14:08:50.255 INFO:tasks.workunit.client.0.smithi047.stdout:1.18 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.814107+0000 34'81 70:118 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:50.255 INFO:tasks.workunit.client.0.smithi047.stdout:1.19 2 0 2 0 0 966 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.813376+0000 32'80 70:127 [2,0] 2 [2,0] 2 32'80 2024-09-18T14:07:07.546410+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T20:01:59.930990+0000 2 0 2024-09-18T14:08:50.255 INFO:tasks.workunit.client.0.smithi047.stdout:1.1a 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.812774+0000 34'81 70:118 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:50.256 INFO:tasks.workunit.client.0.smithi047.stdout:1.1b 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.813412+0000 34'81 70:131 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:50.256 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:08:50.256 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:08:50.256 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:08:50.264 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:771: TEST_divergent_3: ceph osd set noup 2024-09-18T14:08:50.883 INFO:tasks.workunit.client.0.smithi047.stderr:noup is set 2024-09-18T14:08:50.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:772: TEST_divergent_3: activate_osd td/divergent-priors 1 2024-09-18T14:08:50.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-18T14:08:50.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:08:50.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T14:08:50.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:08:50.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-09-18T14:08:50.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T14:08:50.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:08:50.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:08:50.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:08:50.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-18T14:08:50.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-18T14:08:50.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:08:50.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:08:50.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-18T14:08:50.898 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:08:50.898 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:08:50.898 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:08:50.898 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:08:50.899 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:08:50.899 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:08:50.899 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:08:50.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:08:50.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:08:50.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-18T14:08:50.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-18T14:08:50.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:08:50.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:08:50.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:08:50.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:08:50.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:08:50.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:08:50.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-09-18T14:08:50.900 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T14:08:50.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T14:08:50.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:08:50.901 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-09-18T14:08:50.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T14:08:50.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:08:50.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:08:50.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:08:50.928 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:08:50.928+0000 7f545a82d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:08:50.930 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:08:50.930+0000 7f545a82d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:08:50.931 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:08:50.931+0000 7f545a82d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:08:51.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-09-18T14:08:51.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:773: TEST_divergent_3: sleep 5 2024-09-18T14:08:51.448 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:08:51.448+0000 7f545a82d440 -1 Falling back to public interface 2024-09-18T14:08:54.462 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:08:54.462+0000 7f545a82d440 -1 osd.1 47 log_to_monitors true 2024-09-18T14:08:56.315 INFO:tasks.workunit.client.0.smithi047.stdout:delay recovery divergent 2024-09-18T14:08:56.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:775: TEST_divergent_3: echo 'delay recovery divergent' 2024-09-18T14:08:56.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:776: TEST_divergent_3: ceph pg dump pgs 2024-09-18T14:08:56.688 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:08:56.689 INFO:tasks.workunit.client.0.smithi047.stdout:1.1f 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814303+0000 32'80 71:121 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:56.689 INFO:tasks.workunit.client.0.smithi047.stdout:1.1e 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.814587+0000 34'81 71:120 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:56.689 INFO:tasks.workunit.client.0.smithi047.stdout:1.1d 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814571+0000 32'80 71:121 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:56.689 INFO:tasks.workunit.client.0.smithi047.stdout:1.b 4 0 4 0 0 1691 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.787168+0000 34'81 70:132 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:56.689 INFO:tasks.workunit.client.0.smithi047.stdout:1.a 3 0 3 0 0 1208 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.813808+0000 32'80 71:107 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:56.689 INFO:tasks.workunit.client.0.smithi047.stdout:1.9 5 0 5 0 0 2174 0 0 16 66 16 active+undersized+degraded 2024-09-18T14:07:51.812741+0000 34'82 71:133 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:56.689 INFO:tasks.workunit.client.0.smithi047.stdout:1.8 5 0 5 0 0 2174 0 0 16 66 16 active+undersized+degraded 2024-09-18T14:07:51.814091+0000 34'82 71:121 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:56.689 INFO:tasks.workunit.client.0.smithi047.stdout:1.7 3 0 3 0 0 1208 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.812691+0000 32'80 71:121 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:56.689 INFO:tasks.workunit.client.0.smithi047.stdout:1.6 3 0 3 0 0 1208 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814625+0000 32'80 70:130 [0,2] 0 [0,2] 0 32'80 2024-09-18T14:06:58.202394+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:56.690 INFO:tasks.workunit.client.0.smithi047.stdout:1.5 6 0 6 0 0 2898 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.814179+0000 34'81 71:142 [2,0] 2 [2,0] 2 34'81 2024-09-18T14:07:06.420533+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T15:28:19.740089+0000 6 0 2024-09-18T14:08:56.690 INFO:tasks.workunit.client.0.smithi047.stdout:1.4 3 0 3 0 0 1208 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.813528+0000 32'80 70:118 [0,2] 0 [0,2] 0 32'80 2024-09-18T14:07:08.170467+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:56.690 INFO:tasks.workunit.client.0.smithi047.stdout:1.3 4 0 4 0 0 1691 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.815161+0000 34'81 71:134 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:56.690 INFO:tasks.workunit.client.0.smithi047.stdout:1.2 3 0 3 0 0 1208 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814414+0000 32'80 70:130 [0,2] 0 [0,2] 0 32'80 2024-09-18T14:07:04.230275+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:56.690 INFO:tasks.workunit.client.0.smithi047.stdout:1.c 4 0 4 0 0 1691 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.812888+0000 34'81 70:131 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:56.690 INFO:tasks.workunit.client.0.smithi047.stdout:1.1 4 0 4 0 0 1691 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.813144+0000 34'81 71:134 [2,0] 2 [2,0] 2 34'81 2024-09-18T14:06:56.389936+0000 34'81 2024-09-18T14:06:56.389936+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:56.690 INFO:tasks.workunit.client.0.smithi047.stdout:1.1c 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.812871+0000 32'80 70:118 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:56.690 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 3 0 3 0 0 1449 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.815935+0000 34'81 70:142 [0,2] 0 [0,2] 0 32'80 2024-09-18T14:06:54.303691+0000 32'80 2024-09-18T14:06:54.303691+0000 0 1 periodic scrub scheduled @ 2024-09-19T21:42:35.244281+0000 2 0 2024-09-18T14:08:56.690 INFO:tasks.workunit.client.0.smithi047.stdout:1.d 3 0 3 0 0 1449 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.814765+0000 34'81 71:145 [2,0] 2 [2,0] 2 34'81 2024-09-18T14:07:09.554363+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T21:30:11.231921+0000 3 0 2024-09-18T14:08:56.690 INFO:tasks.workunit.client.0.smithi047.stdout:1.e 5 0 5 0 0 1845 0 0 17 66 17 active+undersized+degraded 2024-09-18T14:07:51.815577+0000 70'83 71:122 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:56.690 INFO:tasks.workunit.client.0.smithi047.stdout:1.f 4 0 4 0 0 1932 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814986+0000 32'80 71:131 [2,0] 2 [2,0] 2 32'80 2024-09-18T14:07:00.547567+0000 32'80 2024-09-18T14:07:00.547567+0000 0 1 periodic scrub scheduled @ 2024-09-19T22:57:43.008803+0000 4 0 2024-09-18T14:08:56.690 INFO:tasks.workunit.client.0.smithi047.stdout:1.10 4 0 4 0 0 1691 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.813572+0000 34'81 70:131 [0,2] 0 [0,2] 0 34'81 2024-09-18T14:07:07.204718+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:56.690 INFO:tasks.workunit.client.0.smithi047.stdout:1.11 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:52.902633+0000 32'80 70:118 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:56.691 INFO:tasks.workunit.client.0.smithi047.stdout:1.12 1 0 1 0 0 483 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.812635+0000 32'80 70:138 [0,2] 0 [0,2] 0 32'80 2024-09-18T14:07:08.487332+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T19:22:31.668808+0000 1 0 2024-09-18T14:08:56.691 INFO:tasks.workunit.client.0.smithi047.stdout:1.13 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.813086+0000 34'81 71:134 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:56.691 INFO:tasks.workunit.client.0.smithi047.stdout:1.14 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.816330+0000 34'81 70:131 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:56.691 INFO:tasks.workunit.client.0.smithi047.stdout:1.15 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.815134+0000 32'80 71:121 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:56.691 INFO:tasks.workunit.client.0.smithi047.stdout:1.16 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814810+0000 32'80 70:130 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:56.691 INFO:tasks.workunit.client.0.smithi047.stdout:1.17 1 0 1 0 0 483 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.813569+0000 32'80 71:131 [2,0] 2 [2,0] 2 32'80 2024-09-18T14:07:05.418931+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T19:59:06.890013+0000 1 0 2024-09-18T14:08:56.691 INFO:tasks.workunit.client.0.smithi047.stdout:1.18 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.814107+0000 34'81 71:120 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:56.691 INFO:tasks.workunit.client.0.smithi047.stdout:1.19 2 0 2 0 0 966 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.813376+0000 32'80 71:129 [2,0] 2 [2,0] 2 32'80 2024-09-18T14:07:07.546410+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T20:01:59.930990+0000 2 0 2024-09-18T14:08:56.691 INFO:tasks.workunit.client.0.smithi047.stdout:1.1a 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.812774+0000 34'81 71:120 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:56.691 INFO:tasks.workunit.client.0.smithi047.stdout:1.1b 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.813412+0000 34'81 70:132 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:56.692 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:08:56.692 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:08:56.692 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:08:56.700 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: get_asok_path osd.1 2024-09-18T14:08:56.700 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:08:56.700 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:08:56.700 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:08:56.701 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:08:56.701 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:08:56.701 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32609/ceph-osd.1.asok 2024-09-18T14:08:56.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: CEPH_ARGS= 2024-09-18T14:08:56.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.32609/ceph-osd.1.asok set_recovery_delay 100000 2024-09-18T14:08:56.779 INFO:tasks.workunit.client.0.smithi047.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-18T14:08:56.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:779: TEST_divergent_3: ceph osd unset noup 2024-09-18T14:08:57.407 INFO:tasks.workunit.client.0.smithi047.stderr:noup is unset 2024-09-18T14:08:57.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:781: TEST_divergent_3: wait_for_osd up 0 2024-09-18T14:08:57.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:08:57.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:08:57.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:08:57.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:08:57.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:08:57.419 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:08:57.419 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:08:57.419 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:08:57.419 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:08:57.878 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 57 up_thru 75 down_at 49 last_clean_interval [5,40) [v2:127.0.0.1:6802/68576,v1:127.0.0.1:6803/68576] [v2:127.0.0.1:6804/68576,v1:127.0.0.1:6805/68576] exists,up 2d333a34-fa14-4e6f-aca9-f14a57e36c76 2024-09-18T14:08:57.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:08:57.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:08:57.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:08:57.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:782: TEST_divergent_3: wait_for_osd up 1 2024-09-18T14:08:57.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:08:57.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:08:57.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:08:57.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:08:57.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:08:57.880 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:08:57.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:08:57.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:08:57.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:08:58.501 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 75 up_thru 75 down_at 50 last_clean_interval [10,47) [v2:127.0.0.1:6818/71718,v1:127.0.0.1:6819/71718] [v2:127.0.0.1:6820/71718,v1:127.0.0.1:6821/71718] exists,up d16b04c1-1e01-47e1-87e5-e8058bd968e4 2024-09-18T14:08:58.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:08:58.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:08:58.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:08:58.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:783: TEST_divergent_3: wait_for_osd up 2 2024-09-18T14:08:58.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:08:58.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T14:08:58.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:08:58.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:08:58.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:08:58.502 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:08:58.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:08:58.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:08:58.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:08:58.920 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 61 up_thru 75 down_at 51 last_clean_interval [15,40) [v2:127.0.0.1:6810/69154,v1:127.0.0.1:6811/69154] [v2:127.0.0.1:6812/69154,v1:127.0.0.1:6813/69154] exists,up 20ba0a43-7c90-449e-bd64-6203deaebae3 2024-09-18T14:08:58.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:08:58.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:08:58.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:08:58.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:785: TEST_divergent_3: ceph pg dump pgs 2024-09-18T14:08:59.280 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:08:59.280 INFO:tasks.workunit.client.0.smithi047.stdout:1.1f 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814303+0000 32'80 71:121 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.280 INFO:tasks.workunit.client.0.smithi047.stdout:1.1e 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.814587+0000 34'81 71:120 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.280 INFO:tasks.workunit.client.0.smithi047.stdout:1.1d 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814571+0000 32'80 71:121 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.280 INFO:tasks.workunit.client.0.smithi047.stdout:1.b 4 0 4 0 0 1691 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.787168+0000 34'81 70:132 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.280 INFO:tasks.workunit.client.0.smithi047.stdout:1.a 3 0 3 0 0 1208 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.813808+0000 32'80 71:107 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.281 INFO:tasks.workunit.client.0.smithi047.stdout:1.9 5 0 5 0 0 2174 0 0 16 66 16 active+undersized+degraded 2024-09-18T14:07:51.812741+0000 34'82 71:133 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.281 INFO:tasks.workunit.client.0.smithi047.stdout:1.8 5 0 5 0 0 2174 0 0 16 66 16 active+undersized+degraded 2024-09-18T14:07:51.814091+0000 34'82 71:121 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.281 INFO:tasks.workunit.client.0.smithi047.stdout:1.7 3 0 3 0 0 1208 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.812691+0000 32'80 71:121 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.281 INFO:tasks.workunit.client.0.smithi047.stdout:1.6 3 0 3 0 0 1208 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814625+0000 32'80 70:130 [0,2] 0 [0,2] 0 32'80 2024-09-18T14:06:58.202394+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.281 INFO:tasks.workunit.client.0.smithi047.stdout:1.5 6 0 6 0 0 2898 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.814179+0000 34'81 71:142 [2,0] 2 [2,0] 2 34'81 2024-09-18T14:07:06.420533+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T15:28:19.740089+0000 6 0 2024-09-18T14:08:59.281 INFO:tasks.workunit.client.0.smithi047.stdout:1.4 3 0 3 0 0 1208 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.813528+0000 32'80 70:118 [0,2] 0 [0,2] 0 32'80 2024-09-18T14:07:08.170467+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.281 INFO:tasks.workunit.client.0.smithi047.stdout:1.3 4 0 4 0 0 1691 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.815161+0000 34'81 71:134 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.281 INFO:tasks.workunit.client.0.smithi047.stdout:1.2 3 0 3 0 0 1208 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814414+0000 32'80 70:130 [0,2] 0 [0,2] 0 32'80 2024-09-18T14:07:04.230275+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.281 INFO:tasks.workunit.client.0.smithi047.stdout:1.c 4 0 4 0 0 1691 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.812888+0000 34'81 70:131 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.281 INFO:tasks.workunit.client.0.smithi047.stdout:1.1 4 0 4 0 0 1691 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.813144+0000 34'81 71:134 [2,0] 2 [2,0] 2 34'81 2024-09-18T14:06:56.389936+0000 34'81 2024-09-18T14:06:56.389936+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.281 INFO:tasks.workunit.client.0.smithi047.stdout:1.1c 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.812871+0000 32'80 70:118 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.282 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 3 0 3 0 0 1449 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.815935+0000 34'81 70:142 [0,2] 0 [0,2] 0 32'80 2024-09-18T14:06:54.303691+0000 32'80 2024-09-18T14:06:54.303691+0000 0 1 periodic scrub scheduled @ 2024-09-19T21:42:35.244281+0000 2 0 2024-09-18T14:08:59.282 INFO:tasks.workunit.client.0.smithi047.stdout:1.d 3 0 3 0 0 1449 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.814765+0000 34'81 71:145 [2,0] 2 [2,0] 2 34'81 2024-09-18T14:07:09.554363+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T21:30:11.231921+0000 3 0 2024-09-18T14:08:59.282 INFO:tasks.workunit.client.0.smithi047.stdout:1.e 5 0 5 0 0 1845 0 0 17 66 17 active+undersized+degraded 2024-09-18T14:07:51.815577+0000 70'83 71:122 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.282 INFO:tasks.workunit.client.0.smithi047.stdout:1.f 4 0 4 0 0 1932 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814986+0000 32'80 71:131 [2,0] 2 [2,0] 2 32'80 2024-09-18T14:07:00.547567+0000 32'80 2024-09-18T14:07:00.547567+0000 0 1 periodic scrub scheduled @ 2024-09-19T22:57:43.008803+0000 4 0 2024-09-18T14:08:59.282 INFO:tasks.workunit.client.0.smithi047.stdout:1.10 4 0 4 0 0 1691 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.813572+0000 34'81 70:131 [0,2] 0 [0,2] 0 34'81 2024-09-18T14:07:07.204718+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.282 INFO:tasks.workunit.client.0.smithi047.stdout:1.11 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:52.902633+0000 32'80 70:118 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.282 INFO:tasks.workunit.client.0.smithi047.stdout:1.12 1 0 1 0 0 483 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.812635+0000 32'80 70:138 [0,2] 0 [0,2] 0 32'80 2024-09-18T14:07:08.487332+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T19:22:31.668808+0000 1 0 2024-09-18T14:08:59.282 INFO:tasks.workunit.client.0.smithi047.stdout:1.13 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.813086+0000 34'81 71:134 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.282 INFO:tasks.workunit.client.0.smithi047.stdout:1.14 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.816330+0000 34'81 70:131 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.282 INFO:tasks.workunit.client.0.smithi047.stdout:1.15 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.815134+0000 32'80 71:121 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.282 INFO:tasks.workunit.client.0.smithi047.stdout:1.16 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814810+0000 32'80 70:130 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.282 INFO:tasks.workunit.client.0.smithi047.stdout:1.17 1 0 1 0 0 483 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.813569+0000 32'80 71:131 [2,0] 2 [2,0] 2 32'80 2024-09-18T14:07:05.418931+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T19:59:06.890013+0000 1 0 2024-09-18T14:08:59.282 INFO:tasks.workunit.client.0.smithi047.stdout:1.18 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.814107+0000 34'81 71:120 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.283 INFO:tasks.workunit.client.0.smithi047.stdout:1.19 2 0 2 0 0 966 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.813376+0000 32'80 71:129 [2,0] 2 [2,0] 2 32'80 2024-09-18T14:07:07.546410+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T20:01:59.930990+0000 2 0 2024-09-18T14:08:59.283 INFO:tasks.workunit.client.0.smithi047.stdout:1.1a 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.812774+0000 34'81 71:120 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.283 INFO:tasks.workunit.client.0.smithi047.stdout:1.1b 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.813412+0000 34'81 70:132 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.283 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:08:59.283 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:08:59.283 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:08:59.290 INFO:tasks.workunit.client.0.smithi047.stdout:wait for peering 2024-09-18T14:08:59.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:786: TEST_divergent_3: echo 'wait for peering' 2024-09-18T14:08:59.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:787: TEST_divergent_3: ceph pg dump pgs 2024-09-18T14:08:59.648 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:08:59.648 INFO:tasks.workunit.client.0.smithi047.stdout:1.1f 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814303+0000 32'80 71:121 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.649 INFO:tasks.workunit.client.0.smithi047.stdout:1.1e 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.814587+0000 34'81 71:120 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.649 INFO:tasks.workunit.client.0.smithi047.stdout:1.1d 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814571+0000 32'80 71:121 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.649 INFO:tasks.workunit.client.0.smithi047.stdout:1.b 4 0 4 0 0 1691 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.787168+0000 34'81 70:132 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.649 INFO:tasks.workunit.client.0.smithi047.stdout:1.a 3 0 3 0 0 1208 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.813808+0000 32'80 71:107 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.649 INFO:tasks.workunit.client.0.smithi047.stdout:1.9 5 0 5 0 0 2174 0 0 16 66 16 active+undersized+degraded 2024-09-18T14:07:51.812741+0000 34'82 71:133 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.649 INFO:tasks.workunit.client.0.smithi047.stdout:1.8 5 0 5 0 0 2174 0 0 16 66 16 active+undersized+degraded 2024-09-18T14:07:51.814091+0000 34'82 71:121 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.649 INFO:tasks.workunit.client.0.smithi047.stdout:1.7 3 0 3 0 0 1208 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.812691+0000 32'80 71:121 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.649 INFO:tasks.workunit.client.0.smithi047.stdout:1.6 3 0 3 0 0 1208 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814625+0000 32'80 70:130 [0,2] 0 [0,2] 0 32'80 2024-09-18T14:06:58.202394+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.649 INFO:tasks.workunit.client.0.smithi047.stdout:1.5 6 0 6 0 0 2898 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.814179+0000 34'81 71:142 [2,0] 2 [2,0] 2 34'81 2024-09-18T14:07:06.420533+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T15:28:19.740089+0000 6 0 2024-09-18T14:08:59.649 INFO:tasks.workunit.client.0.smithi047.stdout:1.4 3 0 3 0 0 1208 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.813528+0000 32'80 70:118 [0,2] 0 [0,2] 0 32'80 2024-09-18T14:07:08.170467+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.649 INFO:tasks.workunit.client.0.smithi047.stdout:1.3 4 0 4 0 0 1691 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.815161+0000 34'81 71:134 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.649 INFO:tasks.workunit.client.0.smithi047.stdout:1.2 3 0 3 0 0 1208 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814414+0000 32'80 70:130 [0,2] 0 [0,2] 0 32'80 2024-09-18T14:07:04.230275+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.649 INFO:tasks.workunit.client.0.smithi047.stdout:1.c 4 0 4 0 0 1691 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.812888+0000 34'81 70:131 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.650 INFO:tasks.workunit.client.0.smithi047.stdout:1.1 4 0 4 0 0 1691 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.813144+0000 34'81 71:134 [2,0] 2 [2,0] 2 34'81 2024-09-18T14:06:56.389936+0000 34'81 2024-09-18T14:06:56.389936+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.650 INFO:tasks.workunit.client.0.smithi047.stdout:1.1c 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.812871+0000 32'80 70:118 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.650 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 3 0 3 0 0 1449 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.815935+0000 34'81 70:142 [0,2] 0 [0,2] 0 32'80 2024-09-18T14:06:54.303691+0000 32'80 2024-09-18T14:06:54.303691+0000 0 1 periodic scrub scheduled @ 2024-09-19T21:42:35.244281+0000 2 0 2024-09-18T14:08:59.650 INFO:tasks.workunit.client.0.smithi047.stdout:1.d 3 0 3 0 0 1449 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.814765+0000 34'81 71:145 [2,0] 2 [2,0] 2 34'81 2024-09-18T14:07:09.554363+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T21:30:11.231921+0000 3 0 2024-09-18T14:08:59.650 INFO:tasks.workunit.client.0.smithi047.stdout:1.e 5 0 5 0 0 1845 0 0 17 66 17 active+undersized+degraded 2024-09-18T14:07:51.815577+0000 70'83 71:122 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.650 INFO:tasks.workunit.client.0.smithi047.stdout:1.f 4 0 4 0 0 1932 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814986+0000 32'80 71:131 [2,0] 2 [2,0] 2 32'80 2024-09-18T14:07:00.547567+0000 32'80 2024-09-18T14:07:00.547567+0000 0 1 periodic scrub scheduled @ 2024-09-19T22:57:43.008803+0000 4 0 2024-09-18T14:08:59.650 INFO:tasks.workunit.client.0.smithi047.stdout:1.10 4 0 4 0 0 1691 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.813572+0000 34'81 70:131 [0,2] 0 [0,2] 0 34'81 2024-09-18T14:07:07.204718+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.650 INFO:tasks.workunit.client.0.smithi047.stdout:1.11 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:52.902633+0000 32'80 70:118 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.650 INFO:tasks.workunit.client.0.smithi047.stdout:1.12 1 0 1 0 0 483 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.812635+0000 32'80 70:138 [0,2] 0 [0,2] 0 32'80 2024-09-18T14:07:08.487332+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T19:22:31.668808+0000 1 0 2024-09-18T14:08:59.650 INFO:tasks.workunit.client.0.smithi047.stdout:1.13 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.813086+0000 34'81 71:134 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.650 INFO:tasks.workunit.client.0.smithi047.stdout:1.14 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.816330+0000 34'81 70:131 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.650 INFO:tasks.workunit.client.0.smithi047.stdout:1.15 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.815134+0000 32'80 71:121 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.650 INFO:tasks.workunit.client.0.smithi047.stdout:1.16 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814810+0000 32'80 70:130 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.651 INFO:tasks.workunit.client.0.smithi047.stdout:1.17 1 0 1 0 0 483 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.813569+0000 32'80 71:131 [2,0] 2 [2,0] 2 32'80 2024-09-18T14:07:05.418931+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T19:59:06.890013+0000 1 0 2024-09-18T14:08:59.651 INFO:tasks.workunit.client.0.smithi047.stdout:1.18 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.814107+0000 34'81 71:120 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.651 INFO:tasks.workunit.client.0.smithi047.stdout:1.19 2 0 2 0 0 966 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.813376+0000 32'80 71:129 [2,0] 2 [2,0] 2 32'80 2024-09-18T14:07:07.546410+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T20:01:59.930990+0000 2 0 2024-09-18T14:08:59.651 INFO:tasks.workunit.client.0.smithi047.stdout:1.1a 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.812774+0000 34'81 71:120 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.651 INFO:tasks.workunit.client.0.smithi047.stdout:1.1b 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.813412+0000 34'81 70:132 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:08:59.651 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:08:59.651 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:08:59.651 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:08:59.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:788: TEST_divergent_3: rados -p test put foo /etc/fstab 2024-09-18T14:08:59.785 INFO:tasks.workunit.client.0.smithi047.stdout:killing divergent 1 2024-09-18T14:08:59.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: echo 'killing divergent 1' 2024-09-18T14:08:59.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:793: TEST_divergent_3: ceph pg dump pgs 2024-09-18T14:09:00.142 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:09:00.142 INFO:tasks.workunit.client.0.smithi047.stdout:1.1f 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814303+0000 32'80 71:121 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:00.142 INFO:tasks.workunit.client.0.smithi047.stdout:1.1e 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.814587+0000 34'81 71:120 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:00.142 INFO:tasks.workunit.client.0.smithi047.stdout:1.1d 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814571+0000 32'80 71:121 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:00.143 INFO:tasks.workunit.client.0.smithi047.stdout:1.b 4 0 4 0 0 1691 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.787168+0000 34'81 70:132 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:00.143 INFO:tasks.workunit.client.0.smithi047.stdout:1.a 3 0 3 0 0 1208 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.813808+0000 32'80 71:107 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:00.143 INFO:tasks.workunit.client.0.smithi047.stdout:1.9 5 0 5 0 0 2174 0 0 16 66 16 active+undersized+degraded 2024-09-18T14:07:51.812741+0000 34'82 71:133 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:00.143 INFO:tasks.workunit.client.0.smithi047.stdout:1.8 5 0 5 0 0 2174 0 0 16 66 16 active+undersized+degraded 2024-09-18T14:07:51.814091+0000 34'82 71:121 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:00.143 INFO:tasks.workunit.client.0.smithi047.stdout:1.7 3 0 3 0 0 1208 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.812691+0000 32'80 71:121 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:00.143 INFO:tasks.workunit.client.0.smithi047.stdout:1.6 3 0 3 0 0 1208 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814625+0000 32'80 70:130 [0,2] 0 [0,2] 0 32'80 2024-09-18T14:06:58.202394+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:00.143 INFO:tasks.workunit.client.0.smithi047.stdout:1.5 6 0 6 0 0 2898 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.814179+0000 34'81 71:142 [2,0] 2 [2,0] 2 34'81 2024-09-18T14:07:06.420533+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T15:28:19.740089+0000 6 0 2024-09-18T14:09:00.143 INFO:tasks.workunit.client.0.smithi047.stdout:1.4 3 0 3 0 0 1208 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.813528+0000 32'80 70:118 [0,2] 0 [0,2] 0 32'80 2024-09-18T14:07:08.170467+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:00.143 INFO:tasks.workunit.client.0.smithi047.stdout:1.3 4 0 4 0 0 1691 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.815161+0000 34'81 71:134 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:00.143 INFO:tasks.workunit.client.0.smithi047.stdout:1.2 3 0 3 0 0 1208 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814414+0000 32'80 70:130 [0,2] 0 [0,2] 0 32'80 2024-09-18T14:07:04.230275+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:00.143 INFO:tasks.workunit.client.0.smithi047.stdout:1.c 4 0 4 0 0 1691 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.812888+0000 34'81 70:131 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:00.143 INFO:tasks.workunit.client.0.smithi047.stdout:1.1 4 0 4 0 0 1691 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.813144+0000 34'81 71:134 [2,0] 2 [2,0] 2 34'81 2024-09-18T14:06:56.389936+0000 34'81 2024-09-18T14:06:56.389936+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:00.143 INFO:tasks.workunit.client.0.smithi047.stdout:1.1c 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.812871+0000 32'80 70:118 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:00.143 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 3 0 3 0 0 1449 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.815935+0000 34'81 70:142 [0,2] 0 [0,2] 0 32'80 2024-09-18T14:06:54.303691+0000 32'80 2024-09-18T14:06:54.303691+0000 0 1 periodic scrub scheduled @ 2024-09-19T21:42:35.244281+0000 2 0 2024-09-18T14:09:00.144 INFO:tasks.workunit.client.0.smithi047.stdout:1.d 3 0 3 0 0 1449 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.814765+0000 34'81 71:145 [2,0] 2 [2,0] 2 34'81 2024-09-18T14:07:09.554363+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T21:30:11.231921+0000 3 0 2024-09-18T14:09:00.144 INFO:tasks.workunit.client.0.smithi047.stdout:1.e 5 0 5 0 0 1845 0 0 17 66 17 active+undersized+degraded 2024-09-18T14:07:51.815577+0000 70'83 71:122 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:00.144 INFO:tasks.workunit.client.0.smithi047.stdout:1.f 4 0 4 0 0 1932 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814986+0000 32'80 71:131 [2,0] 2 [2,0] 2 32'80 2024-09-18T14:07:00.547567+0000 32'80 2024-09-18T14:07:00.547567+0000 0 1 periodic scrub scheduled @ 2024-09-19T22:57:43.008803+0000 4 0 2024-09-18T14:09:00.144 INFO:tasks.workunit.client.0.smithi047.stdout:1.10 4 0 4 0 0 1691 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.813572+0000 34'81 70:131 [0,2] 0 [0,2] 0 34'81 2024-09-18T14:07:07.204718+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:00.144 INFO:tasks.workunit.client.0.smithi047.stdout:1.11 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:52.902633+0000 32'80 70:118 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:00.144 INFO:tasks.workunit.client.0.smithi047.stdout:1.12 1 0 1 0 0 483 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.812635+0000 32'80 70:138 [0,2] 0 [0,2] 0 32'80 2024-09-18T14:07:08.487332+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T19:22:31.668808+0000 1 0 2024-09-18T14:09:00.144 INFO:tasks.workunit.client.0.smithi047.stdout:1.13 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.813086+0000 34'81 71:134 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:00.144 INFO:tasks.workunit.client.0.smithi047.stdout:1.14 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.816330+0000 34'81 70:131 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:00.144 INFO:tasks.workunit.client.0.smithi047.stdout:1.15 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.815134+0000 32'80 71:121 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:00.144 INFO:tasks.workunit.client.0.smithi047.stdout:1.16 2 0 2 0 0 1207 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.814810+0000 32'80 70:130 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:00.144 INFO:tasks.workunit.client.0.smithi047.stdout:1.17 1 0 1 0 0 483 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.813569+0000 32'80 71:131 [2,0] 2 [2,0] 2 32'80 2024-09-18T14:07:05.418931+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T19:59:06.890013+0000 1 0 2024-09-18T14:09:00.144 INFO:tasks.workunit.client.0.smithi047.stdout:1.18 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.814107+0000 34'81 71:120 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:00.145 INFO:tasks.workunit.client.0.smithi047.stdout:1.19 2 0 2 0 0 966 0 0 14 66 14 active+undersized+degraded 2024-09-18T14:07:51.813376+0000 32'80 71:129 [2,0] 2 [2,0] 2 32'80 2024-09-18T14:07:07.546410+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T20:01:59.930990+0000 2 0 2024-09-18T14:09:00.145 INFO:tasks.workunit.client.0.smithi047.stdout:1.1a 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.812774+0000 34'81 71:120 [2,0] 2 [2,0] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:00.145 INFO:tasks.workunit.client.0.smithi047.stdout:1.1b 3 0 3 0 0 1690 0 0 15 66 15 active+undersized+degraded 2024-09-18T14:07:51.813412+0000 34'81 70:132 [0,2] 0 [0,2] 0 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:00.145 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:09:00.145 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:09:00.145 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:09:00.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:794: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd.1 2024-09-18T14:09:00.153 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:09:00.153 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:09:00.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:09:00.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:09:00.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:09:00.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:09:00.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: expfile=td/divergent-priors/exp.32609.out 2024-09-18T14:09:00.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:798: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.1b --file td/divergent-priors/exp.32609.out 2024-09-18T14:09:00.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-18T14:09:00.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T14:09:00.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T14:09:00.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T14:09:00.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-18T14:09:00.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.1b --file td/divergent-priors/exp.32609.out 2024-09-18T14:09:01.791 INFO:tasks.workunit.client.0.smithi047.stderr:Exporting 1.1b info 1.1b( v 34'81 lc 0'0 (31'66,34'81] local-lis/les=75/76 n=3 ec=33/23 lis/c=75/39 les/c/f=76/40/0 sis=75) 2024-09-18T14:09:01.809 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d9cbd1d6:::existing_64:head# 2024-09-18T14:09:01.823 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:dcc4ff82:::existing_98:head# 2024-09-18T14:09:01.840 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:dd677bd3:::existing_35:head# 2024-09-18T14:09:01.856 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:dd6d90e7:::existing_46:head# 2024-09-18T14:09:01.865 INFO:tasks.workunit.client.0.smithi047.stderr:Export successful 2024-09-18T14:09:01.866 INFO:tasks.workunit.client.0.smithi047.stdout: marking collection for removal 2024-09-18T14:09:01.866 INFO:tasks.workunit.client.0.smithi047.stdout:setting '_remove' omap key 2024-09-18T14:09:01.866 INFO:tasks.workunit.client.0.smithi047.stdout:finish_remove_pgs 1.1b_head removing 1.1b 2024-09-18T14:09:02.025 INFO:tasks.workunit.client.0.smithi047.stderr:Remove successful 2024-09-18T14:09:02.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.32609.out 2024-09-18T14:09:02.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-18T14:09:02.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T14:09:02.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T14:09:02.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T14:09:02.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-18T14:09:02.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.32609.out 2024-09-18T14:09:03.447 INFO:tasks.workunit.client.0.smithi047.stdout:get_pg_num_history pg_num_history pg_num_history(e77 pg_nums {1={23=1,33=32}} deleted_pools ) 2024-09-18T14:09:03.447 INFO:tasks.workunit.client.0.smithi047.stdout:Importing pgid 1.1b 2024-09-18T14:09:03.447 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d9cbd1d6:::existing_64:head# 2024-09-18T14:09:03.447 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:09:03.520 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:dcc4ff82:::existing_98:head# 2024-09-18T14:09:03.520 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:09:03.553 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:dd677bd3:::existing_35:head# 2024-09-18T14:09:03.553 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:09:03.587 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:dd6d90e7:::existing_46:head# 2024-09-18T14:09:03.587 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T14:09:03.620 INFO:tasks.workunit.client.0.smithi047.stdout:write_pg epoch 76 info 1.1b( v 34'81 lc 0'0 (31'66,34'81] local-lis/les=75/76 n=3 ec=33/23 lis/c=75/39 les/c/f=76/40/0 sis=75) 2024-09-18T14:09:03.662 INFO:tasks.workunit.client.0.smithi047.stdout:Import successful 2024-09-18T14:09:04.178 INFO:tasks.workunit.client.0.smithi047.stdout:reviving divergent 1 2024-09-18T14:09:04.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:801: TEST_divergent_3: echo 'reviving divergent 1' 2024-09-18T14:09:04.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:802: TEST_divergent_3: ceph pg dump pgs 2024-09-18T14:09:04.545 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:09:04.545 INFO:tasks.workunit.client.0.smithi047.stdout:1.1f 2 0 0 0 0 1207 0 0 14 66 14 active+clean 2024-09-18T14:08:58.567946+0000 32'80 77:134 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:04.545 INFO:tasks.workunit.client.0.smithi047.stdout:1.1e 3 0 0 0 0 1690 0 0 15 66 15 active+clean 2024-09-18T14:08:58.106739+0000 34'81 76:136 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:04.545 INFO:tasks.workunit.client.0.smithi047.stdout:1.1d 2 0 0 0 0 1207 0 0 14 66 14 active+clean 2024-09-18T14:08:58.566907+0000 32'80 77:134 [2,1,0] 2 [2,1,0] 2 32'80 2024-09-18T14:07:18.379714+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:04.545 INFO:tasks.workunit.client.0.smithi047.stdout:1.b 4 0 0 0 0 1691 0 0 15 66 15 active+clean 2024-09-18T14:08:58.107534+0000 34'81 76:134 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:04.545 INFO:tasks.workunit.client.0.smithi047.stdout:1.a 3 0 0 0 0 1208 0 0 14 66 14 active+clean 2024-09-18T14:08:58.106542+0000 32'80 76:123 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:04.545 INFO:tasks.workunit.client.0.smithi047.stdout:1.9 5 1 1 0 0 2174 0 0 16 66 16 active+recovery_wait+degraded 2024-09-18T14:08:58.105859+0000 34'82 76:138 [1,2,0] 1 [1,2,0] 1 40'83 2024-09-18T14:07:11.436862+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:04.545 INFO:tasks.workunit.client.0.smithi047.stdout:1.8 5 0 0 0 0 2174 0 0 16 66 16 active+clean 2024-09-18T14:08:58.107093+0000 34'82 76:155 [1,2,0] 1 [1,2,0] 1 34'82 2024-09-18T14:07:15.131268+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:04.546 INFO:tasks.workunit.client.0.smithi047.stdout:1.7 3 0 0 0 0 1208 0 0 14 66 14 active+clean 2024-09-18T14:08:58.107799+0000 32'80 76:133 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:04.546 INFO:tasks.workunit.client.0.smithi047.stdout:1.6 3 0 0 0 0 1208 0 0 14 66 14 active+clean 2024-09-18T14:08:58.107891+0000 32'80 76:145 [1,0,2] 1 [1,0,2] 1 32'80 2024-09-18T14:06:58.202394+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:04.546 INFO:tasks.workunit.client.0.smithi047.stdout:1.5 6 0 0 0 0 2898 0 0 15 66 15 active+clean 2024-09-18T14:08:58.566924+0000 34'81 77:155 [2,0,1] 2 [2,0,1] 2 34'81 2024-09-18T14:07:06.420533+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T20:51:32.180396+0000 6 0 2024-09-18T14:09:04.546 INFO:tasks.workunit.client.0.smithi047.stdout:1.4 3 0 0 0 0 1208 0 0 14 66 14 active+clean 2024-09-18T14:08:58.109305+0000 32'80 76:133 [1,0,2] 1 [1,0,2] 1 32'80 2024-09-18T14:07:08.170467+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:04.546 INFO:tasks.workunit.client.0.smithi047.stdout:1.3 4 0 0 0 0 1691 0 0 15 66 15 active+clean 2024-09-18T14:08:58.107294+0000 34'81 76:134 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:04.546 INFO:tasks.workunit.client.0.smithi047.stdout:1.2 3 0 0 0 0 1208 0 0 14 66 14 active+clean 2024-09-18T14:08:58.106039+0000 32'80 76:145 [0,1,2] 0 [0,1,2] 0 32'80 2024-09-18T14:07:18.506303+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:04.546 INFO:tasks.workunit.client.0.smithi047.stdout:1.c 4 0 0 0 0 1691 0 0 15 66 15 active+clean 2024-09-18T14:08:58.107974+0000 34'81 76:137 [1,0,2] 1 [1,0,2] 1 34'81 2024-09-18T14:07:12.133156+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:04.546 INFO:tasks.workunit.client.0.smithi047.stdout:1.1 4 0 0 0 0 1691 0 0 15 66 15 active+clean 2024-09-18T14:08:58.568154+0000 34'81 77:147 [2,0,1] 2 [2,0,1] 2 34'81 2024-09-18T14:07:13.379437+0000 34'81 2024-09-18T14:06:56.389936+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:04.546 INFO:tasks.workunit.client.0.smithi047.stdout:1.1c 2 0 0 0 0 1207 0 0 14 66 14 active+clean 2024-09-18T14:08:58.105120+0000 32'80 76:139 [1,0,2] 1 [1,0,2] 1 32'80 2024-09-18T14:07:16.092318+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:04.546 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 3 0 0 0 0 1449 0 0 15 66 15 active+clean 2024-09-18T14:08:58.106926+0000 34'81 76:133 [1,0,2] 1 [1,0,2] 1 32'80 2024-09-18T14:06:54.303691+0000 32'80 2024-09-18T14:06:54.303691+0000 0 1 periodic scrub scheduled @ 2024-09-19T17:25:17.224651+0000 2 0 2024-09-18T14:09:04.546 INFO:tasks.workunit.client.0.smithi047.stdout:1.d 3 0 0 0 0 1449 0 0 15 66 15 active+clean 2024-09-18T14:08:58.506466+0000 34'81 77:158 [2,1,0] 2 [2,1,0] 2 34'81 2024-09-18T14:07:09.554363+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T15:27:45.900532+0000 3 0 2024-09-18T14:09:04.546 INFO:tasks.workunit.client.0.smithi047.stdout:1.e 5 1 1 0 0 1845 0 0 17 66 17 active+recovery_wait+degraded 2024-09-18T14:08:58.109667+0000 70'83 76:146 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:04.547 INFO:tasks.workunit.client.0.smithi047.stdout:1.f 4 0 0 0 0 1932 0 0 14 66 14 active+clean 2024-09-18T14:08:58.506406+0000 32'80 77:144 [2,0,1] 2 [2,0,1] 2 32'80 2024-09-18T14:07:00.547567+0000 32'80 2024-09-18T14:07:00.547567+0000 0 1 periodic scrub scheduled @ 2024-09-19T23:25:03.083392+0000 4 0 2024-09-18T14:09:04.547 INFO:tasks.workunit.client.0.smithi047.stdout:1.10 4 0 0 0 0 1691 0 0 15 66 15 active+clean 2024-09-18T14:08:58.006494+0000 34'81 76:146 [0,2,1] 0 [0,2,1] 0 34'81 2024-09-18T14:07:07.204718+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:04.547 INFO:tasks.workunit.client.0.smithi047.stdout:1.11 2 0 1 0 0 1207 0 0 14 66 14 active+recovering+degraded 2024-09-18T14:08:58.110389+0000 32'80 76:133 [0,1,2] 0 [0,1,2] 0 40'81 2024-09-18T14:07:15.411220+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:04.547 INFO:tasks.workunit.client.0.smithi047.stdout:1.12 1 0 0 0 0 483 0 0 14 66 14 active+clean 2024-09-18T14:08:58.105039+0000 32'80 76:153 [0,1,2] 0 [0,1,2] 0 32'80 2024-09-18T14:07:08.487332+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T18:45:41.481278+0000 1 0 2024-09-18T14:09:04.547 INFO:tasks.workunit.client.0.smithi047.stdout:1.13 3 0 0 0 0 1690 0 0 15 66 15 active+clean 2024-09-18T14:08:58.106192+0000 34'81 76:134 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:04.547 INFO:tasks.workunit.client.0.smithi047.stdout:1.14 3 0 0 0 0 1690 0 0 15 66 15 active+clean 2024-09-18T14:08:58.107129+0000 34'81 76:122 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:04.547 INFO:tasks.workunit.client.0.smithi047.stdout:1.15 2 0 0 0 0 1207 0 0 14 66 14 active+clean 2024-09-18T14:08:58.567760+0000 32'80 77:134 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:04.547 INFO:tasks.workunit.client.0.smithi047.stdout:1.16 2 0 0 0 0 1207 0 0 14 66 14 active+clean 2024-09-18T14:08:58.109550+0000 32'80 76:135 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:04.547 INFO:tasks.workunit.client.0.smithi047.stdout:1.17 2 0 0 0 0 1207 0 0 14 66 14 active+clean 2024-09-18T14:08:58.106894+0000 32'80 76:133 [1,2,0] 1 [1,2,0] 1 32'80 2024-09-18T14:07:05.418931+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:04.547 INFO:tasks.workunit.client.0.smithi047.stdout:1.18 3 0 0 0 0 1690 0 0 15 66 15 active+clean 2024-09-18T14:08:58.109482+0000 34'81 76:144 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:04.547 INFO:tasks.workunit.client.0.smithi047.stdout:1.19 2 0 0 0 0 1207 0 0 14 66 14 active+clean 2024-09-18T14:08:58.105027+0000 32'80 76:123 [1,2,0] 1 [1,2,0] 1 32'80 2024-09-18T14:07:07.546410+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:04.547 INFO:tasks.workunit.client.0.smithi047.stdout:1.1a 3 0 0 0 0 1690 0 0 15 66 15 active+clean 2024-09-18T14:08:58.107715+0000 34'81 76:124 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:04.548 INFO:tasks.workunit.client.0.smithi047.stdout:1.1b 3 1 1 0 0 1690 0 0 15 66 15 active+recovery_wait+degraded 2024-09-18T14:08:58.109339+0000 34'81 76:134 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:04.548 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:09:04.548 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:09:04.548 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:09:04.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:803: TEST_divergent_3: activate_osd td/divergent-priors 1 2024-09-18T14:09:04.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-18T14:09:04.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:09:04.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T14:09:04.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:09:04.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-09-18T14:09:04.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T14:09:04.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:09:04.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:09:04.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:09:04.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-18T14:09:04.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-18T14:09:04.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:09:04.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:09:04.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-18T14:09:04.559 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:09:04.559 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:09:04.559 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:09:04.559 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:09:04.559 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:09:04.559 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:09:04.559 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:09:04.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:09:04.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:09:04.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-18T14:09:04.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-18T14:09:04.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:09:04.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:09:04.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:09:04.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:09:04.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:09:04.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:09:04.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-09-18T14:09:04.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T14:09:04.560 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T14:09:04.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:09:04.561 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-09-18T14:09:04.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T14:09:04.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:09:04.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:09:04.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:09:04.588 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:09:04.588+0000 7fd36f636440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:09:04.590 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:09:04.590+0000 7fd36f636440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:09:04.591 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:09:04.591+0000 7fd36f636440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:09:04.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T14:09:04.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:09:04.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:09:04.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:09:04.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:09:04.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:09:04.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:09:04.943 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:09:04.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:09:04.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:09:05.108 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:09:05.108+0000 7fd36f636440 -1 Falling back to public interface 2024-09-18T14:09:05.314 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 75 up_thru 75 down_at 50 last_clean_interval [10,47) [v2:127.0.0.1:6818/71718,v1:127.0.0.1:6819/71718] [v2:127.0.0.1:6820/71718,v1:127.0.0.1:6821/71718] exists,up d16b04c1-1e01-47e1-87e5-e8058bd968e4 2024-09-18T14:09:05.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:09:05.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:09:05.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:09:05.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: wait_for_osd up 1 2024-09-18T14:09:05.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:09:05.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:09:05.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:09:05.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:09:05.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:09:05.316 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:09:05.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:09:05.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:09:05.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:09:05.737 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 75 up_thru 75 down_at 50 last_clean_interval [10,47) [v2:127.0.0.1:6818/71718,v1:127.0.0.1:6819/71718] [v2:127.0.0.1:6820/71718,v1:127.0.0.1:6821/71718] exists,up d16b04c1-1e01-47e1-87e5-e8058bd968e4 2024-09-18T14:09:05.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:09:05.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:09:05.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:09:05.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: sleep 20 2024-09-18T14:09:07.882 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:09:07.882+0000 7fd36f636440 -1 osd.1 77 log_to_monitors true 2024-09-18T14:09:25.740 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: get_asok_path osd.1 2024-09-18T14:09:25.741 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:09:25.741 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:09:25.741 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:09:25.741 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:09:25.741 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:09:25.744 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32609/ceph-osd.1.asok 2024-09-18T14:09:25.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: CEPH_ARGS= 2024-09-18T14:09:25.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.32609/ceph-osd.1.asok dump_ops_in_flight 2024-09-18T14:09:25.825 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T14:09:25.825 INFO:tasks.workunit.client.0.smithi047.stdout: "ops": [], 2024-09-18T14:09:25.825 INFO:tasks.workunit.client.0.smithi047.stdout: "num_ops": 0 2024-09-18T14:09:25.825 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T14:09:25.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:809: TEST_divergent_3: echo 'allowing recovery' 2024-09-18T14:09:25.833 INFO:tasks.workunit.client.0.smithi047.stdout:allowing recovery 2024-09-18T14:09:25.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: ceph pg dump pgs 2024-09-18T14:09:26.191 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:09:26.191 INFO:tasks.workunit.client.0.smithi047.stdout:1.1f 3 0 0 0 0 1449 0 0 14 66 14 active+clean 2024-09-18T14:09:13.455551+0000 32'80 89:170 [2,0,1] 2 [2,0,1] 2 32'80 2024-09-18T14:09:13.455489+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T22:06:37.074912+0000 3 0 2024-09-18T14:09:26.191 INFO:tasks.workunit.client.0.smithi047.stdout:1.1e 3 0 0 0 0 1690 0 0 15 66 15 active+clean 2024-09-18T14:09:09.864459+0000 34'81 88:155 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:26.191 INFO:tasks.workunit.client.0.smithi047.stdout:1.1d 4 0 0 0 0 1932 0 0 14 66 14 active+clean 2024-09-18T14:09:10.464127+0000 32'80 89:170 [2,1,0] 2 [2,1,0] 2 32'80 2024-09-18T14:09:10.464063+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T23:44:24.630162+0000 4 0 2024-09-18T14:09:26.191 INFO:tasks.workunit.client.0.smithi047.stdout:1.b 4 0 0 0 0 1691 0 0 15 66 15 active+clean 2024-09-18T14:09:09.863524+0000 34'81 88:153 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:26.191 INFO:tasks.workunit.client.0.smithi047.stdout:1.a 3 0 0 0 0 1208 0 0 14 66 14 active+clean 2024-09-18T14:09:09.864252+0000 32'80 88:142 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:26.191 INFO:tasks.workunit.client.0.smithi047.stdout:1.9 5 0 0 0 0 2174 0 0 16 66 16 active+clean 2024-09-18T14:09:10.061156+0000 34'82 88:162 [1,2,0] 1 [1,2,0] 1 40'83 2024-09-18T14:07:11.436862+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:26.191 INFO:tasks.workunit.client.0.smithi047.stdout:1.8 3 0 0 0 0 1449 0 0 16 66 16 active+clean 2024-09-18T14:09:12.894866+0000 34'82 88:184 [1,2,0] 1 [1,2,0] 1 34'82 2024-09-18T14:09:12.894756+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T17:08:01.287445+0000 3 0 2024-09-18T14:09:26.192 INFO:tasks.workunit.client.0.smithi047.stdout:1.7 3 0 0 0 0 1208 0 0 14 66 14 active+clean 2024-09-18T14:09:09.867312+0000 32'80 88:152 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:26.192 INFO:tasks.workunit.client.0.smithi047.stdout:1.6 4 0 0 0 0 1691 0 0 15 66 15 active+clean 2024-09-18T14:09:09.864852+0000 77'81 88:165 [1,0,2] 1 [1,0,2] 1 32'80 2024-09-18T14:06:58.202394+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:26.192 INFO:tasks.workunit.client.0.smithi047.stdout:1.5 6 0 0 0 0 2898 0 0 15 66 15 active+clean 2024-09-18T14:09:09.861119+0000 34'81 89:181 [2,0,1] 2 [2,0,1] 2 34'81 2024-09-18T14:07:06.420533+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T18:28:23.207667+0000 6 0 2024-09-18T14:09:26.192 INFO:tasks.workunit.client.0.smithi047.stdout:1.4 3 0 0 0 0 1208 0 0 14 66 14 active+clean 2024-09-18T14:09:09.868313+0000 32'80 88:152 [1,0,2] 1 [1,0,2] 1 32'80 2024-09-18T14:07:08.170467+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:26.192 INFO:tasks.workunit.client.0.smithi047.stdout:1.3 5 0 0 0 0 2415 0 0 15 66 15 active+clean 2024-09-18T14:09:11.849844+0000 34'81 88:163 [1,2,0] 1 [1,2,0] 1 34'81 2024-09-18T14:09:11.849768+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T20:48:57.473210+0000 5 0 2024-09-18T14:09:26.192 INFO:tasks.workunit.client.0.smithi047.stdout:1.2 1 0 0 0 0 483 0 0 14 66 14 active+clean 2024-09-18T14:09:13.383422+0000 32'80 88:178 [0,1,2] 0 [0,1,2] 0 32'80 2024-09-18T14:09:13.383359+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T23:19:09.219612+0000 1 0 2024-09-18T14:09:26.192 INFO:tasks.workunit.client.0.smithi047.stdout:1.c 3 0 0 0 0 1449 0 0 15 66 15 active+clean 2024-09-18T14:09:18.880644+0000 34'81 88:166 [1,0,2] 1 [1,0,2] 1 34'81 2024-09-18T14:09:18.880578+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T19:37:20.279487+0000 3 0 2024-09-18T14:09:26.192 INFO:tasks.workunit.client.0.smithi047.stdout:1.1 2 0 0 0 0 966 0 0 15 66 15 active+clean 2024-09-18T14:09:12.484774+0000 34'81 89:183 [2,0,1] 2 [2,0,1] 2 34'81 2024-09-18T14:09:12.484713+0000 34'81 2024-09-18T14:06:56.389936+0000 0 1 periodic scrub scheduled @ 2024-09-19T16:03:29.191419+0000 2 0 2024-09-18T14:09:26.192 INFO:tasks.workunit.client.0.smithi047.stdout:1.1c 2 0 0 0 0 1207 0 0 14 66 14 active+clean 2024-09-18T14:09:09.866424+0000 32'80 88:158 [1,0,2] 1 [1,0,2] 1 32'80 2024-09-18T14:07:16.092318+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:26.192 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 3 0 0 0 0 1449 0 0 15 66 15 active+clean 2024-09-18T14:09:09.864977+0000 34'81 88:152 [1,0,2] 1 [1,0,2] 1 32'80 2024-09-18T14:06:54.303691+0000 32'80 2024-09-18T14:06:54.303691+0000 0 1 periodic scrub scheduled @ 2024-09-20T01:12:40.953985+0000 2 0 2024-09-18T14:09:26.192 INFO:tasks.workunit.client.0.smithi047.stdout:1.d 3 0 0 0 0 1449 0 0 15 66 15 active+clean 2024-09-18T14:09:09.866585+0000 34'81 89:184 [2,1,0] 2 [2,1,0] 2 34'81 2024-09-18T14:07:09.554363+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T21:29:27.939739+0000 3 0 2024-09-18T14:09:26.192 INFO:tasks.workunit.client.0.smithi047.stdout:1.e 5 0 0 0 0 1845 0 0 17 66 17 active+clean 2024-09-18T14:09:10.287499+0000 70'83 88:170 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:26.193 INFO:tasks.workunit.client.0.smithi047.stdout:1.f 4 0 0 0 0 1932 0 0 14 66 14 active+clean 2024-09-18T14:09:09.861308+0000 32'80 89:170 [2,0,1] 2 [2,0,1] 2 32'80 2024-09-18T14:07:00.547567+0000 32'80 2024-09-18T14:07:00.547567+0000 0 1 periodic scrub scheduled @ 2024-09-20T02:05:36.953009+0000 4 0 2024-09-18T14:09:26.193 INFO:tasks.workunit.client.0.smithi047.stdout:1.10 3 0 0 0 0 1449 0 0 15 66 15 active+clean 2024-09-18T14:09:14.414718+0000 34'81 88:179 [0,2,1] 0 [0,2,1] 0 34'81 2024-09-18T14:09:14.414650+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T16:33:49.238111+0000 3 0 2024-09-18T14:09:26.193 INFO:tasks.workunit.client.0.smithi047.stdout:1.11 3 0 0 0 0 1449 0 0 14 66 14 active+clean+inconsistent 2024-09-18T14:09:10.387193+0000 32'80 88:166 [0,1,2] 0 [0,1,2] 0 32'80 2024-09-18T14:09:10.387126+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T23:57:51.498355+0000 3 0 2024-09-18T14:09:26.193 INFO:tasks.workunit.client.0.smithi047.stdout:1.12 1 0 0 0 0 483 0 0 14 66 14 active+clean 2024-09-18T14:09:09.861386+0000 32'80 88:176 [0,1,2] 0 [0,1,2] 0 32'80 2024-09-18T14:07:08.487332+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-20T00:54:09.481852+0000 1 0 2024-09-18T14:09:26.193 INFO:tasks.workunit.client.0.smithi047.stdout:1.13 3 0 0 0 0 1690 0 0 15 66 15 active+clean 2024-09-18T14:09:17.384966+0000 34'81 88:163 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:26.193 INFO:tasks.workunit.client.0.smithi047.stdout:1.14 3 0 0 0 0 1690 0 0 15 66 15 active+clean 2024-09-18T14:09:19.558009+0000 34'81 89:121 [2,0,1] 2 [2,0,1] 2 34'81 2024-09-18T14:09:14.823462+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:26.193 INFO:tasks.workunit.client.0.smithi047.stdout:1.15 2 0 0 0 0 1207 0 0 14 66 14 active+clean 2024-09-18T14:09:21.655916+0000 32'80 89:116 [1,0,2] 1 [1,0,2] 1 32'80 2024-09-18T14:09:16.496365+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:26.193 INFO:tasks.workunit.client.0.smithi047.stdout:1.16 4 0 0 0 0 1932 0 0 14 66 14 active+clean 2024-09-18T14:09:15.872540+0000 32'80 88:164 [1,0,2] 1 [1,0,2] 1 32'80 2024-09-18T14:09:15.872433+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T22:36:42.916615+0000 4 0 2024-09-18T14:09:26.193 INFO:tasks.workunit.client.0.smithi047.stdout:1.17 2 0 0 0 0 1207 0 0 14 66 14 active+clean 2024-09-18T14:09:09.866663+0000 32'80 88:152 [1,2,0] 1 [1,2,0] 1 32'80 2024-09-18T14:07:05.418931+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:26.193 INFO:tasks.workunit.client.0.smithi047.stdout:1.18 3 0 0 0 0 1690 0 0 15 66 15 active+clean 2024-09-18T14:09:09.868898+0000 34'81 88:163 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:26.193 INFO:tasks.workunit.client.0.smithi047.stdout:1.19 2 0 0 0 0 1207 0 0 14 66 14 active+clean 2024-09-18T14:09:09.863400+0000 32'80 88:142 [1,2,0] 1 [1,2,0] 1 32'80 2024-09-18T14:07:07.546410+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:26.193 INFO:tasks.workunit.client.0.smithi047.stdout:1.1a 3 0 0 0 0 1690 0 0 15 66 15 active+clean 2024-09-18T14:09:09.863739+0000 34'81 88:143 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:26.194 INFO:tasks.workunit.client.0.smithi047.stdout:1.1b 3 0 0 0 0 1690 0 0 15 66 15 active+clean 2024-09-18T14:09:10.178410+0000 34'81 88:158 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:26.194 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:09:26.194 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:09:26.194 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:09:26.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-09-18T14:09:26.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-18T14:09:26.308 INFO:tasks.workunit.client.0.smithi047.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-18T14:09:26.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-09-18T14:09:26.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-18T14:09:26.424 INFO:tasks.workunit.client.0.smithi047.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-18T14:09:26.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-09-18T14:09:26.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-18T14:09:26.541 INFO:tasks.workunit.client.0.smithi047.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-18T14:09:26.549 INFO:tasks.workunit.client.0.smithi047.stdout:reading divergent objects 2024-09-18T14:09:26.549 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:817: TEST_divergent_3: echo 'reading divergent objects' 2024-09-18T14:09:26.549 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:818: TEST_divergent_3: ceph pg dump pgs 2024-09-18T14:09:26.906 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:09:26.907 INFO:tasks.workunit.client.0.smithi047.stdout:1.1f 3 0 0 0 0 1449 0 0 14 66 14 active+clean 2024-09-18T14:09:13.455551+0000 32'80 89:170 [2,0,1] 2 [2,0,1] 2 32'80 2024-09-18T14:09:13.455489+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T22:06:37.074912+0000 3 0 2024-09-18T14:09:26.907 INFO:tasks.workunit.client.0.smithi047.stdout:1.1e 3 0 0 0 0 1690 0 0 15 66 15 active+clean 2024-09-18T14:09:09.864459+0000 34'81 88:155 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:26.907 INFO:tasks.workunit.client.0.smithi047.stdout:1.1d 4 0 0 0 0 1932 0 0 14 66 14 active+clean 2024-09-18T14:09:10.464127+0000 32'80 89:170 [2,1,0] 2 [2,1,0] 2 32'80 2024-09-18T14:09:10.464063+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T23:44:24.630162+0000 4 0 2024-09-18T14:09:26.907 INFO:tasks.workunit.client.0.smithi047.stdout:1.b 4 0 0 0 0 1691 0 0 15 66 15 active+clean 2024-09-18T14:09:09.863524+0000 34'81 88:153 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:26.907 INFO:tasks.workunit.client.0.smithi047.stdout:1.a 3 0 0 0 0 1208 0 0 14 66 14 active+clean 2024-09-18T14:09:09.864252+0000 32'80 88:142 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:26.907 INFO:tasks.workunit.client.0.smithi047.stdout:1.9 5 0 0 0 0 2174 0 0 16 66 16 active+clean 2024-09-18T14:09:10.061156+0000 34'82 88:162 [1,2,0] 1 [1,2,0] 1 40'83 2024-09-18T14:07:11.436862+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:26.907 INFO:tasks.workunit.client.0.smithi047.stdout:1.8 3 0 0 0 0 1449 0 0 16 66 16 active+clean 2024-09-18T14:09:12.894866+0000 34'82 88:184 [1,2,0] 1 [1,2,0] 1 34'82 2024-09-18T14:09:12.894756+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T17:08:01.287445+0000 3 0 2024-09-18T14:09:26.907 INFO:tasks.workunit.client.0.smithi047.stdout:1.7 3 0 0 0 0 1208 0 0 14 66 14 active+clean 2024-09-18T14:09:09.867312+0000 32'80 88:152 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:26.907 INFO:tasks.workunit.client.0.smithi047.stdout:1.6 4 0 0 0 0 1691 0 0 15 66 15 active+clean 2024-09-18T14:09:09.864852+0000 77'81 88:165 [1,0,2] 1 [1,0,2] 1 32'80 2024-09-18T14:06:58.202394+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:26.907 INFO:tasks.workunit.client.0.smithi047.stdout:1.5 6 0 0 0 0 2898 0 0 15 66 15 active+clean 2024-09-18T14:09:09.861119+0000 34'81 89:181 [2,0,1] 2 [2,0,1] 2 34'81 2024-09-18T14:07:06.420533+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T18:28:23.207667+0000 6 0 2024-09-18T14:09:26.908 INFO:tasks.workunit.client.0.smithi047.stdout:1.4 3 0 0 0 0 1208 0 0 14 66 14 active+clean 2024-09-18T14:09:09.868313+0000 32'80 88:152 [1,0,2] 1 [1,0,2] 1 32'80 2024-09-18T14:07:08.170467+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:26.908 INFO:tasks.workunit.client.0.smithi047.stdout:1.3 5 0 0 0 0 2415 0 0 15 66 15 active+clean 2024-09-18T14:09:11.849844+0000 34'81 88:163 [1,2,0] 1 [1,2,0] 1 34'81 2024-09-18T14:09:11.849768+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T20:48:57.473210+0000 5 0 2024-09-18T14:09:26.908 INFO:tasks.workunit.client.0.smithi047.stdout:1.2 1 0 0 0 0 483 0 0 14 66 14 active+clean 2024-09-18T14:09:13.383422+0000 32'80 88:178 [0,1,2] 0 [0,1,2] 0 32'80 2024-09-18T14:09:13.383359+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T23:19:09.219612+0000 1 0 2024-09-18T14:09:26.908 INFO:tasks.workunit.client.0.smithi047.stdout:1.c 3 0 0 0 0 1449 0 0 15 66 15 active+clean 2024-09-18T14:09:18.880644+0000 34'81 88:166 [1,0,2] 1 [1,0,2] 1 34'81 2024-09-18T14:09:18.880578+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T19:37:20.279487+0000 3 0 2024-09-18T14:09:26.908 INFO:tasks.workunit.client.0.smithi047.stdout:1.1 2 0 0 0 0 966 0 0 15 66 15 active+clean 2024-09-18T14:09:12.484774+0000 34'81 89:183 [2,0,1] 2 [2,0,1] 2 34'81 2024-09-18T14:09:12.484713+0000 34'81 2024-09-18T14:06:56.389936+0000 0 1 periodic scrub scheduled @ 2024-09-19T16:03:29.191419+0000 2 0 2024-09-18T14:09:26.908 INFO:tasks.workunit.client.0.smithi047.stdout:1.1c 2 0 0 0 0 1207 0 0 14 66 14 active+clean 2024-09-18T14:09:09.866424+0000 32'80 88:158 [1,0,2] 1 [1,0,2] 1 32'80 2024-09-18T14:07:16.092318+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:26.908 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 3 0 0 0 0 1449 0 0 15 66 15 active+clean 2024-09-18T14:09:09.864977+0000 34'81 88:152 [1,0,2] 1 [1,0,2] 1 32'80 2024-09-18T14:06:54.303691+0000 32'80 2024-09-18T14:06:54.303691+0000 0 1 periodic scrub scheduled @ 2024-09-20T01:12:40.953985+0000 2 0 2024-09-18T14:09:26.908 INFO:tasks.workunit.client.0.smithi047.stdout:1.d 3 0 0 0 0 1449 0 0 15 66 15 active+clean 2024-09-18T14:09:09.866585+0000 34'81 89:184 [2,1,0] 2 [2,1,0] 2 34'81 2024-09-18T14:07:09.554363+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T21:29:27.939739+0000 3 0 2024-09-18T14:09:26.908 INFO:tasks.workunit.client.0.smithi047.stdout:1.e 5 0 0 0 0 1845 0 0 17 66 17 active+clean 2024-09-18T14:09:10.287499+0000 70'83 88:170 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:26.908 INFO:tasks.workunit.client.0.smithi047.stdout:1.f 4 0 0 0 0 1932 0 0 14 66 14 active+clean 2024-09-18T14:09:09.861308+0000 32'80 89:170 [2,0,1] 2 [2,0,1] 2 32'80 2024-09-18T14:07:00.547567+0000 32'80 2024-09-18T14:07:00.547567+0000 0 1 periodic scrub scheduled @ 2024-09-20T02:05:36.953009+0000 4 0 2024-09-18T14:09:26.908 INFO:tasks.workunit.client.0.smithi047.stdout:1.10 3 0 0 0 0 1449 0 0 15 66 15 active+clean 2024-09-18T14:09:14.414718+0000 34'81 88:179 [0,2,1] 0 [0,2,1] 0 34'81 2024-09-18T14:09:14.414650+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T16:33:49.238111+0000 3 0 2024-09-18T14:09:26.908 INFO:tasks.workunit.client.0.smithi047.stdout:1.11 3 0 0 0 0 1449 0 0 14 66 14 active+clean+inconsistent 2024-09-18T14:09:10.387193+0000 32'80 88:166 [0,1,2] 0 [0,1,2] 0 32'80 2024-09-18T14:09:10.387126+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T23:57:51.498355+0000 3 0 2024-09-18T14:09:26.908 INFO:tasks.workunit.client.0.smithi047.stdout:1.12 1 0 0 0 0 483 0 0 14 66 14 active+clean 2024-09-18T14:09:09.861386+0000 32'80 88:176 [0,1,2] 0 [0,1,2] 0 32'80 2024-09-18T14:07:08.487332+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-20T00:54:09.481852+0000 1 0 2024-09-18T14:09:26.909 INFO:tasks.workunit.client.0.smithi047.stdout:1.13 3 0 0 0 0 1690 0 0 15 66 15 active+clean 2024-09-18T14:09:17.384966+0000 34'81 88:163 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:26.909 INFO:tasks.workunit.client.0.smithi047.stdout:1.14 3 0 0 0 0 1690 0 0 15 66 15 active+clean 2024-09-18T14:09:19.558009+0000 34'81 89:121 [2,0,1] 2 [2,0,1] 2 34'81 2024-09-18T14:09:14.823462+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:26.909 INFO:tasks.workunit.client.0.smithi047.stdout:1.15 2 0 0 0 0 1207 0 0 14 66 14 active+clean 2024-09-18T14:09:21.655916+0000 32'80 89:116 [1,0,2] 1 [1,0,2] 1 32'80 2024-09-18T14:09:16.496365+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:26.909 INFO:tasks.workunit.client.0.smithi047.stdout:1.16 4 0 0 0 0 1932 0 0 14 66 14 active+clean 2024-09-18T14:09:15.872540+0000 32'80 88:164 [1,0,2] 1 [1,0,2] 1 32'80 2024-09-18T14:09:15.872433+0000 0'0 2024-09-18T14:06:27.302596+0000 0 1 periodic scrub scheduled @ 2024-09-19T22:36:42.916615+0000 4 0 2024-09-18T14:09:26.909 INFO:tasks.workunit.client.0.smithi047.stdout:1.17 2 0 0 0 0 1207 0 0 14 66 14 active+clean 2024-09-18T14:09:09.866663+0000 32'80 88:152 [1,2,0] 1 [1,2,0] 1 32'80 2024-09-18T14:07:05.418931+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:26.909 INFO:tasks.workunit.client.0.smithi047.stdout:1.18 3 0 0 0 0 1690 0 0 15 66 15 active+clean 2024-09-18T14:09:09.868898+0000 34'81 88:163 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:26.909 INFO:tasks.workunit.client.0.smithi047.stdout:1.19 2 0 0 0 0 1207 0 0 14 66 14 active+clean 2024-09-18T14:09:09.863400+0000 32'80 88:142 [1,2,0] 1 [1,2,0] 1 32'80 2024-09-18T14:07:07.546410+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:26.909 INFO:tasks.workunit.client.0.smithi047.stdout:1.1a 3 0 0 0 0 1690 0 0 15 66 15 active+clean 2024-09-18T14:09:09.863739+0000 34'81 88:143 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:26.909 INFO:tasks.workunit.client.0.smithi047.stdout:1.1b 3 0 0 0 0 1690 0 0 15 66 15 active+clean 2024-09-18T14:09:10.178410+0000 34'81 88:158 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:06:27.302596+0000 0'0 2024-09-18T14:06:27.302596+0000 0 0 queued for scrub 0 0 2024-09-18T14:09:26.909 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:09:26.909 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:09:26.910 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:09:26.917 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: expr 2 + 2 2024-09-18T14:09:26.918 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: seq 1 4 2024-09-18T14:09:26.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-18T14:09:26.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_1 td/divergent-priors/existing 2024-09-18T14:09:27.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-18T14:09:27.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_2 td/divergent-priors/existing 2024-09-18T14:09:27.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-18T14:09:27.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_3 td/divergent-priors/existing 2024-09-18T14:09:27.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-18T14:09:27.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_4 td/divergent-priors/existing 2024-09-18T14:09:27.520 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: seq 1 2 2024-09-18T14:09:27.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-18T14:09:27.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:825: TEST_divergent_3: rados -p test get newobject_1 td/divergent-priors/existing 2024-09-18T14:09:27.663 INFO:tasks.workunit.client.0.smithi047.stderr:error getting test/newobject_1: (2) No such file or directory 2024-09-18T14:09:27.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-18T14:09:27.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:825: TEST_divergent_3: rados -p test get newobject_2 td/divergent-priors/existing 2024-09-18T14:09:27.694 INFO:tasks.workunit.client.0.smithi047.stderr:error getting test/newobject_2: (2) No such file or directory 2024-09-18T14:09:27.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:827: TEST_divergent_3: rm -f td/divergent-priors/existing 2024-09-18T14:09:27.697 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-09-18T14:09:27.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-18T14:09:27.717 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.0.log:2024-09-18T14:08:57.590+0000 7f3f089c5640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 40'81 (31'3) delete 1:8881e6d8:::existing_3:head by client.4658.0:1 2024-09-18T14:07:10.130307+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2024-09-18T14:09:27.718 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.0.log:2024-09-18T14:08:57.590+0000 7f3f089c5640 20 _merge_object_divergent_entries: keeping 40'81 (31'3) delete 1:8881e6d8:::existing_3:head by client.4658.0:1 2024-09-18T14:07:10.130307+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2024-09-18T14:09:27.718 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.0.log:2024-09-18T14:08:57.590+0000 7f3f089c5640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-09-18T14:09:27.718 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.0.log:2024-09-18T14:08:57.590+0000 7f3f089c5640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 31'3 first_divergent_update: 40'81 last_divergent_update: 40'81 2024-09-18T14:09:27.718 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.0.log:2024-09-18T14:08:57.590+0000 7f3f089c5640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-09-18T14:09:27.718 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.0.log:2024-09-18T14:08:57.590+0000 7f3f089c5640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-09-18T14:09:27.718 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.0.log:2024-09-18T14:08:57.590+0000 7f3f089c5640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 40'81 2024-09-18T14:09:27.718 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.0.log:2024-09-18T14:08:57.590+0000 7f3f089c5640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 40'81 (31'3) delete 1:8881e6d8:::existing_3:head by client.4658.0:1 2024-09-18T14:07:10.130307+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2024-09-18T14:09:27.718 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.0.log:2024-09-18T14:08:57.590+0000 7f3f089c5640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-09-18T14:09:27.718 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.0.log:2024-09-18T14:08:57.590+0000 7f3f089c5640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 31'3 <= info.log_tail 31'66 2024-09-18T14:09:27.718 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.595+0000 7f543b7c3640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 40'84 (31'4) delete 1:70b3bf12:::existing_4:head by client.4659.0:1 2024-09-18T14:07:10.131256+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2024-09-18T14:09:27.718 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.595+0000 7f543b7c3640 20 _merge_object_divergent_entries: keeping 40'84 (31'4) delete 1:70b3bf12:::existing_4:head by client.4659.0:1 2024-09-18T14:07:10.131256+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2024-09-18T14:09:27.718 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.595+0000 7f543b7c3640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-18T14:09:27.719 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.595+0000 7f543b7c3640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 31'4 first_divergent_update: 40'84 last_divergent_update: 40'84 2024-09-18T14:09:27.719 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.595+0000 7f543b7c3640 10 _merge_object_divergent_entries: more recent entry found: 70'83 (31'4) modify 1:70b3bf12:::existing_4:head by client.4776.0:1 2024-09-18T14:08:13.097513+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0, already merged 2024-09-18T14:09:27.719 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.595+0000 7f543b7c3640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 40'83 (0'0) modify 1:75ac5091:::newobject_2:head by client.4657.0:1 2024-09-18T14:07:10.129086+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2024-09-18T14:09:27.719 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.595+0000 7f543b7c3640 20 _merge_object_divergent_entries: keeping 40'83 (0'0) modify 1:75ac5091:::newobject_2:head by client.4657.0:1 2024-09-18T14:07:10.129086+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2024-09-18T14:09:27.719 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.595+0000 7f543b7c3640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-18T14:09:27.719 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.595+0000 7f543b7c3640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 40'83 last_divergent_update: 40'83 2024-09-18T14:09:27.719 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.595+0000 7f543b7c3640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2024-09-18T14:09:27.719 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.595+0000 7f543b7c3640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2024-09-18T14:09:27.719 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.596+0000 7f543c7c5640 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 40'83 (31'1) modify 1:d858c035:::existing_1:head by client.4656.0:1 2024-09-18T14:07:10.128699+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-18T14:09:27.719 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.596+0000 7f543c7c5640 20 _merge_object_divergent_entries: keeping 40'83 (31'1) modify 1:d858c035:::existing_1:head by client.4656.0:1 2024-09-18T14:07:10.128699+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-18T14:09:27.719 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.596+0000 7f543c7c5640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-18T14:09:27.719 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.596+0000 7f543c7c5640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 31'1 first_divergent_update: 40'83 last_divergent_update: 40'83 2024-09-18T14:09:27.720 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.596+0000 7f543c7c5640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2024-09-18T14:09:27.720 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.596+0000 7f543c7c5640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2024-09-18T14:09:27.720 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.596+0000 7f543c7c5640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 40'83 2024-09-18T14:09:27.720 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.596+0000 7f543c7c5640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 40'83 (31'1) modify 1:d858c035:::existing_1:head by client.4656.0:1 2024-09-18T14:07:10.128699+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-18T14:09:27.720 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.596+0000 7f543c7c5640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2024-09-18T14:09:27.720 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.596+0000 7f543c7c5640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 31'1 <= info.log_tail 31'66 2024-09-18T14:09:27.720 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.596+0000 7f543c7c5640 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 40'82 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4646.0:1 2024-09-18T14:07:10.038644+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2024-09-18T14:09:27.720 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.596+0000 7f543c7c5640 20 _merge_object_divergent_entries: keeping 40'82 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4646.0:1 2024-09-18T14:07:10.038644+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2024-09-18T14:09:27.720 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.596+0000 7f543c7c5640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-18T14:09:27.720 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.596+0000 7f543c7c5640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 40'82 last_divergent_update: 40'82 2024-09-18T14:09:27.721 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.596+0000 7f543c7c5640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2024-09-18T14:09:27.721 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.596+0000 7f543c7c5640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2024-09-18T14:09:27.721 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.669+0000 7f543b7c3640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 40'83 (31'2) modify 1:97db2a0f:::existing_2:head by client.4647.0:1 2024-09-18T14:07:10.040535+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-18T14:09:27.721 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.669+0000 7f543b7c3640 20 _merge_object_divergent_entries: keeping 40'83 (31'2) modify 1:97db2a0f:::existing_2:head by client.4647.0:1 2024-09-18T14:07:10.040535+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-18T14:09:27.721 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.669+0000 7f543b7c3640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-18T14:09:27.721 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.669+0000 7f543b7c3640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 31'2 first_divergent_update: 40'83 last_divergent_update: 40'83 2024-09-18T14:09:27.721 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.669+0000 7f543b7c3640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2024-09-18T14:09:27.721 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.669+0000 7f543b7c3640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2024-09-18T14:09:27.721 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.669+0000 7f543b7c3640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 40'83 2024-09-18T14:09:27.721 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.669+0000 7f543b7c3640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 40'83 (31'2) modify 1:97db2a0f:::existing_2:head by client.4647.0:1 2024-09-18T14:07:10.040535+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-18T14:09:27.721 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.669+0000 7f543b7c3640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2024-09-18T14:09:27.721 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.669+0000 7f543b7c3640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 31'2 <= info.log_tail 31'66 2024-09-18T14:09:27.721 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.896+0000 7f543c7c5640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 40'81 (31'3) delete 1:8881e6d8:::existing_3:head by client.4658.0:1 2024-09-18T14:07:10.130307+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2024-09-18T14:09:27.722 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.896+0000 7f543c7c5640 20 _merge_object_divergent_entries: keeping 40'81 (31'3) delete 1:8881e6d8:::existing_3:head by client.4658.0:1 2024-09-18T14:07:10.130307+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2024-09-18T14:09:27.722 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.896+0000 7f543c7c5640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-09-18T14:09:27.722 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.896+0000 7f543c7c5640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 31'3 first_divergent_update: 40'81 last_divergent_update: 40'81 2024-09-18T14:09:27.722 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.896+0000 7f543c7c5640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-09-18T14:09:27.731 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.896+0000 7f543c7c5640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-09-18T14:09:27.731 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.896+0000 7f543c7c5640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 40'81 2024-09-18T14:09:27.731 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.896+0000 7f543c7c5640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 40'81 (31'3) delete 1:8881e6d8:::existing_3:head by client.4658.0:1 2024-09-18T14:07:10.130307+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2024-09-18T14:09:27.732 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.896+0000 7f543c7c5640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-09-18T14:09:27.732 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:08:57.896+0000 7f543c7c5640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 31'3 <= info.log_tail 31'66 2024-09-18T14:09:27.732 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-09-18T14:09:27.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-18T14:09:27.750 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:837: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-09-18T14:09:27.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:837: TEST_divergent_3: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-18T14:09:27.771 INFO:tasks.workunit.client.0.smithi047.stdout:success 2024-09-18T14:09:27.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:842: TEST_divergent_3: echo success 2024-09-18T14:09:27.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:844: TEST_divergent_3: rm td/divergent-priors/td/divergent-priors/exp.32609.out 2024-09-18T14:09:27.772 INFO:tasks.workunit.client.0.smithi047.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.32609.out': No such file or directory 2024-09-18T14:09:27.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:846: TEST_divergent_3: delete_pool test 2024-09-18T14:09:27.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-18T14:09:27.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-18T14:09:28.353 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test' does not exist 2024-09-18T14:09:28.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:847: TEST_divergent_3: kill_daemons td/divergent-priors 2024-09-18T14:09:28.364 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:09:28.364 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:09:28.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:09:28.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:09:28.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:09:28.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:09:28.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-18T14:09:28.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-18T14:09:28.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:09:28.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-18T14:09:28.674 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:09:28.674 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:09:28.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:09:28.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:09:28.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:09:28.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:09:28.679 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:09:28.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:09:28.681 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:09:28.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:09:28.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:09:28.682 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:09:28.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:09:28.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:09:28.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:09:28.683 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:09:28.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:09:28.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:09:28.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-18T14:09:28.719 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:09:28.719 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:09:28.720 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:09:28.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.32609 2024-09-18T14:09:28.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:09:28.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:09:28.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-18T14:09:28.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-18T14:09:28.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-09-18T14:09:28.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-09-18T14:09:28.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-18T14:09:28.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:09:28.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-18T14:09:28.723 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:09:28.723 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:09:28.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:09:28.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:09:28.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:09:28.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:09:28.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:09:28.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:09:28.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:09:28.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:09:28.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:09:28.726 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:09:28.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:09:28.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:09:28.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:09:28.727 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:09:28.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:09:28.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:09:28.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-18T14:09:28.730 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:09:28.730 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:09:28.730 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:09:28.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.32609 2024-09-18T14:09:28.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:09:28.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:09:28.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-09-18T14:09:28.733 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T14:09:28.733 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:09:28.733 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:09:28.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.32609 2024-09-18T14:09:28.735 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T14:09:28.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T14:09:28.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T14:09:28.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-18T14:09:28.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_ec td/divergent-priors 2024-09-18T14:09:28.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:228: TEST_divergent_ec: local dir=td/divergent-priors 2024-09-18T14:09:28.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:231: TEST_divergent_ec: local dummyfile=/etc/fstab 2024-09-18T14:09:28.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:232: TEST_divergent_ec: local dummyfile2=/etc/resolv.conf 2024-09-18T14:09:28.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:234: TEST_divergent_ec: local num_osds=3 2024-09-18T14:09:28.735 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: expr 3 - 1 2024-09-18T14:09:28.736 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: seq 0 2 2024-09-18T14:09:28.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: local 'osds=0 2024-09-18T14:09:28.737 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T14:09:28.737 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T14:09:28.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:236: TEST_divergent_ec: run_mon td/divergent-priors a 2024-09-18T14:09:28.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-09-18T14:09:28.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T14:09:28.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T14:09:28.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T14:09:28.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-09-18T14:09:28.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-18T14:09:29.087 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T14:09:29.087 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:09:29.087 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:09:29.088 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:09:29.088 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:09:29.088 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:09:29.088 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:09:29.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T14:09:29.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T14:09:29.332 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T14:09:29.333 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:09:29.333 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:09:29.333 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T14:09:29.333 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T14:09:29.333 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:09:29.333 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:09:29.334 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:09:29.334 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:09:29.334 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:09:29.334 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:09:29.334 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32609/ceph-mon.a.asok 2024-09-18T14:09:29.334 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:09:29.335 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.32609/ceph-mon.a.asok config get fsid 2024-09-18T14:09:29.411 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T14:09:29.411 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:09:29.412 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:09:29.412 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T14:09:29.412 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T14:09:29.412 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:09:29.412 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:09:29.412 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:09:29.412 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:09:29.412 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:09:29.412 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:09:29.412 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32609/ceph-mon.a.asok 2024-09-18T14:09:29.413 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:09:29.413 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.32609/ceph-mon.a.asok config get mon_host 2024-09-18T14:09:29.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:237: TEST_divergent_ec: run_mgr td/divergent-priors x 2024-09-18T14:09:29.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-09-18T14:09:29.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T14:09:29.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T14:09:29.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T14:09:29.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-09-18T14:09:29.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T14:09:30.344 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T14:09:30.344 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:09:30.344 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:09:30.344 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:09:30.344 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:09:30.344 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:09:30.344 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:09:30.345 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:09:30.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:09:30.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-09-18T14:09:30.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 0 2024-09-18T14:09:30.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-18T14:09:30.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:09:30.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T14:09:30.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:09:30.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-09-18T14:09:30.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T14:09:30.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:09:30.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:09:30.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:09:30.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-18T14:09:30.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-18T14:09:30.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:09:30.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:09:30.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-18T14:09:30.378 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:09:30.378 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:09:30.378 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:09:30.378 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:09:30.378 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:09:30.378 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:09:30.378 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:09:30.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:09:30.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:09:30.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:09:30.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:09:30.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-18T14:09:30.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-18T14:09:30.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:09:30.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:09:30.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:09:30.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:09:30.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:09:30.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:09:30.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-09-18T14:09:30.379 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:09:30.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=50835fc1-8964-4c12-b447-46f109dbb8e4 2024-09-18T14:09:30.380 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 50835fc1-8964-4c12-b447-46f109dbb8e4 2024-09-18T14:09:30.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 50835fc1-8964-4c12-b447-46f109dbb8e4' 2024-09-18T14:09:30.380 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:09:30.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAa3+pmOxy5FxAACB5OVp/0gVUQuW0KBagA4g== 2024-09-18T14:09:30.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAa3+pmOxy5FxAACB5OVp/0gVUQuW0KBagA4g=="}' 2024-09-18T14:09:30.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 50835fc1-8964-4c12-b447-46f109dbb8e4 -i td/divergent-priors/0/new.json 2024-09-18T14:09:30.719 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:09:30.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-09-18T14:09:30.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAa3+pmOxy5FxAACB5OVp/0gVUQuW0KBagA4g== --osd-uuid 50835fc1-8964-4c12-b447-46f109dbb8e4 2024-09-18T14:09:30.762 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:09:30.761+0000 7fd27a52c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:09:30.764 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:09:30.764+0000 7fd27a52c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:09:30.765 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:09:30.765+0000 7fd27a52c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:09:30.766 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:09:30.766+0000 7fd27a52c440 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-18T14:09:30.766 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:09:30.766+0000 7fd27a52c440 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-18T14:09:30.766 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:09:30.766+0000 7fd27a52c440 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-18T14:09:30.796 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:09:30.796+0000 7fd27a52c440 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-18T14:09:36.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-18T14:09:36.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:09:36.092 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T14:09:36.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T14:09:36.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:09:36.624 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T14:09:36.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T14:09:36.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:09:36.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:09:36.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:09:36.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:09:36.654 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:09:36.654+0000 7f6d7140b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:09:36.663 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:09:36.663+0000 7f6d7140b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:09:36.664 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:09:36.664+0000 7f6d7140b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:09:37.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T14:09:37.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:09:37.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:09:37.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:09:37.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:09:37.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:09:37.068 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:09:37.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:09:37.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:09:37.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:09:37.199 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:09:37.199+0000 7f6d7140b440 -1 Falling back to public interface 2024-09-18T14:09:37.477 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:09:38.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:09:38.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:09:38.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:09:38.479 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:09:38.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:09:38.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:09:38.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:09:39.693 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:09:39.693+0000 7f6d7140b440 -1 osd.0 0 log_to_monitors true 2024-09-18T14:09:39.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:09:39.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:09:39.897 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:09:39.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:09:39.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:09:39.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:09:40.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:09:41.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:09:41.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:09:41.394 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:09:41.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:09:41.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:09:41.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:09:41.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:09:42.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:09:42.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:09:42.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:09:42.809 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:09:42.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:09:42.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:09:43.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:09:44.197 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:09:44.197 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:09:44.198 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:09:44.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:09:44.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:09:44.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:09:44.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:09:45.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:09:45.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:09:45.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:09:45.645 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:09:45.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:09:45.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:09:46.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:09:47.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:09:47.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:09:47.061 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:09:47.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:09:47.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:09:47.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:09:47.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:09:48.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:09:48.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:09:48.486 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:09:48.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:09:48.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:09:48.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:09:48.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:09:49.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:09:49.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:09:49.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:09:49.891 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:09:49.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:09:49.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:09:50.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:09:51.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:09:51.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:09:51.288 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:09:51.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:09:51.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:09:51.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:09:52.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:09:53.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:09:53.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:09:53.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:09:53.141 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:09:53.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:09:53.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:09:53.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:09:54.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:09:54.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:09:54.824 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:09:54.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:09:54.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:09:54.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:09:55.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:09:56.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:09:56.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:09:56.537 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:09:56.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:09:56.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:09:56.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:09:57.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:09:58.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:09:58.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:09:58.272 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:09:58.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:09:58.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:09:58.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:09:59.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:10:00.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:10:00.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:10:00.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:10:00.387 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:10:00.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:10:00.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:10:01.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:10:02.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:10:02.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:10:02.217 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:10:02.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:10:02.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:10:02.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:10:02.632 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/75135,v1:127.0.0.1:6803/75135] [v2:127.0.0.1:6804/75135,v1:127.0.0.1:6805/75135] exists,up 50835fc1-8964-4c12-b447-46f109dbb8e4 2024-09-18T14:10:02.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:10:02.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:10:02.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:10:02.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-09-18T14:10:02.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 1 2024-09-18T14:10:02.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-18T14:10:02.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:10:02.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T14:10:02.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:10:02.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-09-18T14:10:02.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T14:10:02.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:10:02.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:10:02.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:10:02.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-18T14:10:02.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-18T14:10:02.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:10:02.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:10:02.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-18T14:10:02.637 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:10:02.638 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:10:02.638 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:10:02.638 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:10:02.638 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:10:02.638 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:10:02.638 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:10:02.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:10:02.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:10:02.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:10:02.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:10:02.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-18T14:10:02.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-18T14:10:02.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:10:02.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:10:02.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:10:02.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:10:02.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:10:02.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:10:02.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-09-18T14:10:02.639 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:10:02.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f4d67a04-7819-45b5-bc8c-088b7d24eed7 2024-09-18T14:10:02.639 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 f4d67a04-7819-45b5-bc8c-088b7d24eed7 2024-09-18T14:10:02.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f4d67a04-7819-45b5-bc8c-088b7d24eed7' 2024-09-18T14:10:02.640 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:10:02.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA63+pmDaMhJxAApBtGhw9hSXMBZKxiHtsUBg== 2024-09-18T14:10:02.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA63+pmDaMhJxAApBtGhw9hSXMBZKxiHtsUBg=="}' 2024-09-18T14:10:02.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f4d67a04-7819-45b5-bc8c-088b7d24eed7 -i td/divergent-priors/1/new.json 2024-09-18T14:10:03.159 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:10:03.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-09-18T14:10:03.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA63+pmDaMhJxAApBtGhw9hSXMBZKxiHtsUBg== --osd-uuid f4d67a04-7819-45b5-bc8c-088b7d24eed7 2024-09-18T14:10:03.202 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:10:03.202+0000 7f368262e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:10:03.204 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:10:03.204+0000 7f368262e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:10:03.205 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:10:03.205+0000 7f368262e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:10:03.205 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:10:03.205+0000 7f368262e440 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-18T14:10:03.205 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:10:03.205+0000 7f368262e440 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-18T14:10:03.205 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:10:03.205+0000 7f368262e440 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-18T14:10:03.283 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:10:03.283+0000 7f368262e440 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-18T14:10:08.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-18T14:10:08.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:10:08.880 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T14:10:08.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T14:10:08.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:10:09.395 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T14:10:09.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T14:10:09.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:10:09.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:10:09.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:10:09.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:10:09.426 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:10:09.427+0000 7f96a351e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:10:09.431 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:10:09.432+0000 7f96a351e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:10:09.432 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:10:09.433+0000 7f96a351e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:10:09.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T14:10:09.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:10:09.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:10:09.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:10:09.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:10:09.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:10:09.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:10:09.820 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:10:09.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:10:09.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:10:09.954 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:10:09.955+0000 7f96a351e440 -1 Falling back to public interface 2024-09-18T14:10:10.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:10:11.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:10:11.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:10:11.242 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:10:11.242 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:10:11.242 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:10:11.242 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:10:11.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:10:12.553 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:10:12.553+0000 7f96a351e440 -1 osd.1 0 log_to_monitors true 2024-09-18T14:10:12.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:10:12.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:10:12.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:10:12.628 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:10:12.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:10:12.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:10:13.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:10:14.079 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:10:14.080 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:10:14.080 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:10:14.080 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:10:14.080 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:10:14.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:10:14.549 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:10:15.551 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:10:15.551 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:10:15.551 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:10:15.551 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:10:15.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:10:15.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:10:15.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:10:16.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:10:16.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:10:16.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:10:16.939 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:10:16.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:10:16.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:10:17.390 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:10:18.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:10:18.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:10:18.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:10:18.391 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:10:18.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:10:18.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:10:18.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:10:19.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:10:19.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:10:19.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:10:19.922 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:10:19.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:10:19.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:10:20.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:10:21.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:10:21.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:10:21.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:10:21.304 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:10:21.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:10:21.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:10:21.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:10:22.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:10:22.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:10:22.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:10:22.753 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:10:22.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:10:22.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:10:23.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:10:24.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:10:24.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:10:24.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:10:24.241 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:10:24.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:10:24.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:10:24.988 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:10:25.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:10:25.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:10:25.990 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:10:25.991 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:10:25.991 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:10:25.991 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:10:26.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:10:27.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:10:27.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:10:27.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:10:27.501 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:10:27.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:10:27.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:10:28.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:10:29.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:10:29.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:10:29.274 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:10:29.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:10:29.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:10:29.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:10:30.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:10:31.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:10:31.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:10:31.123 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:10:31.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:10:31.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:10:31.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:10:31.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:10:32.875 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:10:32.875 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:10:32.875 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:10:32.875 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:10:32.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:10:32.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:10:33.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:10:34.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:10:34.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:10:34.923 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:10:34.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:10:34.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:10:34.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:10:35.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:10:36.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:10:36.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:10:36.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:10:36.503 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T14:10:36.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:10:36.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:10:36.912 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/76864,v1:127.0.0.1:6811/76864] [v2:127.0.0.1:6812/76864,v1:127.0.0.1:6813/76864] exists,up f4d67a04-7819-45b5-bc8c-088b7d24eed7 2024-09-18T14:10:36.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:10:36.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:10:36.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:10:36.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-09-18T14:10:36.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 2 2024-09-18T14:10:36.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-18T14:10:36.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:10:36.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T14:10:36.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:10:36.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-09-18T14:10:36.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T14:10:36.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:10:36.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:10:36.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:10:36.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-18T14:10:36.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-18T14:10:36.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:10:36.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:10:36.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-18T14:10:36.917 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:10:36.917 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:10:36.917 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:10:36.917 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:10:36.917 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:10:36.917 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:10:36.918 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:10:36.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:10:36.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:10:36.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:10:36.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:10:36.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-18T14:10:36.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-18T14:10:36.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:10:36.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:10:36.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:10:36.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:10:36.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:10:36.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:10:36.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-09-18T14:10:36.919 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:10:36.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2d53c532-9e0f-4778-a2d9-277af85698d4 2024-09-18T14:10:36.919 INFO:tasks.workunit.client.0.smithi047.stdout:add osd2 2d53c532-9e0f-4778-a2d9-277af85698d4 2024-09-18T14:10:36.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 2d53c532-9e0f-4778-a2d9-277af85698d4' 2024-09-18T14:10:36.919 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:10:36.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBc3+pmaOjINxAAEDIY5H5sj4YhBNlGAKPX8Q== 2024-09-18T14:10:36.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBc3+pmaOjINxAAEDIY5H5sj4YhBNlGAKPX8Q=="}' 2024-09-18T14:10:36.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2d53c532-9e0f-4778-a2d9-277af85698d4 -i td/divergent-priors/2/new.json 2024-09-18T14:10:37.437 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:10:37.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-09-18T14:10:37.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBc3+pmaOjINxAAEDIY5H5sj4YhBNlGAKPX8Q== --osd-uuid 2d53c532-9e0f-4778-a2d9-277af85698d4 2024-09-18T14:10:37.481 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:10:37.480+0000 7fde8ba15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:10:37.482 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:10:37.482+0000 7fde8ba15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:10:37.483 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:10:37.483+0000 7fde8ba15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:10:37.484 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:10:37.484+0000 7fde8ba15440 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-18T14:10:37.484 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:10:37.484+0000 7fde8ba15440 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-18T14:10:37.484 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:10:37.484+0000 7fde8ba15440 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-18T14:10:37.552 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:10:37.552+0000 7fde8ba15440 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-18T14:10:43.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-18T14:10:43.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:10:43.218 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd2 key to auth repository 2024-09-18T14:10:43.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T14:10:43.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:10:43.734 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T14:10:43.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T14:10:43.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:10:43.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:10:43.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:10:43.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:10:43.765 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:10:43.765+0000 7f0d68833440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:10:43.784 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:10:43.784+0000 7f0d68833440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:10:43.785 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:10:43.785+0000 7f0d68833440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:10:44.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T14:10:44.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:10:44.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T14:10:44.135 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:10:44.135 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:10:44.135 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:10:44.135 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:10:44.135 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:10:44.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:10:44.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:10:44.316 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:10:44.316+0000 7f0d68833440 -1 Falling back to public interface 2024-09-18T14:10:44.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:10:45.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:10:45.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:10:45.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:10:45.525 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:10:45.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:10:45.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:10:45.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:10:46.890 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:10:46.890+0000 7f0d68833440 -1 osd.2 0 log_to_monitors true 2024-09-18T14:10:46.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:10:46.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:10:46.952 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:10:46.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:10:46.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:10:46.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:10:47.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:10:48.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:10:48.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:10:48.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:10:48.509 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:10:48.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:10:48.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:10:48.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:10:49.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:10:49.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:10:49.939 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:10:49.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:10:49.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:10:49.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:10:50.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:10:51.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:10:51.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:10:51.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:10:51.325 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:10:51.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:10:51.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:10:51.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:10:52.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:10:52.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:10:52.810 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:10:52.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:10:52.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:10:52.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:10:53.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:10:54.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:10:54.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:10:54.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:10:54.227 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:10:54.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:10:54.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:10:54.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:10:55.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:10:55.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:10:55.617 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:10:55.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:10:55.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:10:55.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:10:56.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:10:57.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:10:57.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:10:57.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:10:57.031 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:10:57.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:10:57.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:10:57.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:10:58.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:10:58.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:10:58.564 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:10:58.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:10:58.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:10:58.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:10:59.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:11:00.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:11:00.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:11:00.354 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:11:00.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:11:00.355 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:11:00.355 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:11:01.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:11:02.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:11:02.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:11:02.051 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:11:02.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:11:02.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:11:02.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:11:02.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:11:03.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:11:03.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:11:03.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:11:03.747 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:11:03.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:11:03.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:11:04.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:11:05.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:11:05.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:11:05.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:11:05.312 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:11:05.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:11:05.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:11:06.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:11:07.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:11:07.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:11:07.093 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:11:07.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:11:07.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:11:07.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:11:07.907 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:11:08.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:11:08.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:11:08.909 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:11:08.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:11:08.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:11:08.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:11:09.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:11:10.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:11:10.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:11:10.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:11:10.297 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T14:11:10.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:11:10.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:11:10.680 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/78621,v1:127.0.0.1:6819/78621] [v2:127.0.0.1:6820/78621,v1:127.0.0.1:6821/78621] exists,up 2d53c532-9e0f-4778-a2d9-277af85698d4 2024-09-18T14:11:10.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:11:10.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:11:10.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:11:10.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:243: TEST_divergent_ec: ceph osd set noout 2024-09-18T14:11:11.355 INFO:tasks.workunit.client.0.smithi047.stderr:noout is set 2024-09-18T14:11:11.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:244: TEST_divergent_ec: ceph osd set noin 2024-09-18T14:11:12.243 INFO:tasks.workunit.client.0.smithi047.stderr:noin is set 2024-09-18T14:11:12.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:245: TEST_divergent_ec: ceph osd set nodown 2024-09-18T14:11:13.030 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T14:11:13.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:246: TEST_divergent_ec: create_ec_pool test true k=2 m=1 2024-09-18T14:11:13.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: create_ec_pool: local pool_name=test 2024-09-18T14:11:13.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: create_ec_pool: shift 2024-09-18T14:11:13.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: create_ec_pool: local allow_overwrites=true 2024-09-18T14:11:13.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: create_ec_pool: shift 2024-09-18T14:11:13.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-09-18T14:11:13.761 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: create_ec_pool: create_pool test 1 1 erasure myprofile 2024-09-18T14:11:13.761 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 erasure myprofile 2024-09-18T14:11:14.565 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test' already exists 2024-09-18T14:11:14.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T14:11:15.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: create_ec_pool: '[' true = true ']' 2024-09-18T14:11:15.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: create_ec_pool: ceph osd pool set test allow_ec_overwrites true 2024-09-18T14:11:16.419 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 1 allow_ec_overwrites to true 2024-09-18T14:11:16.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: create_ec_pool: wait_for_clean 2024-09-18T14:11:16.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:11:16.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:11:16.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:11:16.432 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:11:16.432 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:11:16.432 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:11:16.432 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:11:16.432 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:11:16.432 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:11:16.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T14:11:16.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:11:16.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:11:16.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:11:16.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:11:16.525 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:11:16.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:11:16.958 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T14:11:16.959 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T14:11:16.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:11:16.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:11:16.959 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:11:17.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836504 2024-09-18T14:11:17.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836504 2024-09-18T14:11:17.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836504' 2024-09-18T14:11:17.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:11:17.115 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:11:17.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672972 2024-09-18T14:11:17.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672972 2024-09-18T14:11:17.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836504 1-42949672972' 2024-09-18T14:11:17.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:11:17.232 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:11:17.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509448 2024-09-18T14:11:17.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509448 2024-09-18T14:11:17.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836504 1-42949672972 2-64424509448' 2024-09-18T14:11:17.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:11:17.348 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836504 2024-09-18T14:11:17.348 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:11:17.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:11:17.349 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836504 2024-09-18T14:11:17.349 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:11:17.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836504 2024-09-18T14:11:17.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-09-18T14:11:17.350 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836504 2024-09-18T14:11:17.350 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:11:17.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836501 -lt 21474836504 2024-09-18T14:11:17.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:11:18.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:11:18.726 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:11:19.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836504 -lt 21474836504 2024-09-18T14:11:19.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:11:19.111 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672972 2024-09-18T14:11:19.111 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:11:19.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:11:19.112 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672972 2024-09-18T14:11:19.112 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:11:19.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672972 2024-09-18T14:11:19.113 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672972 2024-09-18T14:11:19.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-09-18T14:11:19.114 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:11:19.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672972 -lt 42949672972 2024-09-18T14:11:19.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:11:19.513 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509448 2024-09-18T14:11:19.513 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:11:19.514 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:11:19.514 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509448 2024-09-18T14:11:19.514 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:11:19.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509448 2024-09-18T14:11:19.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-09-18T14:11:19.516 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509448 2024-09-18T14:11:19.516 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:11:19.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509448 -lt 64424509448 2024-09-18T14:11:19.972 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:11:19.973 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:11:19.973 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:11:20.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T14:11:20.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:11:20.475 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:11:20.475 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:11:20.475 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:11:20.475 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:11:20.475 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:11:20.475 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:11:20.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T14:11:20.863 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:11:20.863 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:11:20.864 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:11:21.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T14:11:21.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:11:21.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:11:21.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: create_ec_pool: return 0 2024-09-18T14:11:21.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:248: TEST_divergent_ec: flush_pg_stats 2024-09-18T14:11:21.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:11:21.365 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:11:21.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:11:21.793 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T14:11:21.793 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T14:11:21.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:11:21.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:11:21.793 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:11:21.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836506 2024-09-18T14:11:21.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836506 2024-09-18T14:11:21.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836506' 2024-09-18T14:11:21.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:11:21.910 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:11:22.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672975 2024-09-18T14:11:22.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672975 2024-09-18T14:11:22.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836506 1-42949672975' 2024-09-18T14:11:22.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:11:22.027 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:11:22.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509451 2024-09-18T14:11:22.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509451 2024-09-18T14:11:22.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836506 1-42949672975 2-64424509451' 2024-09-18T14:11:22.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:11:22.143 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836506 2024-09-18T14:11:22.143 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:11:22.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:11:22.144 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836506 2024-09-18T14:11:22.145 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:11:22.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836506 2024-09-18T14:11:22.146 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836506 2024-09-18T14:11:22.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-09-18T14:11:22.146 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:11:22.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836506 -lt 21474836506 2024-09-18T14:11:22.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:11:22.537 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672975 2024-09-18T14:11:22.537 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:11:22.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:11:22.538 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672975 2024-09-18T14:11:22.538 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:11:22.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672975 2024-09-18T14:11:22.540 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672975 2024-09-18T14:11:22.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-09-18T14:11:22.540 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:11:22.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672975 -lt 42949672975 2024-09-18T14:11:22.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:11:22.980 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509451 2024-09-18T14:11:22.980 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:11:22.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:11:22.981 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509451 2024-09-18T14:11:22.981 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:11:22.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509451 2024-09-18T14:11:22.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-09-18T14:11:22.983 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509451 2024-09-18T14:11:22.983 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:11:23.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509449 -lt 64424509451 2024-09-18T14:11:23.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:11:24.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:11:24.398 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:11:24.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509451 -lt 64424509451 2024-09-18T14:11:24.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:249: TEST_divergent_ec: wait_for_clean 2024-09-18T14:11:24.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:11:24.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:11:24.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:11:24.829 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:11:24.830 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:11:24.830 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:11:24.830 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:11:24.830 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:11:24.830 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:11:24.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T14:11:24.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:11:24.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:11:24.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:11:24.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:11:24.923 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:11:25.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:11:25.351 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T14:11:25.351 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T14:11:25.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:11:25.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:11:25.351 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:11:25.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836509 2024-09-18T14:11:25.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836509 2024-09-18T14:11:25.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509' 2024-09-18T14:11:25.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:11:25.467 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:11:25.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672977 2024-09-18T14:11:25.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672977 2024-09-18T14:11:25.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509 1-42949672977' 2024-09-18T14:11:25.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:11:25.583 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:11:25.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509454 2024-09-18T14:11:25.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509454 2024-09-18T14:11:25.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509 1-42949672977 2-64424509454' 2024-09-18T14:11:25.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:11:25.698 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836509 2024-09-18T14:11:25.699 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:11:25.699 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:11:25.700 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836509 2024-09-18T14:11:25.700 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:11:25.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836509 2024-09-18T14:11:25.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-09-18T14:11:25.701 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836509 2024-09-18T14:11:25.702 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:11:26.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836507 -lt 21474836509 2024-09-18T14:11:26.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:11:27.119 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:11:27.120 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:11:27.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836509 -lt 21474836509 2024-09-18T14:11:27.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:11:27.529 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672977 2024-09-18T14:11:27.529 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:11:27.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:11:27.530 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672977 2024-09-18T14:11:27.530 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:11:27.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672977 2024-09-18T14:11:27.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-09-18T14:11:27.532 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672977 2024-09-18T14:11:27.532 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:11:27.938 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672977 -lt 42949672977 2024-09-18T14:11:27.938 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:11:27.938 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509454 2024-09-18T14:11:27.938 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:11:27.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:11:27.940 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509454 2024-09-18T14:11:27.940 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:11:27.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509454 2024-09-18T14:11:27.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-09-18T14:11:27.941 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509454 2024-09-18T14:11:27.942 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:11:28.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509454 -lt 64424509454 2024-09-18T14:11:28.321 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:11:28.321 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:11:28.321 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:11:28.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T14:11:28.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:11:28.825 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:11:28.825 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:11:28.825 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:11:28.825 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:11:28.825 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:11:28.825 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:11:29.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T14:11:29.230 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:11:29.231 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:11:29.231 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:11:29.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T14:11:29.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:11:29.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:11:29.734 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: ceph pg dump pgs --format=json 2024-09-18T14:11:29.734 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: jq '.pg_stats[0].up_primary' 2024-09-18T14:11:30.120 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:11:30.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: local divergent=1 2024-09-18T14:11:30.131 INFO:tasks.workunit.client.0.smithi047.stdout:primary and soon to be divergent is 1 2024-09-18T14:11:30.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:253: TEST_divergent_ec: echo 'primary and soon to be divergent is 1' 2024-09-18T14:11:30.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:254: TEST_divergent_ec: ceph pg dump pgs 2024-09-18T14:11:30.490 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:11:30.490 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:11:16.058884+0000 0'0 26:10 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:11:14.315267+0000 0'0 2024-09-18T14:11:14.315267+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:18:49.152218+0000 0 0 2024-09-18T14:11:30.490 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:11:30.490 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:11:30.490 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:11:30.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:255: TEST_divergent_ec: local non_divergent= 2024-09-18T14:11:30.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-09-18T14:11:30.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 0 = 1 ']' 2024-09-18T14:11:30.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0' 2024-09-18T14:11:30.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-09-18T14:11:30.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 1 = 1 ']' 2024-09-18T14:11:30.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:259: TEST_divergent_ec: continue 2024-09-18T14:11:30.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-09-18T14:11:30.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 2 = 1 ']' 2024-09-18T14:11:30.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0 2' 2024-09-18T14:11:30.502 INFO:tasks.workunit.client.0.smithi047.stdout:writing initial objects 2024-09-18T14:11:30.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:264: TEST_divergent_ec: echo 'writing initial objects' 2024-09-18T14:11:30.502 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: seq 1 100 2024-09-18T14:11:30.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:30.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_1 /etc/fstab 2024-09-18T14:11:30.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:30.660 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_2 /etc/fstab 2024-09-18T14:11:30.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:30.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_3 /etc/fstab 2024-09-18T14:11:31.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:31.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_4 /etc/fstab 2024-09-18T14:11:31.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:31.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_5 /etc/fstab 2024-09-18T14:11:31.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:31.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_6 /etc/fstab 2024-09-18T14:11:31.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:31.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_7 /etc/fstab 2024-09-18T14:11:31.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:31.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_8 /etc/fstab 2024-09-18T14:11:32.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:32.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_9 /etc/fstab 2024-09-18T14:11:32.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:32.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_10 /etc/fstab 2024-09-18T14:11:32.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:32.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_11 /etc/fstab 2024-09-18T14:11:32.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:32.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_12 /etc/fstab 2024-09-18T14:11:33.052 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:33.052 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_13 /etc/fstab 2024-09-18T14:11:33.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:33.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_14 /etc/fstab 2024-09-18T14:11:33.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:33.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_15 /etc/fstab 2024-09-18T14:11:33.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:33.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_16 /etc/fstab 2024-09-18T14:11:33.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:33.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_17 /etc/fstab 2024-09-18T14:11:34.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:34.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_18 /etc/fstab 2024-09-18T14:11:34.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:34.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_19 /etc/fstab 2024-09-18T14:11:34.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:34.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_20 /etc/fstab 2024-09-18T14:11:34.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:34.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_21 /etc/fstab 2024-09-18T14:11:34.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:34.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_22 /etc/fstab 2024-09-18T14:11:35.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:35.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_23 /etc/fstab 2024-09-18T14:11:35.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:35.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_24 /etc/fstab 2024-09-18T14:11:35.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:35.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_25 /etc/fstab 2024-09-18T14:11:35.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:35.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_26 /etc/fstab 2024-09-18T14:11:36.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:36.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_27 /etc/fstab 2024-09-18T14:11:36.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:36.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_28 /etc/fstab 2024-09-18T14:11:36.666 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:36.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_29 /etc/fstab 2024-09-18T14:11:36.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:36.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_30 /etc/fstab 2024-09-18T14:11:37.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:37.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_31 /etc/fstab 2024-09-18T14:11:37.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:37.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_32 /etc/fstab 2024-09-18T14:11:37.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:37.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_33 /etc/fstab 2024-09-18T14:11:37.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:37.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_34 /etc/fstab 2024-09-18T14:11:37.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:37.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_35 /etc/fstab 2024-09-18T14:11:38.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:38.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_36 /etc/fstab 2024-09-18T14:11:38.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:38.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_37 /etc/fstab 2024-09-18T14:11:38.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:38.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_38 /etc/fstab 2024-09-18T14:11:38.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:38.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_39 /etc/fstab 2024-09-18T14:11:39.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:39.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_40 /etc/fstab 2024-09-18T14:11:39.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:39.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_41 /etc/fstab 2024-09-18T14:11:39.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:39.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_42 /etc/fstab 2024-09-18T14:11:39.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:39.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_43 /etc/fstab 2024-09-18T14:11:39.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:39.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_44 /etc/fstab 2024-09-18T14:11:40.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:40.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_45 /etc/fstab 2024-09-18T14:11:40.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:40.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_46 /etc/fstab 2024-09-18T14:11:40.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:40.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_47 /etc/fstab 2024-09-18T14:11:40.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:40.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_48 /etc/fstab 2024-09-18T14:11:40.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:40.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_49 /etc/fstab 2024-09-18T14:11:41.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:41.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_50 /etc/fstab 2024-09-18T14:11:41.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:41.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_51 /etc/fstab 2024-09-18T14:11:41.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:41.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_52 /etc/fstab 2024-09-18T14:11:41.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:41.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_53 /etc/fstab 2024-09-18T14:11:42.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:42.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_54 /etc/fstab 2024-09-18T14:11:42.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:42.264 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_55 /etc/fstab 2024-09-18T14:11:42.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:42.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_56 /etc/fstab 2024-09-18T14:11:42.666 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:42.666 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_57 /etc/fstab 2024-09-18T14:11:42.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:42.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_58 /etc/fstab 2024-09-18T14:11:43.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:43.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_59 /etc/fstab 2024-09-18T14:11:43.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:43.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_60 /etc/fstab 2024-09-18T14:11:43.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:43.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_61 /etc/fstab 2024-09-18T14:11:43.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:43.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_62 /etc/fstab 2024-09-18T14:11:43.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:43.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_63 /etc/fstab 2024-09-18T14:11:44.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:44.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_64 /etc/fstab 2024-09-18T14:11:44.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:44.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_65 /etc/fstab 2024-09-18T14:11:44.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:44.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_66 /etc/fstab 2024-09-18T14:11:44.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:44.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_67 /etc/fstab 2024-09-18T14:11:45.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:45.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_68 /etc/fstab 2024-09-18T14:11:45.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:45.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_69 /etc/fstab 2024-09-18T14:11:45.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:45.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_70 /etc/fstab 2024-09-18T14:11:45.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:45.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_71 /etc/fstab 2024-09-18T14:11:45.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:45.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_72 /etc/fstab 2024-09-18T14:11:46.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:46.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_73 /etc/fstab 2024-09-18T14:11:46.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:46.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_74 /etc/fstab 2024-09-18T14:11:46.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:46.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_75 /etc/fstab 2024-09-18T14:11:46.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:46.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_76 /etc/fstab 2024-09-18T14:11:47.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:47.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_77 /etc/fstab 2024-09-18T14:11:47.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:47.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_78 /etc/fstab 2024-09-18T14:11:47.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:47.432 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_79 /etc/fstab 2024-09-18T14:11:47.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:47.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_80 /etc/fstab 2024-09-18T14:11:47.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:47.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_81 /etc/fstab 2024-09-18T14:11:48.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:48.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_82 /etc/fstab 2024-09-18T14:11:48.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:48.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_83 /etc/fstab 2024-09-18T14:11:48.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:48.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_84 /etc/fstab 2024-09-18T14:11:48.740 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:48.741 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_85 /etc/fstab 2024-09-18T14:11:48.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:48.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_86 /etc/fstab 2024-09-18T14:11:49.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:49.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_87 /etc/fstab 2024-09-18T14:11:49.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:49.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_88 /etc/fstab 2024-09-18T14:11:49.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:49.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_89 /etc/fstab 2024-09-18T14:11:49.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:49.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_90 /etc/fstab 2024-09-18T14:11:49.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:49.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_91 /etc/fstab 2024-09-18T14:11:50.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:50.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_92 /etc/fstab 2024-09-18T14:11:50.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:50.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_93 /etc/fstab 2024-09-18T14:11:50.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:50.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_94 /etc/fstab 2024-09-18T14:11:50.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:50.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_95 /etc/fstab 2024-09-18T14:11:51.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:51.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_96 /etc/fstab 2024-09-18T14:11:51.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:51.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_97 /etc/fstab 2024-09-18T14:11:51.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:51.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_98 /etc/fstab 2024-09-18T14:11:51.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:51.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2024-09-18T14:11:51.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-18T14:11:51.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2024-09-18T14:11:52.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-18T14:11:52.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: wait_for_clean 2024-09-18T14:11:52.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:11:52.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:11:52.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:11:52.100 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-18T14:11:52.100 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:11:52.100 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:11:52.100 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:11:52.100 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:11:52.100 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:11:52.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-18T14:11:52.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:11:52.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:11:52.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:11:52.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:11:52.165 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:11:52.551 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:11:52.551 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T14:11:52.551 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T14:11:52.551 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:11:52.551 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:11:52.551 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:11:52.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836517 2024-09-18T14:11:52.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836517 2024-09-18T14:11:52.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836517' 2024-09-18T14:11:52.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:11:52.667 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:11:52.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672985 2024-09-18T14:11:52.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672985 2024-09-18T14:11:52.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836517 1-42949672985' 2024-09-18T14:11:52.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:11:52.783 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:11:52.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509461 2024-09-18T14:11:52.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509461 2024-09-18T14:11:52.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836517 1-42949672985 2-64424509461' 2024-09-18T14:11:52.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:11:52.900 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836517 2024-09-18T14:11:52.901 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:11:52.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:11:52.902 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836517 2024-09-18T14:11:52.902 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:11:52.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836517 2024-09-18T14:11:52.903 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836517 2024-09-18T14:11:52.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836517' 2024-09-18T14:11:52.903 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:11:53.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836514 -lt 21474836517 2024-09-18T14:11:53.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:11:54.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:11:54.321 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:11:54.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836517 -lt 21474836517 2024-09-18T14:11:54.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:11:54.708 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672985 2024-09-18T14:11:54.708 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:11:54.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:11:54.709 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672985 2024-09-18T14:11:54.710 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:11:54.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672985 2024-09-18T14:11:54.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-09-18T14:11:54.711 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672985 2024-09-18T14:11:54.711 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:11:55.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672985 -lt 42949672985 2024-09-18T14:11:55.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:11:55.131 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509461 2024-09-18T14:11:55.131 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:11:55.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:11:55.132 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509461 2024-09-18T14:11:55.133 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:11:55.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509461 2024-09-18T14:11:55.134 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509461 2024-09-18T14:11:55.135 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-09-18T14:11:55.135 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:11:55.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509461 -lt 64424509461 2024-09-18T14:11:55.549 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:11:55.549 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:11:55.549 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:11:56.096 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T14:11:56.096 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:11:56.096 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:11:56.096 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:11:56.096 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:11:56.096 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:11:56.097 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:11:56.097 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:11:56.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T14:11:56.457 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:11:56.458 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:11:56.458 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:11:56.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T14:11:56.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:11:56.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:11:56.957 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: get_pg test existing_1 2024-09-18T14:11:56.957 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-18T14:11:56.957 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-09-18T14:11:56.958 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-09-18T14:11:56.958 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T14:11:57.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: local pgid=1.0 2024-09-18T14:11:57.386 INFO:tasks.workunit.client.0.smithi047.stdout:blackholing osds 0 2 2024-09-18T14:11:57.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:276: TEST_divergent_ec: echo 'blackholing osds 0 2' 2024-09-18T14:11:57.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:277: TEST_divergent_ec: ceph pg dump pgs 2024-09-18T14:11:57.743 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:11:57.743 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 100 0 0 0 0 48300 0 0 16 84 16 active+clean 2024-09-18T14:11:16.058884+0000 28'100 28:110 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:11:14.315267+0000 0'0 2024-09-18T14:11:14.315267+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:18:49.152218+0000 0 0 2024-09-18T14:11:57.743 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:11:57.743 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:11:57.743 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:11:57.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2024-09-18T14:11:57.755 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.0 2024-09-18T14:11:57.755 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:11:57.755 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:11:57.755 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:11:57.755 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:11:57.755 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:11:57.755 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32609/ceph-osd.0.asok 2024-09-18T14:11:57.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2024-09-18T14:11:57.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.32609/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-18T14:11:57.829 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T14:11:57.829 INFO:tasks.workunit.client.0.smithi047.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-18T14:11:57.830 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T14:11:57.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2024-09-18T14:11:57.837 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.2 2024-09-18T14:11:57.837 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:11:57.837 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:11:57.837 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:11:57.837 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:11:57.837 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:11:57.837 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32609/ceph-osd.2.asok 2024-09-18T14:11:57.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2024-09-18T14:11:57.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.32609/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-18T14:11:57.912 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T14:11:57.912 INFO:tasks.workunit.client.0.smithi047.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-18T14:11:57.912 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T14:11:57.919 INFO:tasks.workunit.client.0.smithi047.stdout:writing divergent object 2024-09-18T14:11:57.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:284: TEST_divergent_ec: echo 'writing divergent object' 2024-09-18T14:11:57.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:286: TEST_divergent_ec: sleep 1 2024-09-18T14:11:57.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:285: TEST_divergent_ec: rados -p test put existing_100 /etc/resolv.conf 2024-09-18T14:11:58.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:288: TEST_divergent_ec: rados -p test mksnap snap1 2024-09-18T14:11:58.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:287: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2024-09-18T14:11:59.112 INFO:tasks.workunit.client.0.smithi047.stdout:created pool test snap snap1 2024-09-18T14:11:59.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:290: TEST_divergent_ec: sleep 10 2024-09-18T14:11:59.115 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: expr 100 - 1 2024-09-18T14:11:59.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2024-09-18T14:12:09.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:291: TEST_divergent_ec: killall -9 rados 2024-09-18T14:12:09.121 INFO:tasks.workunit.client.0.smithi047.stderr:rados: no process found 2024-09-18T14:12:09.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:294: TEST_divergent_ec: echo 'killing all the osds' 2024-09-18T14:12:09.121 INFO:tasks.workunit.client.0.smithi047.stdout:killing all the osds 2024-09-18T14:12:09.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:295: TEST_divergent_ec: ceph pg dump pgs 2024-09-18T14:12:09.487 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:12:09.487 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 101 0 0 0 0 48783 0 0 14 90 14 active+clean 2024-09-18T14:11:16.058884+0000 29'104 29:114 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:11:14.315267+0000 0'0 2024-09-18T14:11:14.315267+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:18:49.152218+0000 0 0 2024-09-18T14:12:09.487 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:12:09.487 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:12:09.488 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:12:09.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:296: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd 2024-09-18T14:12:09.498 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:12:09.498 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:12:09.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:12:09.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:12:09.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:12:09.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:12:09.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-09-18T14:12:09.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.0 2024-09-18T14:12:10.116 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already down. 2024-09-18T14:12:10.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-09-18T14:12:10.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.1 2024-09-18T14:12:10.738 INFO:tasks.workunit.client.0.smithi047.stderr:osd.1 is already down. 2024-09-18T14:12:10.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-09-18T14:12:10.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.2 2024-09-18T14:12:11.231 INFO:tasks.workunit.client.0.smithi047.stderr:osd.2 is already down. 2024-09-18T14:12:11.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2024-09-18T14:12:11.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.0 2024-09-18T14:12:11.774 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already out. 2024-09-18T14:12:11.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2024-09-18T14:12:11.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.2 2024-09-18T14:12:12.391 INFO:tasks.workunit.client.0.smithi047.stderr:osd.2 is already out. 2024-09-18T14:12:12.402 INFO:tasks.workunit.client.0.smithi047.stdout:bringing up non_divergent 0 2 2024-09-18T14:12:12.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:307: TEST_divergent_ec: echo 'bringing up non_divergent 0 2' 2024-09-18T14:12:12.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:308: TEST_divergent_ec: ceph pg dump pgs 2024-09-18T14:12:12.760 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:12:12.760 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 101 0 0 0 0 48783 0 0 14 90 14 stale+active+clean 2024-09-18T14:11:16.058884+0000 29'104 29:114 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:11:14.315267+0000 0'0 2024-09-18T14:11:14.315267+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:18:49.152218+0000 0 0 2024-09-18T14:12:12.760 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:12:12.760 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:12:12.761 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:12:12.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2024-09-18T14:12:12.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 0 2024-09-18T14:12:12.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-18T14:12:12.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:12:12.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T14:12:12.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:12:12.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-09-18T14:12:12.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T14:12:12.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:12:12.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:12:12.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:12:12.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-18T14:12:12.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-18T14:12:12.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:12:12.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:12:12.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-18T14:12:12.775 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:12:12.775 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:12:12.775 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:12:12.775 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:12:12.775 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:12:12.775 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:12:12.775 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:12:12.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:12:12.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:12:12.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-18T14:12:12.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-18T14:12:12.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:12:12.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:12:12.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:12:12.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:12:12.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:12:12.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:12:12.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-09-18T14:12:12.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T14:12:12.776 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T14:12:12.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:12:12.777 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-09-18T14:12:12.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T14:12:12.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:12:12.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:12:12.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:12:12.804 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:12:12.804+0000 7f5769f2a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:12:12.806 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:12:12.806+0000 7f5769f2a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:12:12.807 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:12:12.807+0000 7f5769f2a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:12:13.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T14:12:13.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:12:13.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:12:13.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:12:13.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:12:13.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:12:13.188 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:12:13.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:12:13.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:12:13.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:12:13.325 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:12:13.325+0000 7f5769f2a440 -1 Falling back to public interface 2024-09-18T14:12:13.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:12:14.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:12:14.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:12:14.592 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:12:14.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:12:14.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:12:14.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:12:15.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:12:16.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:12:16.025 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:12:16.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:12:16.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:12:16.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:12:16.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:12:16.166 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:12:16.166+0000 7f5769f2a440 -1 osd.0 28 log_to_monitors true 2024-09-18T14:12:16.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:12:17.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:12:17.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:12:17.455 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:12:17.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:12:17.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:12:17.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:12:17.873 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,28) [v2:127.0.0.1:6802/83233,v1:127.0.0.1:6803/83233] [v2:127.0.0.1:6804/83233,v1:127.0.0.1:6805/83233] exists,up 50835fc1-8964-4c12-b447-46f109dbb8e4 2024-09-18T14:12:17.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:12:17.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:12:17.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:12:17.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2024-09-18T14:12:17.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 2 2024-09-18T14:12:17.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-18T14:12:17.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:12:17.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T14:12:17.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:12:17.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-09-18T14:12:17.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T14:12:17.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:12:17.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:12:17.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:12:17.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-18T14:12:17.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-18T14:12:17.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:12:17.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:12:17.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-18T14:12:17.878 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:12:17.878 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:12:17.878 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:12:17.878 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:12:17.878 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:12:17.878 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:12:17.878 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:12:17.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:12:17.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:12:17.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-18T14:12:17.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-18T14:12:17.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:12:17.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:12:17.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:12:17.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:12:17.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:12:17.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:12:17.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-09-18T14:12:17.879 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T14:12:17.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T14:12:17.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:12:17.880 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-09-18T14:12:17.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T14:12:17.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:12:17.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:12:17.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:12:17.907 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:12:17.906+0000 7f5c5c40c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:12:17.909 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:12:17.908+0000 7f5c5c40c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:12:17.910 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:12:17.909+0000 7f5c5c40c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:12:18.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T14:12:18.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:12:18.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T14:12:18.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:12:18.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:12:18.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:12:18.318 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:12:18.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:12:18.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:12:18.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:12:18.427 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:12:18.427+0000 7f5c5c40c440 -1 Falling back to public interface 2024-09-18T14:12:18.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:12:19.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:12:19.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:12:19.725 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:12:19.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:12:19.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:12:19.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:12:20.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:12:21.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:12:21.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:12:21.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:12:21.179 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:12:21.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:12:21.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:12:21.372 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:12:21.371+0000 7f5c5c40c440 -1 osd.2 28 log_to_monitors true 2024-09-18T14:12:21.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:12:22.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:12:22.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:12:22.570 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:12:22.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:12:22.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:12:22.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:12:23.004 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,28) [v2:127.0.0.1:6810/83811,v1:127.0.0.1:6811/83811] [v2:127.0.0.1:6812/83811,v1:127.0.0.1:6813/83811] exists,up 2d53c532-9e0f-4778-a2d9-277af85698d4 2024-09-18T14:12:23.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:12:23.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:12:23.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:12:23.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2024-09-18T14:12:23.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.0 2024-09-18T14:12:23.545 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already in. 2024-09-18T14:12:23.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2024-09-18T14:12:23.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.2 2024-09-18T14:12:24.113 INFO:tasks.workunit.client.0.smithi047.stderr:osd.2 is already in. 2024-09-18T14:12:24.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:318: TEST_divergent_ec: sleep 5 2024-09-18T14:12:29.126 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: expr 2 + 2 2024-09-18T14:12:29.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: objname=existing_4 2024-09-18T14:12:29.127 INFO:tasks.workunit.client.0.smithi047.stdout:writing non-divergent object existing_4 2024-09-18T14:12:29.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:323: TEST_divergent_ec: echo 'writing non-divergent object existing_4' 2024-09-18T14:12:29.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:324: TEST_divergent_ec: ceph pg dump pgs 2024-09-18T14:12:29.492 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:12:29.492 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 100 0 100 0 0 48300 0 0 16 84 16 active+undersized+degraded 2024-09-18T14:12:24.532097+0000 28'100 41:121 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-18T14:11:14.315267+0000 0'0 2024-09-18T14:11:14.315267+0000 0 0 periodic scrub scheduled @ 2024-09-19T23:56:27.109661+0000 0 0 2024-09-18T14:12:29.493 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:12:29.493 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:12:29.493 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:12:29.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:325: TEST_divergent_ec: rados -p test put existing_4 /etc/resolv.conf 2024-09-18T14:12:29.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-18T14:12:29.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: wait_for_clean 2024-09-18T14:12:29.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:12:29.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:12:29.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:12:29.596 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-18T14:12:29.597 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:12:29.597 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:12:29.597 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:12:29.597 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:12:29.597 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:12:29.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-18T14:12:29.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:12:29.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:12:29.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:12:29.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:12:29.654 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:12:30.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:12:30.048 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T14:12:30.048 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T14:12:30.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:12:30.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:12:30.048 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:12:30.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=154618822661 2024-09-18T14:12:30.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 154618822661 2024-09-18T14:12:30.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-154618822661' 2024-09-18T14:12:30.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:12:30.170 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:12:30.255 INFO:tasks.workunit.client.0.smithi047.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-18T14:12:30.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq= 2024-09-18T14:12:30.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z '' 2024-09-18T14:12:30.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: flush_pg_stats: continue 2024-09-18T14:12:30.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:12:30.257 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:12:30.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=163208757252 2024-09-18T14:12:30.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 163208757252 2024-09-18T14:12:30.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-154618822661 2-163208757252' 2024-09-18T14:12:30.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:12:30.373 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-154618822661 2024-09-18T14:12:30.373 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:12:30.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:12:30.374 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-154618822661 2024-09-18T14:12:30.374 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:12:30.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=154618822661 2024-09-18T14:12:30.376 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 154618822661 2024-09-18T14:12:30.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 154618822661' 2024-09-18T14:12:30.376 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:12:30.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 154618822659 -lt 154618822661 2024-09-18T14:12:30.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:12:31.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:12:31.818 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:12:32.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 154618822661 -lt 154618822661 2024-09-18T14:12:32.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:12:32.205 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-163208757252 2024-09-18T14:12:32.205 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:12:32.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:12:32.207 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-163208757252 2024-09-18T14:12:32.207 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:12:32.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=163208757252 2024-09-18T14:12:32.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 163208757252' 2024-09-18T14:12:32.208 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 163208757252 2024-09-18T14:12:32.209 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:12:32.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 163208757252 -lt 163208757252 2024-09-18T14:12:32.580 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:12:32.580 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:12:32.581 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:12:33.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T14:12:33.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:12:33.090 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:12:33.091 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:12:33.091 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:12:33.091 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:12:33.091 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:12:33.091 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:12:33.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:12:33.469 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:12:33.469 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:12:33.469 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:12:33.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:12:33.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-18T14:12:33.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T14:12:33.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-18T14:12:33.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:12:33.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:12:34.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:12:34.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:12:34.068 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:12:34.068 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:12:34.069 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:12:34.069 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:12:34.069 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:12:34.069 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:12:34.432 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:12:34.432 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:12:34.432 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:12:34.432 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:12:34.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:12:34.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:12:34.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:12:34.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:12:34.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:12:34.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:12:34.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:12:34.973 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:12:34.973 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:12:35.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:12:35.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:12:35.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-09-18T14:12:35.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:12:35.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-18T14:12:35.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:12:35.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:12:35.722 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:12:35.722 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:12:35.722 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:12:35.722 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:12:35.722 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:12:35.722 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:12:36.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:12:36.086 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:12:36.087 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:12:36.087 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:12:36.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:12:36.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:12:36.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:12:36.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:12:36.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:12:36.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:12:36.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:12:36.590 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:12:36.590 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:12:37.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:12:37.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:12:37.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-09-18T14:12:37.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:12:37.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-18T14:12:37.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:12:37.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:12:37.519 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:12:37.519 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:12:37.519 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:12:37.519 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:12:37.519 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:12:37.519 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:12:37.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:12:37.882 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:12:37.882 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:12:37.882 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:12:38.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:12:38.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:12:38.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:12:38.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:12:38.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:12:38.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:12:38.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:12:38.388 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:12:38.388 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:12:38.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:12:38.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:12:38.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-09-18T14:12:38.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:12:38.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-09-18T14:12:39.699 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:12:39.699 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:12:39.700 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:12:39.700 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:12:39.700 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:12:39.701 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:12:39.701 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:12:39.701 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:12:40.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:12:40.075 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:12:40.076 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:12:40.076 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:12:40.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:12:40.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:12:40.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:12:40.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:12:40.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:12:40.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:12:40.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:12:40.576 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:12:40.576 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:12:41.096 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:12:41.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:12:41.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-09-18T14:12:41.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:12:41.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-09-18T14:12:42.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:12:42.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:12:42.699 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:12:42.699 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:12:42.699 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:12:42.699 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:12:42.699 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:12:42.699 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:12:43.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:12:43.075 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:12:43.075 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:12:43.076 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:12:43.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:12:43.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:12:43.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:12:43.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:12:43.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:12:43.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:12:43.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:12:43.589 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:12:43.589 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:12:44.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:12:44.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:12:44.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-09-18T14:12:44.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:12:44.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-09-18T14:12:47.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:12:47.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:12:47.338 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:12:47.338 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:12:47.338 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:12:47.338 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:12:47.339 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:12:47.339 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:12:47.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:12:47.711 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:12:47.711 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:12:47.711 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:12:48.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:12:48.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:12:48.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:12:48.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:12:48.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:12:48.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:12:48.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:12:48.219 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:12:48.219 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:12:48.740 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:12:48.740 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:12:48.740 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-09-18T14:12:48.740 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:12:48.740 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-09-18T14:12:55.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:12:55.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:12:55.142 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:12:55.142 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:12:55.142 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:12:55.142 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:12:55.143 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:12:55.143 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:12:55.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:12:55.516 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:12:55.517 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:12:55.517 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:12:56.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:12:56.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:12:56.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:12:56.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:12:56.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:12:56.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:12:56.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:12:56.029 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:12:56.029 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:12:56.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:12:56.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:12:56.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-09-18T14:12:56.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:12:56.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-09-18T14:13:03.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:13:03.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:13:03.835 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:13:03.835 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:13:03.835 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:13:03.835 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:13:03.835 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:13:03.835 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:13:04.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:13:04.213 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:13:04.213 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:13:04.213 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:13:04.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:13:04.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:13:04.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:13:04.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:13:04.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:13:04.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:13:04.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:13:04.715 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:13:04.715 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:13:05.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:13:05.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:13:05.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-09-18T14:13:05.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-18T14:13:05.745 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T14:13:05.745 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_fingerprint": "182cb370-8208-468f-b53f-d6c2f80a20b3", 2024-09-18T14:13:05.745 INFO:tasks.workunit.client.0.smithi047.stdout: "version": "17.2.7-1645-g7061ea6c", 2024-09-18T14:13:05.745 INFO:tasks.workunit.client.0.smithi047.stdout: "commit": "7061ea6ca50908d104e9f3979dc615e118d9e153", 2024-09-18T14:13:05.745 INFO:tasks.workunit.client.0.smithi047.stdout: "timestamp": "2024-09-18T14:13:05.744620+0000", 2024-09-18T14:13:05.745 INFO:tasks.workunit.client.0.smithi047.stdout: "tag": "", 2024-09-18T14:13:05.745 INFO:tasks.workunit.client.0.smithi047.stdout: "health": { 2024-09-18T14:13:05.746 INFO:tasks.workunit.client.0.smithi047.stdout: "status": "HEALTH_WARN", 2024-09-18T14:13:05.746 INFO:tasks.workunit.client.0.smithi047.stdout: "checks": { 2024-09-18T14:13:05.746 INFO:tasks.workunit.client.0.smithi047.stdout: "OSDMAP_FLAGS": { 2024-09-18T14:13:05.746 INFO:tasks.workunit.client.0.smithi047.stdout: "severity": "HEALTH_WARN", 2024-09-18T14:13:05.746 INFO:tasks.workunit.client.0.smithi047.stdout: "summary": { 2024-09-18T14:13:05.746 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-18T14:13:05.746 INFO:tasks.workunit.client.0.smithi047.stdout: "count": 17 2024-09-18T14:13:05.746 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.746 INFO:tasks.workunit.client.0.smithi047.stdout: "detail": [], 2024-09-18T14:13:05.746 INFO:tasks.workunit.client.0.smithi047.stdout: "muted": false 2024-09-18T14:13:05.746 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.746 INFO:tasks.workunit.client.0.smithi047.stdout: "OSD_DOWN": { 2024-09-18T14:13:05.746 INFO:tasks.workunit.client.0.smithi047.stdout: "severity": "HEALTH_WARN", 2024-09-18T14:13:05.746 INFO:tasks.workunit.client.0.smithi047.stdout: "summary": { 2024-09-18T14:13:05.747 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "1 osds down", 2024-09-18T14:13:05.747 INFO:tasks.workunit.client.0.smithi047.stdout: "count": 1 2024-09-18T14:13:05.747 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.747 INFO:tasks.workunit.client.0.smithi047.stdout: "detail": [ 2024-09-18T14:13:05.747 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.747 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "osd.1 (root=default,host=smithi047) is down" 2024-09-18T14:13:05.747 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.747 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:05.747 INFO:tasks.workunit.client.0.smithi047.stdout: "muted": false 2024-09-18T14:13:05.747 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.747 INFO:tasks.workunit.client.0.smithi047.stdout: "PG_DEGRADED": { 2024-09-18T14:13:05.747 INFO:tasks.workunit.client.0.smithi047.stdout: "severity": "HEALTH_WARN", 2024-09-18T14:13:05.747 INFO:tasks.workunit.client.0.smithi047.stdout: "summary": { 2024-09-18T14:13:05.747 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2024-09-18T14:13:05.747 INFO:tasks.workunit.client.0.smithi047.stdout: "count": 1 2024-09-18T14:13:05.748 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.748 INFO:tasks.workunit.client.0.smithi047.stdout: "detail": [ 2024-09-18T14:13:05.748 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.748 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2024-09-18T14:13:05.748 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.748 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:05.748 INFO:tasks.workunit.client.0.smithi047.stdout: "muted": false 2024-09-18T14:13:05.748 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.748 INFO:tasks.workunit.client.0.smithi047.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-18T14:13:05.748 INFO:tasks.workunit.client.0.smithi047.stdout: "severity": "HEALTH_WARN", 2024-09-18T14:13:05.748 INFO:tasks.workunit.client.0.smithi047.stdout: "summary": { 2024-09-18T14:13:05.748 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-18T14:13:05.749 INFO:tasks.workunit.client.0.smithi047.stdout: "count": 1 2024-09-18T14:13:05.749 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.749 INFO:tasks.workunit.client.0.smithi047.stdout: "detail": [ 2024-09-18T14:13:05.749 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.749 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "application not enabled on pool 'test'" 2024-09-18T14:13:05.749 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.749 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.749 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-18T14:13:05.749 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.749 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:05.749 INFO:tasks.workunit.client.0.smithi047.stdout: "muted": false 2024-09-18T14:13:05.749 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.749 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.749 INFO:tasks.workunit.client.0.smithi047.stdout: "mutes": [] 2024-09-18T14:13:05.749 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.750 INFO:tasks.workunit.client.0.smithi047.stdout: "monmap_first_committed": 1, 2024-09-18T14:13:05.750 INFO:tasks.workunit.client.0.smithi047.stdout: "monmap_last_committed": 1, 2024-09-18T14:13:05.750 INFO:tasks.workunit.client.0.smithi047.stdout: "monmap": { 2024-09-18T14:13:05.750 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 1, 2024-09-18T14:13:05.750 INFO:tasks.workunit.client.0.smithi047.stdout: "fsid": "d7330bd7-8062-424a-bbd8-8c587903c456", 2024-09-18T14:13:05.750 INFO:tasks.workunit.client.0.smithi047.stdout: "modified": "2024-09-18T14:09:28.759690Z", 2024-09-18T14:13:05.750 INFO:tasks.workunit.client.0.smithi047.stdout: "created": "2024-09-18T14:09:28.759690Z", 2024-09-18T14:13:05.750 INFO:tasks.workunit.client.0.smithi047.stdout: "min_mon_release": 17, 2024-09-18T14:13:05.750 INFO:tasks.workunit.client.0.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-18T14:13:05.750 INFO:tasks.workunit.client.0.smithi047.stdout: "election_strategy": 1, 2024-09-18T14:13:05.750 INFO:tasks.workunit.client.0.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-18T14:13:05.750 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_mode": false, 2024-09-18T14:13:05.750 INFO:tasks.workunit.client.0.smithi047.stdout: "tiebreaker_mon": "", 2024-09-18T14:13:05.750 INFO:tasks.workunit.client.0.smithi047.stdout: "removed_ranks: ": "", 2024-09-18T14:13:05.750 INFO:tasks.workunit.client.0.smithi047.stdout: "features": { 2024-09-18T14:13:05.751 INFO:tasks.workunit.client.0.smithi047.stdout: "persistent": [ 2024-09-18T14:13:05.751 INFO:tasks.workunit.client.0.smithi047.stdout: "kraken", 2024-09-18T14:13:05.751 INFO:tasks.workunit.client.0.smithi047.stdout: "luminous", 2024-09-18T14:13:05.751 INFO:tasks.workunit.client.0.smithi047.stdout: "mimic", 2024-09-18T14:13:05.751 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap-prune", 2024-09-18T14:13:05.751 INFO:tasks.workunit.client.0.smithi047.stdout: "nautilus", 2024-09-18T14:13:05.751 INFO:tasks.workunit.client.0.smithi047.stdout: "octopus", 2024-09-18T14:13:05.751 INFO:tasks.workunit.client.0.smithi047.stdout: "pacific", 2024-09-18T14:13:05.751 INFO:tasks.workunit.client.0.smithi047.stdout: "elector-pinging", 2024-09-18T14:13:05.751 INFO:tasks.workunit.client.0.smithi047.stdout: "quincy" 2024-09-18T14:13:05.752 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:05.752 INFO:tasks.workunit.client.0.smithi047.stdout: "optional": [] 2024-09-18T14:13:05.752 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.752 INFO:tasks.workunit.client.0.smithi047.stdout: "mons": [ 2024-09-18T14:13:05.752 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.752 INFO:tasks.workunit.client.0.smithi047.stdout: "rank": 0, 2024-09-18T14:13:05.752 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "a", 2024-09-18T14:13:05.752 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addrs": { 2024-09-18T14:13:05.752 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:13:05.752 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.752 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:13:05.752 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:7115", 2024-09-18T14:13:05.752 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 0 2024-09-18T14:13:05.752 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.753 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:05.753 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.753 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:7115/0", 2024-09-18T14:13:05.753 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-18T14:13:05.753 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 0, 2024-09-18T14:13:05.753 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 0, 2024-09-18T14:13:05.753 INFO:tasks.workunit.client.0.smithi047.stdout: "crush_location": "{}" 2024-09-18T14:13:05.753 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.753 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:05.753 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.753 INFO:tasks.workunit.client.0.smithi047.stdout: "quorum": [ 2024-09-18T14:13:05.754 INFO:tasks.workunit.client.0.smithi047.stdout: 0 2024-09-18T14:13:05.754 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:05.754 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap": { 2024-09-18T14:13:05.754 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 41, 2024-09-18T14:13:05.754 INFO:tasks.workunit.client.0.smithi047.stdout: "fsid": "d7330bd7-8062-424a-bbd8-8c587903c456", 2024-09-18T14:13:05.754 INFO:tasks.workunit.client.0.smithi047.stdout: "created": "2024-09-18T14:09:29.543271+0000", 2024-09-18T14:13:05.754 INFO:tasks.workunit.client.0.smithi047.stdout: "modified": "2024-09-18T14:12:24.172651+0000", 2024-09-18T14:13:05.754 INFO:tasks.workunit.client.0.smithi047.stdout: "last_up_change": "2024-09-18T14:12:21.580310+0000", 2024-09-18T14:13:05.754 INFO:tasks.workunit.client.0.smithi047.stdout: "last_in_change": "2024-09-18T14:12:23.898951+0000", 2024-09-18T14:13:05.754 INFO:tasks.workunit.client.0.smithi047.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-18T14:13:05.754 INFO:tasks.workunit.client.0.smithi047.stdout: "flags_num": 5800384, 2024-09-18T14:13:05.754 INFO:tasks.workunit.client.0.smithi047.stdout: "flags_set": [ 2024-09-18T14:13:05.755 INFO:tasks.workunit.client.0.smithi047.stdout: "nodown", 2024-09-18T14:13:05.755 INFO:tasks.workunit.client.0.smithi047.stdout: "noin", 2024-09-18T14:13:05.755 INFO:tasks.workunit.client.0.smithi047.stdout: "noout", 2024-09-18T14:13:05.755 INFO:tasks.workunit.client.0.smithi047.stdout: "pglog_hardlimit", 2024-09-18T14:13:05.755 INFO:tasks.workunit.client.0.smithi047.stdout: "purged_snapdirs", 2024-09-18T14:13:05.755 INFO:tasks.workunit.client.0.smithi047.stdout: "recovery_deletes", 2024-09-18T14:13:05.755 INFO:tasks.workunit.client.0.smithi047.stdout: "sortbitwise" 2024-09-18T14:13:05.755 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:05.755 INFO:tasks.workunit.client.0.smithi047.stdout: "crush_version": 10, 2024-09-18T14:13:05.755 INFO:tasks.workunit.client.0.smithi047.stdout: "full_ratio": 0.99000000953674316, 2024-09-18T14:13:05.755 INFO:tasks.workunit.client.0.smithi047.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-18T14:13:05.755 INFO:tasks.workunit.client.0.smithi047.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-18T14:13:05.755 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_snapshot": "", 2024-09-18T14:13:05.755 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_max": 1, 2024-09-18T14:13:05.756 INFO:tasks.workunit.client.0.smithi047.stdout: "max_osd": 3, 2024-09-18T14:13:05.756 INFO:tasks.workunit.client.0.smithi047.stdout: "require_min_compat_client": "luminous", 2024-09-18T14:13:05.756 INFO:tasks.workunit.client.0.smithi047.stdout: "min_compat_client": "jewel", 2024-09-18T14:13:05.756 INFO:tasks.workunit.client.0.smithi047.stdout: "require_osd_release": "quincy", 2024-09-18T14:13:05.756 INFO:tasks.workunit.client.0.smithi047.stdout: "pools": [ 2024-09-18T14:13:05.756 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.756 INFO:tasks.workunit.client.0.smithi047.stdout: "pool": 1, 2024-09-18T14:13:05.756 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_name": "test", 2024-09-18T14:13:05.756 INFO:tasks.workunit.client.0.smithi047.stdout: "create_time": "2024-09-18T14:11:14.265067+0000", 2024-09-18T14:13:05.756 INFO:tasks.workunit.client.0.smithi047.stdout: "flags": 16389, 2024-09-18T14:13:05.756 INFO:tasks.workunit.client.0.smithi047.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2024-09-18T14:13:05.756 INFO:tasks.workunit.client.0.smithi047.stdout: "type": 3, 2024-09-18T14:13:05.756 INFO:tasks.workunit.client.0.smithi047.stdout: "size": 3, 2024-09-18T14:13:05.756 INFO:tasks.workunit.client.0.smithi047.stdout: "min_size": 2, 2024-09-18T14:13:05.757 INFO:tasks.workunit.client.0.smithi047.stdout: "crush_rule": 1, 2024-09-18T14:13:05.757 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_count": 0, 2024-09-18T14:13:05.757 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_target": 0, 2024-09-18T14:13:05.757 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T14:13:05.757 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T14:13:05.757 INFO:tasks.workunit.client.0.smithi047.stdout: "object_hash": 2, 2024-09-18T14:13:05.757 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_autoscale_mode": "off", 2024-09-18T14:13:05.757 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_num": 1, 2024-09-18T14:13:05.757 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_placement_num": 1, 2024-09-18T14:13:05.757 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_placement_num_target": 1, 2024-09-18T14:13:05.757 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_num_target": 1, 2024-09-18T14:13:05.757 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_num_pending": 1, 2024-09-18T14:13:05.757 INFO:tasks.workunit.client.0.smithi047.stdout: "last_pg_merge_meta": { 2024-09-18T14:13:05.758 INFO:tasks.workunit.client.0.smithi047.stdout: "source_pgid": "0.0", 2024-09-18T14:13:05.758 INFO:tasks.workunit.client.0.smithi047.stdout: "ready_epoch": 0, 2024-09-18T14:13:05.758 INFO:tasks.workunit.client.0.smithi047.stdout: "last_epoch_started": 0, 2024-09-18T14:13:05.758 INFO:tasks.workunit.client.0.smithi047.stdout: "last_epoch_clean": 0, 2024-09-18T14:13:05.758 INFO:tasks.workunit.client.0.smithi047.stdout: "source_version": "0'0", 2024-09-18T14:13:05.758 INFO:tasks.workunit.client.0.smithi047.stdout: "target_version": "0'0" 2024-09-18T14:13:05.758 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.758 INFO:tasks.workunit.client.0.smithi047.stdout: "last_change": "29", 2024-09-18T14:13:05.758 INFO:tasks.workunit.client.0.smithi047.stdout: "last_force_op_resend": "0", 2024-09-18T14:13:05.758 INFO:tasks.workunit.client.0.smithi047.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T14:13:05.759 INFO:tasks.workunit.client.0.smithi047.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T14:13:05.759 INFO:tasks.workunit.client.0.smithi047.stdout: "auid": 0, 2024-09-18T14:13:05.759 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_mode": "pool", 2024-09-18T14:13:05.759 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_seq": 1, 2024-09-18T14:13:05.759 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_epoch": 29, 2024-09-18T14:13:05.759 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_snaps": [ 2024-09-18T14:13:05.759 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.759 INFO:tasks.workunit.client.0.smithi047.stdout: "snapid": 1, 2024-09-18T14:13:05.759 INFO:tasks.workunit.client.0.smithi047.stdout: "stamp": "2024-09-18T14:11:58.955067+0000", 2024-09-18T14:13:05.759 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "snap1" 2024-09-18T14:13:05.759 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.759 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:05.759 INFO:tasks.workunit.client.0.smithi047.stdout: "removed_snaps": "[]", 2024-09-18T14:13:05.759 INFO:tasks.workunit.client.0.smithi047.stdout: "quota_max_bytes": 0, 2024-09-18T14:13:05.760 INFO:tasks.workunit.client.0.smithi047.stdout: "quota_max_objects": 0, 2024-09-18T14:13:05.760 INFO:tasks.workunit.client.0.smithi047.stdout: "tiers": [], 2024-09-18T14:13:05.760 INFO:tasks.workunit.client.0.smithi047.stdout: "tier_of": -1, 2024-09-18T14:13:05.760 INFO:tasks.workunit.client.0.smithi047.stdout: "read_tier": -1, 2024-09-18T14:13:05.760 INFO:tasks.workunit.client.0.smithi047.stdout: "write_tier": -1, 2024-09-18T14:13:05.760 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_mode": "none", 2024-09-18T14:13:05.760 INFO:tasks.workunit.client.0.smithi047.stdout: "target_max_bytes": 0, 2024-09-18T14:13:05.760 INFO:tasks.workunit.client.0.smithi047.stdout: "target_max_objects": 0, 2024-09-18T14:13:05.760 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T14:13:05.760 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T14:13:05.760 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T14:13:05.761 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_min_flush_age": 0, 2024-09-18T14:13:05.761 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_min_evict_age": 0, 2024-09-18T14:13:05.761 INFO:tasks.workunit.client.0.smithi047.stdout: "erasure_code_profile": "myprofile", 2024-09-18T14:13:05.761 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_params": { 2024-09-18T14:13:05.761 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "none" 2024-09-18T14:13:05.761 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.761 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_period": 0, 2024-09-18T14:13:05.761 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_count": 0, 2024-09-18T14:13:05.761 INFO:tasks.workunit.client.0.smithi047.stdout: "use_gmt_hitset": true, 2024-09-18T14:13:05.761 INFO:tasks.workunit.client.0.smithi047.stdout: "min_read_recency_for_promote": 0, 2024-09-18T14:13:05.761 INFO:tasks.workunit.client.0.smithi047.stdout: "min_write_recency_for_promote": 0, 2024-09-18T14:13:05.761 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T14:13:05.761 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_search_last_n": 0, 2024-09-18T14:13:05.761 INFO:tasks.workunit.client.0.smithi047.stdout: "grade_table": [], 2024-09-18T14:13:05.761 INFO:tasks.workunit.client.0.smithi047.stdout: "stripe_width": 8192, 2024-09-18T14:13:05.762 INFO:tasks.workunit.client.0.smithi047.stdout: "expected_num_objects": 0, 2024-09-18T14:13:05.762 INFO:tasks.workunit.client.0.smithi047.stdout: "fast_read": false, 2024-09-18T14:13:05.762 INFO:tasks.workunit.client.0.smithi047.stdout: "options": {}, 2024-09-18T14:13:05.762 INFO:tasks.workunit.client.0.smithi047.stdout: "application_metadata": {} 2024-09-18T14:13:05.762 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.762 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:05.762 INFO:tasks.workunit.client.0.smithi047.stdout: "osds": [ 2024-09-18T14:13:05.762 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.762 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 0, 2024-09-18T14:13:05.762 INFO:tasks.workunit.client.0.smithi047.stdout: "uuid": "50835fc1-8964-4c12-b447-46f109dbb8e4", 2024-09-18T14:13:05.763 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 1, 2024-09-18T14:13:05.763 INFO:tasks.workunit.client.0.smithi047.stdout: "in": 1, 2024-09-18T14:13:05.763 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 1, 2024-09-18T14:13:05.763 INFO:tasks.workunit.client.0.smithi047.stdout: "primary_affinity": 1, 2024-09-18T14:13:05.763 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_begin": 5, 2024-09-18T14:13:05.763 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_end": 28, 2024-09-18T14:13:05.763 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 36, 2024-09-18T14:13:05.763 INFO:tasks.workunit.client.0.smithi047.stdout: "up_thru": 40, 2024-09-18T14:13:05.763 INFO:tasks.workunit.client.0.smithi047.stdout: "down_at": 30, 2024-09-18T14:13:05.763 INFO:tasks.workunit.client.0.smithi047.stdout: "lost_at": 0, 2024-09-18T14:13:05.764 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addrs": { 2024-09-18T14:13:05.764 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:13:05.764 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.764 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:13:05.764 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6802", 2024-09-18T14:13:05.764 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 83233 2024-09-18T14:13:05.764 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.764 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.764 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:13:05.765 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6803", 2024-09-18T14:13:05.765 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 83233 2024-09-18T14:13:05.765 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.765 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:05.765 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.765 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addrs": { 2024-09-18T14:13:05.765 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:13:05.765 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.765 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:13:05.765 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6804", 2024-09-18T14:13:05.765 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 83233 2024-09-18T14:13:05.765 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.765 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.765 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:13:05.765 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6805", 2024-09-18T14:13:05.766 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 83233 2024-09-18T14:13:05.766 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.766 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:05.766 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.766 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addrs": { 2024-09-18T14:13:05.766 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:13:05.766 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.766 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:13:05.766 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6808", 2024-09-18T14:13:05.766 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 83233 2024-09-18T14:13:05.766 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.766 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.766 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:13:05.766 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6809", 2024-09-18T14:13:05.766 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 83233 2024-09-18T14:13:05.767 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.767 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:05.767 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.767 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addrs": { 2024-09-18T14:13:05.767 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:13:05.767 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.767 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:13:05.767 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6806", 2024-09-18T14:13:05.767 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 83233 2024-09-18T14:13:05.767 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.767 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.767 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:13:05.767 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6807", 2024-09-18T14:13:05.767 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 83233 2024-09-18T14:13:05.767 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.768 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:05.768 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.768 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addr": "127.0.0.1:6803/83233", 2024-09-18T14:13:05.768 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addr": "127.0.0.1:6805/83233", 2024-09-18T14:13:05.768 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addr": "127.0.0.1:6809/83233", 2024-09-18T14:13:05.768 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addr": "127.0.0.1:6807/83233", 2024-09-18T14:13:05.768 INFO:tasks.workunit.client.0.smithi047.stdout: "state": [ 2024-09-18T14:13:05.768 INFO:tasks.workunit.client.0.smithi047.stdout: "exists", 2024-09-18T14:13:05.768 INFO:tasks.workunit.client.0.smithi047.stdout: "up" 2024-09-18T14:13:05.768 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:05.768 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.768 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.768 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 1, 2024-09-18T14:13:05.768 INFO:tasks.workunit.client.0.smithi047.stdout: "uuid": "f4d67a04-7819-45b5-bc8c-088b7d24eed7", 2024-09-18T14:13:05.769 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 0, 2024-09-18T14:13:05.769 INFO:tasks.workunit.client.0.smithi047.stdout: "in": 1, 2024-09-18T14:13:05.769 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 1, 2024-09-18T14:13:05.769 INFO:tasks.workunit.client.0.smithi047.stdout: "primary_affinity": 1, 2024-09-18T14:13:05.769 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_begin": 0, 2024-09-18T14:13:05.769 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_end": 0, 2024-09-18T14:13:05.769 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 10, 2024-09-18T14:13:05.769 INFO:tasks.workunit.client.0.smithi047.stdout: "up_thru": 25, 2024-09-18T14:13:05.769 INFO:tasks.workunit.client.0.smithi047.stdout: "down_at": 31, 2024-09-18T14:13:05.769 INFO:tasks.workunit.client.0.smithi047.stdout: "lost_at": 0, 2024-09-18T14:13:05.769 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addrs": { 2024-09-18T14:13:05.769 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:13:05.770 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.770 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:13:05.770 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6810", 2024-09-18T14:13:05.770 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 76864 2024-09-18T14:13:05.770 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.770 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.770 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:13:05.770 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6811", 2024-09-18T14:13:05.770 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 76864 2024-09-18T14:13:05.770 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.770 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:05.770 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.770 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addrs": { 2024-09-18T14:13:05.771 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:13:05.771 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.771 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:13:05.771 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6812", 2024-09-18T14:13:05.771 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 76864 2024-09-18T14:13:05.771 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.771 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.771 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:13:05.771 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6813", 2024-09-18T14:13:05.771 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 76864 2024-09-18T14:13:05.771 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.771 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:05.771 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.771 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addrs": { 2024-09-18T14:13:05.771 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:13:05.772 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.772 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:13:05.772 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6816", 2024-09-18T14:13:05.772 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 76864 2024-09-18T14:13:05.772 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.772 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.772 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:13:05.772 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6817", 2024-09-18T14:13:05.772 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 76864 2024-09-18T14:13:05.772 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.772 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:05.773 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.773 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addrs": { 2024-09-18T14:13:05.773 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:13:05.773 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.773 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:13:05.773 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6814", 2024-09-18T14:13:05.773 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 76864 2024-09-18T14:13:05.773 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.773 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.773 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:13:05.773 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6815", 2024-09-18T14:13:05.773 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 76864 2024-09-18T14:13:05.773 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.774 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:05.774 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.774 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addr": "127.0.0.1:6811/76864", 2024-09-18T14:13:05.774 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addr": "127.0.0.1:6813/76864", 2024-09-18T14:13:05.774 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addr": "127.0.0.1:6817/76864", 2024-09-18T14:13:05.774 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addr": "127.0.0.1:6815/76864", 2024-09-18T14:13:05.774 INFO:tasks.workunit.client.0.smithi047.stdout: "state": [ 2024-09-18T14:13:05.774 INFO:tasks.workunit.client.0.smithi047.stdout: "exists" 2024-09-18T14:13:05.774 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:05.774 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.774 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.774 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 2, 2024-09-18T14:13:05.775 INFO:tasks.workunit.client.0.smithi047.stdout: "uuid": "2d53c532-9e0f-4778-a2d9-277af85698d4", 2024-09-18T14:13:05.775 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 1, 2024-09-18T14:13:05.775 INFO:tasks.workunit.client.0.smithi047.stdout: "in": 1, 2024-09-18T14:13:05.775 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 1, 2024-09-18T14:13:05.775 INFO:tasks.workunit.client.0.smithi047.stdout: "primary_affinity": 1, 2024-09-18T14:13:05.775 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_begin": 15, 2024-09-18T14:13:05.775 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_end": 28, 2024-09-18T14:13:05.775 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 38, 2024-09-18T14:13:05.775 INFO:tasks.workunit.client.0.smithi047.stdout: "up_thru": 0, 2024-09-18T14:13:05.775 INFO:tasks.workunit.client.0.smithi047.stdout: "down_at": 32, 2024-09-18T14:13:05.775 INFO:tasks.workunit.client.0.smithi047.stdout: "lost_at": 0, 2024-09-18T14:13:05.775 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addrs": { 2024-09-18T14:13:05.775 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:13:05.775 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.776 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:13:05.776 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6810", 2024-09-18T14:13:05.776 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 83811 2024-09-18T14:13:05.776 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.776 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.776 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:13:05.776 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6811", 2024-09-18T14:13:05.776 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 83811 2024-09-18T14:13:05.776 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.776 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:05.776 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.776 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addrs": { 2024-09-18T14:13:05.776 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:13:05.776 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.776 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:13:05.777 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6812", 2024-09-18T14:13:05.777 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 83811 2024-09-18T14:13:05.777 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.777 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.777 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:13:05.777 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6813", 2024-09-18T14:13:05.777 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 83811 2024-09-18T14:13:05.777 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.777 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:05.777 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.777 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addrs": { 2024-09-18T14:13:05.777 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:13:05.777 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.777 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:13:05.777 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6816", 2024-09-18T14:13:05.778 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 83811 2024-09-18T14:13:05.778 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.778 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.778 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:13:05.778 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6817", 2024-09-18T14:13:05.778 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 83811 2024-09-18T14:13:05.778 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.778 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:05.778 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.778 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addrs": { 2024-09-18T14:13:05.778 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:13:05.778 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.778 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:13:05.778 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6814", 2024-09-18T14:13:05.778 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 83811 2024-09-18T14:13:05.779 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.779 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.779 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:13:05.779 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6815", 2024-09-18T14:13:05.779 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 83811 2024-09-18T14:13:05.779 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.779 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:05.779 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.779 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addr": "127.0.0.1:6811/83811", 2024-09-18T14:13:05.779 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addr": "127.0.0.1:6813/83811", 2024-09-18T14:13:05.779 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addr": "127.0.0.1:6817/83811", 2024-09-18T14:13:05.779 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addr": "127.0.0.1:6815/83811", 2024-09-18T14:13:05.779 INFO:tasks.workunit.client.0.smithi047.stdout: "state": [ 2024-09-18T14:13:05.779 INFO:tasks.workunit.client.0.smithi047.stdout: "exists", 2024-09-18T14:13:05.779 INFO:tasks.workunit.client.0.smithi047.stdout: "up" 2024-09-18T14:13:05.780 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:05.780 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.780 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:05.780 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_xinfo": [ 2024-09-18T14:13:05.780 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.780 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 0, 2024-09-18T14:13:05.780 INFO:tasks.workunit.client.0.smithi047.stdout: "down_stamp": "2024-09-18T14:12:09.923225+0000", 2024-09-18T14:13:05.780 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_probability": 0, 2024-09-18T14:13:05.780 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_interval": 0, 2024-09-18T14:13:05.780 INFO:tasks.workunit.client.0.smithi047.stdout: "features": 4540138320759226367, 2024-09-18T14:13:05.780 INFO:tasks.workunit.client.0.smithi047.stdout: "old_weight": 0, 2024-09-18T14:13:05.780 INFO:tasks.workunit.client.0.smithi047.stdout: "last_purged_snaps_scrub": "2024-09-18T14:10:01.318019+0000", 2024-09-18T14:13:05.780 INFO:tasks.workunit.client.0.smithi047.stdout: "dead_epoch": 0 2024-09-18T14:13:05.780 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.780 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.781 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 1, 2024-09-18T14:13:05.781 INFO:tasks.workunit.client.0.smithi047.stdout: "down_stamp": "2024-09-18T14:12:10.481921+0000", 2024-09-18T14:13:05.781 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_probability": 0, 2024-09-18T14:13:05.781 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_interval": 0, 2024-09-18T14:13:05.781 INFO:tasks.workunit.client.0.smithi047.stdout: "features": 4540138320759226367, 2024-09-18T14:13:05.781 INFO:tasks.workunit.client.0.smithi047.stdout: "old_weight": 0, 2024-09-18T14:13:05.781 INFO:tasks.workunit.client.0.smithi047.stdout: "last_purged_snaps_scrub": "2024-09-18T14:10:35.227876+0000", 2024-09-18T14:13:05.781 INFO:tasks.workunit.client.0.smithi047.stdout: "dead_epoch": 0 2024-09-18T14:13:05.781 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.781 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.782 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 2, 2024-09-18T14:13:05.782 INFO:tasks.workunit.client.0.smithi047.stdout: "down_stamp": "2024-09-18T14:12:11.067329+0000", 2024-09-18T14:13:05.782 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_probability": 0, 2024-09-18T14:13:05.782 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_interval": 0, 2024-09-18T14:13:05.782 INFO:tasks.workunit.client.0.smithi047.stdout: "features": 4540138320759226367, 2024-09-18T14:13:05.782 INFO:tasks.workunit.client.0.smithi047.stdout: "old_weight": 0, 2024-09-18T14:13:05.782 INFO:tasks.workunit.client.0.smithi047.stdout: "last_purged_snaps_scrub": "2024-09-18T14:11:08.904096+0000", 2024-09-18T14:13:05.782 INFO:tasks.workunit.client.0.smithi047.stdout: "dead_epoch": 0 2024-09-18T14:13:05.782 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.782 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:05.782 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_upmap": [], 2024-09-18T14:13:05.782 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_upmap_items": [], 2024-09-18T14:13:05.782 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_temp": [], 2024-09-18T14:13:05.782 INFO:tasks.workunit.client.0.smithi047.stdout: "primary_temp": [], 2024-09-18T14:13:05.782 INFO:tasks.workunit.client.0.smithi047.stdout: "blocklist": {}, 2024-09-18T14:13:05.783 INFO:tasks.workunit.client.0.smithi047.stdout: "range_blocklist": {}, 2024-09-18T14:13:05.783 INFO:tasks.workunit.client.0.smithi047.stdout: "erasure_code_profiles": { 2024-09-18T14:13:05.783 INFO:tasks.workunit.client.0.smithi047.stdout: "default": { 2024-09-18T14:13:05.783 INFO:tasks.workunit.client.0.smithi047.stdout: "k": "2", 2024-09-18T14:13:05.783 INFO:tasks.workunit.client.0.smithi047.stdout: "m": "2", 2024-09-18T14:13:05.783 INFO:tasks.workunit.client.0.smithi047.stdout: "plugin": "jerasure", 2024-09-18T14:13:05.783 INFO:tasks.workunit.client.0.smithi047.stdout: "technique": "reed_sol_van" 2024-09-18T14:13:05.783 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.783 INFO:tasks.workunit.client.0.smithi047.stdout: "myprofile": { 2024-09-18T14:13:05.783 INFO:tasks.workunit.client.0.smithi047.stdout: "crush-device-class": "", 2024-09-18T14:13:05.783 INFO:tasks.workunit.client.0.smithi047.stdout: "crush-failure-domain": "osd", 2024-09-18T14:13:05.783 INFO:tasks.workunit.client.0.smithi047.stdout: "crush-root": "default", 2024-09-18T14:13:05.783 INFO:tasks.workunit.client.0.smithi047.stdout: "jerasure-per-chunk-alignment": "false", 2024-09-18T14:13:05.783 INFO:tasks.workunit.client.0.smithi047.stdout: "k": "2", 2024-09-18T14:13:05.783 INFO:tasks.workunit.client.0.smithi047.stdout: "m": "1", 2024-09-18T14:13:05.784 INFO:tasks.workunit.client.0.smithi047.stdout: "plugin": "jerasure", 2024-09-18T14:13:05.784 INFO:tasks.workunit.client.0.smithi047.stdout: "technique": "reed_sol_van", 2024-09-18T14:13:05.784 INFO:tasks.workunit.client.0.smithi047.stdout: "w": "8" 2024-09-18T14:13:05.784 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.784 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.784 INFO:tasks.workunit.client.0.smithi047.stdout: "removed_snaps_queue": [], 2024-09-18T14:13:05.784 INFO:tasks.workunit.client.0.smithi047.stdout: "new_removed_snaps": [], 2024-09-18T14:13:05.784 INFO:tasks.workunit.client.0.smithi047.stdout: "new_purged_snaps": [], 2024-09-18T14:13:05.784 INFO:tasks.workunit.client.0.smithi047.stdout: "crush_node_flags": {}, 2024-09-18T14:13:05.784 INFO:tasks.workunit.client.0.smithi047.stdout: "device_class_flags": {}, 2024-09-18T14:13:05.784 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_mode": { 2024-09-18T14:13:05.784 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_mode_enabled": false, 2024-09-18T14:13:05.784 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_bucket_count": 0, 2024-09-18T14:13:05.784 INFO:tasks.workunit.client.0.smithi047.stdout: "degraded_stretch_mode": 0, 2024-09-18T14:13:05.784 INFO:tasks.workunit.client.0.smithi047.stdout: "recovering_stretch_mode": 0, 2024-09-18T14:13:05.785 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_mode_bucket": 0 2024-09-18T14:13:05.785 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.785 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.785 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_metadata": [ 2024-09-18T14:13:05.785 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.785 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T14:13:05.785 INFO:tasks.workunit.client.0.smithi047.stdout: "arch": "x86_64", 2024-09-18T14:13:05.785 INFO:tasks.workunit.client.0.smithi047.stdout: "back_addr": "[v2:127.0.0.1:6804/83233,v1:127.0.0.1:6805/83233]", 2024-09-18T14:13:05.785 INFO:tasks.workunit.client.0.smithi047.stdout: "back_iface": "", 2024-09-18T14:13:05.785 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs": "1", 2024-09-18T14:13:05.785 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_db": "0", 2024-09-18T14:13:05.785 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_wal": "0", 2024-09-18T14:13:05.785 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_single_shared_device": "1", 2024-09-18T14:13:05.786 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_allocation_from_file": "0", 2024-09-18T14:13:05.786 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_access_mode": "file", 2024-09-18T14:13:05.786 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_block_size": "4096", 2024-09-18T14:13:05.786 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_devices": "sda", 2024-09-18T14:13:05.786 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-18T14:13:05.786 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-18T14:13:05.786 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-18T14:13:05.786 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_rotational": "1", 2024-09-18T14:13:05.786 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_size": "107374182400", 2024-09-18T14:13:05.786 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_support_discard": "1", 2024-09-18T14:13:05.786 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_type": "hdd", 2024-09-18T14:13:05.786 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_min_alloc_size": "4096", 2024-09-18T14:13:05.786 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_release": "quincy", 2024-09-18T14:13:05.786 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:13:05.787 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_short": "17.2.7-1645-g7061ea6c", 2024-09-18T14:13:05.787 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_when_created": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:13:05.787 INFO:tasks.workunit.client.0.smithi047.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-18T14:13:05.787 INFO:tasks.workunit.client.0.smithi047.stdout: "created_at": "2024-09-18T14:09:35.435983Z", 2024-09-18T14:13:05.787 INFO:tasks.workunit.client.0.smithi047.stdout: "default_device_class": "hdd", 2024-09-18T14:13:05.787 INFO:tasks.workunit.client.0.smithi047.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4HN80", 2024-09-18T14:13:05.787 INFO:tasks.workunit.client.0.smithi047.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-18T14:13:05.787 INFO:tasks.workunit.client.0.smithi047.stdout: "devices": "sda", 2024-09-18T14:13:05.787 INFO:tasks.workunit.client.0.smithi047.stdout: "distro": "centos", 2024-09-18T14:13:05.787 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_description": "CentOS Stream 9", 2024-09-18T14:13:05.787 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_version": "9", 2024-09-18T14:13:05.787 INFO:tasks.workunit.client.0.smithi047.stdout: "front_addr": "[v2:127.0.0.1:6802/83233,v1:127.0.0.1:6803/83233]", 2024-09-18T14:13:05.787 INFO:tasks.workunit.client.0.smithi047.stdout: "front_iface": "", 2024-09-18T14:13:05.787 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/83233,v1:127.0.0.1:6809/83233]", 2024-09-18T14:13:05.787 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/83233,v1:127.0.0.1:6807/83233]", 2024-09-18T14:13:05.788 INFO:tasks.workunit.client.0.smithi047.stdout: "hostname": "smithi047", 2024-09-18T14:13:05.788 INFO:tasks.workunit.client.0.smithi047.stdout: "journal_rotational": "1", 2024-09-18T14:13:05.788 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-18T14:13:05.788 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-18T14:13:05.788 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_swap_kb": "0", 2024-09-18T14:13:05.788 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_total_kb": "32493492", 2024-09-18T14:13:05.788 INFO:tasks.workunit.client.0.smithi047.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-18T14:13:05.788 INFO:tasks.workunit.client.0.smithi047.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-18T14:13:05.788 INFO:tasks.workunit.client.0.smithi047.stdout: "os": "Linux", 2024-09-18T14:13:05.788 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_data": "td/divergent-priors/0", 2024-09-18T14:13:05.788 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_objectstore": "bluestore", 2024-09-18T14:13:05.788 INFO:tasks.workunit.client.0.smithi047.stdout: "osdspec_affinity": "", 2024-09-18T14:13:05.788 INFO:tasks.workunit.client.0.smithi047.stdout: "rotational": "1" 2024-09-18T14:13:05.788 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.788 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.789 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 1, 2024-09-18T14:13:05.789 INFO:tasks.workunit.client.0.smithi047.stdout: "arch": "x86_64", 2024-09-18T14:13:05.789 INFO:tasks.workunit.client.0.smithi047.stdout: "back_addr": "[v2:127.0.0.1:6812/76864,v1:127.0.0.1:6813/76864]", 2024-09-18T14:13:05.789 INFO:tasks.workunit.client.0.smithi047.stdout: "back_iface": "", 2024-09-18T14:13:05.789 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs": "1", 2024-09-18T14:13:05.789 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_db": "0", 2024-09-18T14:13:05.789 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_wal": "0", 2024-09-18T14:13:05.789 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_single_shared_device": "1", 2024-09-18T14:13:05.789 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_allocation_from_file": "0", 2024-09-18T14:13:05.789 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_access_mode": "file", 2024-09-18T14:13:05.789 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_block_size": "4096", 2024-09-18T14:13:05.789 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_devices": "sda", 2024-09-18T14:13:05.789 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-18T14:13:05.789 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-18T14:13:05.790 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-18T14:13:05.790 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_rotational": "1", 2024-09-18T14:13:05.790 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_size": "107374182400", 2024-09-18T14:13:05.790 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_support_discard": "1", 2024-09-18T14:13:05.790 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_type": "hdd", 2024-09-18T14:13:05.790 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_min_alloc_size": "4096", 2024-09-18T14:13:05.790 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_release": "quincy", 2024-09-18T14:13:05.790 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:13:05.790 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_short": "17.2.7-1645-g7061ea6c", 2024-09-18T14:13:05.790 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_when_created": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:13:05.790 INFO:tasks.workunit.client.0.smithi047.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-18T14:13:05.790 INFO:tasks.workunit.client.0.smithi047.stdout: "created_at": "2024-09-18T14:10:08.245708Z", 2024-09-18T14:13:05.791 INFO:tasks.workunit.client.0.smithi047.stdout: "default_device_class": "hdd", 2024-09-18T14:13:05.791 INFO:tasks.workunit.client.0.smithi047.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4HN80", 2024-09-18T14:13:05.791 INFO:tasks.workunit.client.0.smithi047.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-18T14:13:05.791 INFO:tasks.workunit.client.0.smithi047.stdout: "devices": "sda", 2024-09-18T14:13:05.791 INFO:tasks.workunit.client.0.smithi047.stdout: "distro": "centos", 2024-09-18T14:13:05.791 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_description": "CentOS Stream 9", 2024-09-18T14:13:05.791 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_version": "9", 2024-09-18T14:13:05.791 INFO:tasks.workunit.client.0.smithi047.stdout: "front_addr": "[v2:127.0.0.1:6810/76864,v1:127.0.0.1:6811/76864]", 2024-09-18T14:13:05.791 INFO:tasks.workunit.client.0.smithi047.stdout: "front_iface": "", 2024-09-18T14:13:05.791 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/76864,v1:127.0.0.1:6817/76864]", 2024-09-18T14:13:05.791 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/76864,v1:127.0.0.1:6815/76864]", 2024-09-18T14:13:05.791 INFO:tasks.workunit.client.0.smithi047.stdout: "hostname": "smithi047", 2024-09-18T14:13:05.791 INFO:tasks.workunit.client.0.smithi047.stdout: "journal_rotational": "1", 2024-09-18T14:13:05.791 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-18T14:13:05.792 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-18T14:13:05.792 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_swap_kb": "0", 2024-09-18T14:13:05.792 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_total_kb": "32493492", 2024-09-18T14:13:05.792 INFO:tasks.workunit.client.0.smithi047.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-18T14:13:05.792 INFO:tasks.workunit.client.0.smithi047.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-18T14:13:05.792 INFO:tasks.workunit.client.0.smithi047.stdout: "os": "Linux", 2024-09-18T14:13:05.792 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_data": "td/divergent-priors/1", 2024-09-18T14:13:05.792 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_objectstore": "bluestore", 2024-09-18T14:13:05.792 INFO:tasks.workunit.client.0.smithi047.stdout: "osdspec_affinity": "", 2024-09-18T14:13:05.792 INFO:tasks.workunit.client.0.smithi047.stdout: "rotational": "1" 2024-09-18T14:13:05.792 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.792 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.792 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T14:13:05.792 INFO:tasks.workunit.client.0.smithi047.stdout: "arch": "x86_64", 2024-09-18T14:13:05.792 INFO:tasks.workunit.client.0.smithi047.stdout: "back_addr": "[v2:127.0.0.1:6812/83811,v1:127.0.0.1:6813/83811]", 2024-09-18T14:13:05.793 INFO:tasks.workunit.client.0.smithi047.stdout: "back_iface": "", 2024-09-18T14:13:05.793 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs": "1", 2024-09-18T14:13:05.793 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_db": "0", 2024-09-18T14:13:05.793 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_wal": "0", 2024-09-18T14:13:05.793 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_single_shared_device": "1", 2024-09-18T14:13:05.793 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_allocation_from_file": "0", 2024-09-18T14:13:05.793 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_access_mode": "file", 2024-09-18T14:13:05.793 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_block_size": "4096", 2024-09-18T14:13:05.793 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_devices": "sda", 2024-09-18T14:13:05.793 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-18T14:13:05.793 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-18T14:13:05.793 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-18T14:13:05.793 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_rotational": "1", 2024-09-18T14:13:05.793 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_size": "107374182400", 2024-09-18T14:13:05.793 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_support_discard": "1", 2024-09-18T14:13:05.794 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_type": "hdd", 2024-09-18T14:13:05.794 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_min_alloc_size": "4096", 2024-09-18T14:13:05.794 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_release": "quincy", 2024-09-18T14:13:05.794 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:13:05.794 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_short": "17.2.7-1645-g7061ea6c", 2024-09-18T14:13:05.794 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_when_created": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:13:05.794 INFO:tasks.workunit.client.0.smithi047.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-18T14:13:05.794 INFO:tasks.workunit.client.0.smithi047.stdout: "created_at": "2024-09-18T14:10:42.506913Z", 2024-09-18T14:13:05.794 INFO:tasks.workunit.client.0.smithi047.stdout: "default_device_class": "hdd", 2024-09-18T14:13:05.794 INFO:tasks.workunit.client.0.smithi047.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4HN80", 2024-09-18T14:13:05.795 INFO:tasks.workunit.client.0.smithi047.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-18T14:13:05.795 INFO:tasks.workunit.client.0.smithi047.stdout: "devices": "sda", 2024-09-18T14:13:05.795 INFO:tasks.workunit.client.0.smithi047.stdout: "distro": "centos", 2024-09-18T14:13:05.795 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_description": "CentOS Stream 9", 2024-09-18T14:13:05.795 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_version": "9", 2024-09-18T14:13:05.795 INFO:tasks.workunit.client.0.smithi047.stdout: "front_addr": "[v2:127.0.0.1:6810/83811,v1:127.0.0.1:6811/83811]", 2024-09-18T14:13:05.795 INFO:tasks.workunit.client.0.smithi047.stdout: "front_iface": "", 2024-09-18T14:13:05.795 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/83811,v1:127.0.0.1:6817/83811]", 2024-09-18T14:13:05.795 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/83811,v1:127.0.0.1:6815/83811]", 2024-09-18T14:13:05.795 INFO:tasks.workunit.client.0.smithi047.stdout: "hostname": "smithi047", 2024-09-18T14:13:05.795 INFO:tasks.workunit.client.0.smithi047.stdout: "journal_rotational": "1", 2024-09-18T14:13:05.796 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-18T14:13:05.796 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-18T14:13:05.796 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_swap_kb": "0", 2024-09-18T14:13:05.796 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_total_kb": "32493492", 2024-09-18T14:13:05.796 INFO:tasks.workunit.client.0.smithi047.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-18T14:13:05.796 INFO:tasks.workunit.client.0.smithi047.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-18T14:13:05.796 INFO:tasks.workunit.client.0.smithi047.stdout: "os": "Linux", 2024-09-18T14:13:05.796 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_data": "td/divergent-priors/2", 2024-09-18T14:13:05.796 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_objectstore": "bluestore", 2024-09-18T14:13:05.796 INFO:tasks.workunit.client.0.smithi047.stdout: "osdspec_affinity": "", 2024-09-18T14:13:05.796 INFO:tasks.workunit.client.0.smithi047.stdout: "rotational": "1" 2024-09-18T14:13:05.797 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.797 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:05.797 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap_clean_epochs": { 2024-09-18T14:13:05.797 INFO:tasks.workunit.client.0.smithi047.stdout: "min_last_epoch_clean": 0, 2024-09-18T14:13:05.797 INFO:tasks.workunit.client.0.smithi047.stdout: "last_epoch_clean": { 2024-09-18T14:13:05.797 INFO:tasks.workunit.client.0.smithi047.stdout: "per_pool": [] 2024-09-18T14:13:05.797 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.797 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_epochs": [ 2024-09-18T14:13:05.797 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.797 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T14:13:05.797 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 36 2024-09-18T14:13:05.797 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.797 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.797 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T14:13:05.797 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 38 2024-09-18T14:13:05.798 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.798 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:05.798 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.798 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap_first_committed": 1, 2024-09-18T14:13:05.798 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap_last_committed": 41, 2024-09-18T14:13:05.798 INFO:tasks.workunit.client.0.smithi047.stdout: "crushmap": { 2024-09-18T14:13:05.798 INFO:tasks.workunit.client.0.smithi047.stdout: "devices": [ 2024-09-18T14:13:05.798 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.798 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T14:13:05.798 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "osd.0", 2024-09-18T14:13:05.798 INFO:tasks.workunit.client.0.smithi047.stdout: "class": "hdd" 2024-09-18T14:13:05.798 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.798 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.798 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 1, 2024-09-18T14:13:05.798 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "osd.1", 2024-09-18T14:13:05.799 INFO:tasks.workunit.client.0.smithi047.stdout: "class": "hdd" 2024-09-18T14:13:05.799 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.799 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.799 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T14:13:05.799 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "osd.2", 2024-09-18T14:13:05.799 INFO:tasks.workunit.client.0.smithi047.stdout: "class": "hdd" 2024-09-18T14:13:05.799 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.799 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:05.799 INFO:tasks.workunit.client.0.smithi047.stdout: "types": [ 2024-09-18T14:13:05.799 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.799 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 0, 2024-09-18T14:13:05.799 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "osd" 2024-09-18T14:13:05.799 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.799 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.799 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 1, 2024-09-18T14:13:05.799 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "host" 2024-09-18T14:13:05.800 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.800 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.800 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 2, 2024-09-18T14:13:05.800 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "chassis" 2024-09-18T14:13:05.800 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.800 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.800 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 3, 2024-09-18T14:13:05.800 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "rack" 2024-09-18T14:13:05.800 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.800 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.800 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 4, 2024-09-18T14:13:05.801 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "row" 2024-09-18T14:13:05.801 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.801 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.801 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 5, 2024-09-18T14:13:05.801 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "pdu" 2024-09-18T14:13:05.801 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.801 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.801 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 6, 2024-09-18T14:13:05.801 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "pod" 2024-09-18T14:13:05.801 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.801 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.801 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 7, 2024-09-18T14:13:05.801 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "room" 2024-09-18T14:13:05.801 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.801 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.802 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 8, 2024-09-18T14:13:05.802 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "datacenter" 2024-09-18T14:13:05.802 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.802 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.802 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 9, 2024-09-18T14:13:05.802 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "zone" 2024-09-18T14:13:05.802 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.802 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.802 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 10, 2024-09-18T14:13:05.802 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "region" 2024-09-18T14:13:05.802 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.802 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.802 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 11, 2024-09-18T14:13:05.802 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "root" 2024-09-18T14:13:05.802 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.803 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:05.803 INFO:tasks.workunit.client.0.smithi047.stdout: "buckets": [ 2024-09-18T14:13:05.803 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.803 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -1, 2024-09-18T14:13:05.803 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "default", 2024-09-18T14:13:05.803 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 11, 2024-09-18T14:13:05.803 INFO:tasks.workunit.client.0.smithi047.stdout: "type_name": "root", 2024-09-18T14:13:05.803 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:13:05.803 INFO:tasks.workunit.client.0.smithi047.stdout: "alg": "straw2", 2024-09-18T14:13:05.803 INFO:tasks.workunit.client.0.smithi047.stdout: "hash": "rjenkins1", 2024-09-18T14:13:05.803 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T14:13:05.803 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.803 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -3, 2024-09-18T14:13:05.803 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:13:05.803 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 0 2024-09-18T14:13:05.804 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.804 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:05.804 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.804 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.804 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -2, 2024-09-18T14:13:05.804 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "default~hdd", 2024-09-18T14:13:05.804 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 11, 2024-09-18T14:13:05.804 INFO:tasks.workunit.client.0.smithi047.stdout: "type_name": "root", 2024-09-18T14:13:05.804 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:13:05.804 INFO:tasks.workunit.client.0.smithi047.stdout: "alg": "straw2", 2024-09-18T14:13:05.804 INFO:tasks.workunit.client.0.smithi047.stdout: "hash": "rjenkins1", 2024-09-18T14:13:05.804 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T14:13:05.804 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.804 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -4, 2024-09-18T14:13:05.805 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:13:05.805 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 0 2024-09-18T14:13:05.805 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.805 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:05.805 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.805 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.805 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -3, 2024-09-18T14:13:05.805 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "smithi047", 2024-09-18T14:13:05.805 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 1, 2024-09-18T14:13:05.805 INFO:tasks.workunit.client.0.smithi047.stdout: "type_name": "host", 2024-09-18T14:13:05.805 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:13:05.805 INFO:tasks.workunit.client.0.smithi047.stdout: "alg": "straw2", 2024-09-18T14:13:05.805 INFO:tasks.workunit.client.0.smithi047.stdout: "hash": "rjenkins1", 2024-09-18T14:13:05.805 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T14:13:05.805 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.806 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T14:13:05.806 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:13:05.806 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 0 2024-09-18T14:13:05.806 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.806 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.806 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 1, 2024-09-18T14:13:05.806 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:13:05.806 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 1 2024-09-18T14:13:05.806 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.806 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.806 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T14:13:05.806 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:13:05.807 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 2 2024-09-18T14:13:05.807 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.807 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:05.807 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.807 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.807 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -4, 2024-09-18T14:13:05.807 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "smithi047~hdd", 2024-09-18T14:13:05.807 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 1, 2024-09-18T14:13:05.807 INFO:tasks.workunit.client.0.smithi047.stdout: "type_name": "host", 2024-09-18T14:13:05.807 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:13:05.807 INFO:tasks.workunit.client.0.smithi047.stdout: "alg": "straw2", 2024-09-18T14:13:05.807 INFO:tasks.workunit.client.0.smithi047.stdout: "hash": "rjenkins1", 2024-09-18T14:13:05.808 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T14:13:05.808 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.808 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T14:13:05.808 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:13:05.808 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 0 2024-09-18T14:13:05.808 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.808 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.808 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 1, 2024-09-18T14:13:05.808 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:13:05.808 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 1 2024-09-18T14:13:05.808 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.809 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.809 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T14:13:05.809 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:13:05.809 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 2 2024-09-18T14:13:05.809 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.809 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:05.809 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.809 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:05.809 INFO:tasks.workunit.client.0.smithi047.stdout: "rules": [ 2024-09-18T14:13:05.809 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.809 INFO:tasks.workunit.client.0.smithi047.stdout: "rule_id": 0, 2024-09-18T14:13:05.809 INFO:tasks.workunit.client.0.smithi047.stdout: "rule_name": "replicated_rule", 2024-09-18T14:13:05.809 INFO:tasks.workunit.client.0.smithi047.stdout: "type": 1, 2024-09-18T14:13:05.809 INFO:tasks.workunit.client.0.smithi047.stdout: "steps": [ 2024-09-18T14:13:05.809 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.810 INFO:tasks.workunit.client.0.smithi047.stdout: "op": "take", 2024-09-18T14:13:05.810 INFO:tasks.workunit.client.0.smithi047.stdout: "item": -1, 2024-09-18T14:13:05.810 INFO:tasks.workunit.client.0.smithi047.stdout: "item_name": "default" 2024-09-18T14:13:05.810 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.810 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.810 INFO:tasks.workunit.client.0.smithi047.stdout: "op": "choose_firstn", 2024-09-18T14:13:05.810 INFO:tasks.workunit.client.0.smithi047.stdout: "num": 0, 2024-09-18T14:13:05.810 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "osd" 2024-09-18T14:13:05.810 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.810 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.810 INFO:tasks.workunit.client.0.smithi047.stdout: "op": "emit" 2024-09-18T14:13:05.810 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.810 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:05.810 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.810 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.811 INFO:tasks.workunit.client.0.smithi047.stdout: "rule_id": 1, 2024-09-18T14:13:05.811 INFO:tasks.workunit.client.0.smithi047.stdout: "rule_name": "test", 2024-09-18T14:13:05.811 INFO:tasks.workunit.client.0.smithi047.stdout: "type": 3, 2024-09-18T14:13:05.811 INFO:tasks.workunit.client.0.smithi047.stdout: "steps": [ 2024-09-18T14:13:05.811 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.811 INFO:tasks.workunit.client.0.smithi047.stdout: "op": "set_chooseleaf_tries", 2024-09-18T14:13:05.811 INFO:tasks.workunit.client.0.smithi047.stdout: "num": 5 2024-09-18T14:13:05.811 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.811 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.811 INFO:tasks.workunit.client.0.smithi047.stdout: "op": "set_choose_tries", 2024-09-18T14:13:05.811 INFO:tasks.workunit.client.0.smithi047.stdout: "num": 100 2024-09-18T14:13:05.811 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.811 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.811 INFO:tasks.workunit.client.0.smithi047.stdout: "op": "take", 2024-09-18T14:13:05.811 INFO:tasks.workunit.client.0.smithi047.stdout: "item": -1, 2024-09-18T14:13:05.812 INFO:tasks.workunit.client.0.smithi047.stdout: "item_name": "default" 2024-09-18T14:13:05.812 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.812 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.812 INFO:tasks.workunit.client.0.smithi047.stdout: "op": "choose_indep", 2024-09-18T14:13:05.812 INFO:tasks.workunit.client.0.smithi047.stdout: "num": 0, 2024-09-18T14:13:05.812 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "osd" 2024-09-18T14:13:05.812 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.812 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.812 INFO:tasks.workunit.client.0.smithi047.stdout: "op": "emit" 2024-09-18T14:13:05.812 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.812 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:05.812 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.812 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:05.812 INFO:tasks.workunit.client.0.smithi047.stdout: "tunables": { 2024-09-18T14:13:05.812 INFO:tasks.workunit.client.0.smithi047.stdout: "choose_local_tries": 0, 2024-09-18T14:13:05.813 INFO:tasks.workunit.client.0.smithi047.stdout: "choose_local_fallback_tries": 0, 2024-09-18T14:13:05.813 INFO:tasks.workunit.client.0.smithi047.stdout: "choose_total_tries": 50, 2024-09-18T14:13:05.813 INFO:tasks.workunit.client.0.smithi047.stdout: "chooseleaf_descend_once": 1, 2024-09-18T14:13:05.813 INFO:tasks.workunit.client.0.smithi047.stdout: "chooseleaf_vary_r": 1, 2024-09-18T14:13:05.813 INFO:tasks.workunit.client.0.smithi047.stdout: "chooseleaf_stable": 1, 2024-09-18T14:13:05.813 INFO:tasks.workunit.client.0.smithi047.stdout: "straw_calc_version": 1, 2024-09-18T14:13:05.813 INFO:tasks.workunit.client.0.smithi047.stdout: "allowed_bucket_algs": 54, 2024-09-18T14:13:05.813 INFO:tasks.workunit.client.0.smithi047.stdout: "profile": "jewel", 2024-09-18T14:13:05.813 INFO:tasks.workunit.client.0.smithi047.stdout: "optimal_tunables": 1, 2024-09-18T14:13:05.813 INFO:tasks.workunit.client.0.smithi047.stdout: "legacy_tunables": 0, 2024-09-18T14:13:05.813 INFO:tasks.workunit.client.0.smithi047.stdout: "minimum_required_version": "jewel", 2024-09-18T14:13:05.814 INFO:tasks.workunit.client.0.smithi047.stderr:report 622140245 2024-09-18T14:13:05.814 INFO:tasks.workunit.client.0.smithi047.stdout: "require_feature_tunables": 1, 2024-09-18T14:13:05.814 INFO:tasks.workunit.client.0.smithi047.stdout: "require_feature_tunables2": 1, 2024-09-18T14:13:05.814 INFO:tasks.workunit.client.0.smithi047.stdout: "has_v2_rules": 1, 2024-09-18T14:13:05.814 INFO:tasks.workunit.client.0.smithi047.stdout: "require_feature_tunables3": 1, 2024-09-18T14:13:05.814 INFO:tasks.workunit.client.0.smithi047.stdout: "has_v3_rules": 0, 2024-09-18T14:13:05.814 INFO:tasks.workunit.client.0.smithi047.stdout: "has_v4_buckets": 1, 2024-09-18T14:13:05.814 INFO:tasks.workunit.client.0.smithi047.stdout: "require_feature_tunables5": 1, 2024-09-18T14:13:05.814 INFO:tasks.workunit.client.0.smithi047.stdout: "has_v5_rules": 0 2024-09-18T14:13:05.814 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.815 INFO:tasks.workunit.client.0.smithi047.stdout: "choose_args": {} 2024-09-18T14:13:05.815 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.815 INFO:tasks.workunit.client.0.smithi047.stdout: "fsmap": { 2024-09-18T14:13:05.815 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 1, 2024-09-18T14:13:05.815 INFO:tasks.workunit.client.0.smithi047.stdout: "default_fscid": -1, 2024-09-18T14:13:05.815 INFO:tasks.workunit.client.0.smithi047.stdout: "compat": { 2024-09-18T14:13:05.815 INFO:tasks.workunit.client.0.smithi047.stdout: "compat": {}, 2024-09-18T14:13:05.815 INFO:tasks.workunit.client.0.smithi047.stdout: "ro_compat": {}, 2024-09-18T14:13:05.815 INFO:tasks.workunit.client.0.smithi047.stdout: "incompat": { 2024-09-18T14:13:05.815 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_1": "base v0.20", 2024-09-18T14:13:05.815 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_2": "client writeable ranges", 2024-09-18T14:13:05.815 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_3": "default file layouts on dirs", 2024-09-18T14:13:05.815 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_4": "dir inode in separate object", 2024-09-18T14:13:05.815 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_5": "mds uses versioned encoding", 2024-09-18T14:13:05.815 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-18T14:13:05.816 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_8": "no anchor table", 2024-09-18T14:13:05.816 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_9": "file layout v2", 2024-09-18T14:13:05.816 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_10": "snaprealm v2" 2024-09-18T14:13:05.816 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.816 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.816 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_flags": { 2024-09-18T14:13:05.816 INFO:tasks.workunit.client.0.smithi047.stdout: "enable_multiple": true, 2024-09-18T14:13:05.816 INFO:tasks.workunit.client.0.smithi047.stdout: "ever_enabled_multiple": true 2024-09-18T14:13:05.816 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.816 INFO:tasks.workunit.client.0.smithi047.stdout: "standbys": [], 2024-09-18T14:13:05.816 INFO:tasks.workunit.client.0.smithi047.stdout: "filesystems": [] 2024-09-18T14:13:05.816 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.816 INFO:tasks.workunit.client.0.smithi047.stdout: "mdsmap_first_committed": 1, 2024-09-18T14:13:05.816 INFO:tasks.workunit.client.0.smithi047.stdout: "mdsmap_last_committed": 1, 2024-09-18T14:13:05.816 INFO:tasks.workunit.client.0.smithi047.stdout: "auth": { 2024-09-18T14:13:05.817 INFO:tasks.workunit.client.0.smithi047.stdout: "first_committed": 1, 2024-09-18T14:13:05.817 INFO:tasks.workunit.client.0.smithi047.stdout: "last_committed": 5, 2024-09-18T14:13:05.817 INFO:tasks.workunit.client.0.smithi047.stdout: "num_secrets": 3 2024-09-18T14:13:05.817 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.817 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg": 1, 2024-09-18T14:13:05.817 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg_active": 1, 2024-09-18T14:13:05.817 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg_unknown": 0, 2024-09-18T14:13:05.817 INFO:tasks.workunit.client.0.smithi047.stdout: "num_osd": 3, 2024-09-18T14:13:05.817 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_sum": { 2024-09-18T14:13:05.817 INFO:tasks.workunit.client.0.smithi047.stdout: "stat_sum": { 2024-09-18T14:13:05.817 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes": 48454, 2024-09-18T14:13:05.817 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects": 101, 2024-09-18T14:13:05.817 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_clones": 1, 2024-09-18T14:13:05.817 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_copies": 303, 2024-09-18T14:13:05.817 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing_on_primary": 0, 2024-09-18T14:13:05.818 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing": 0, 2024-09-18T14:13:05.818 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_degraded": 101, 2024-09-18T14:13:05.818 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_misplaced": 0, 2024-09-18T14:13:05.818 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_unfound": 0, 2024-09-18T14:13:05.818 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_dirty": 101, 2024-09-18T14:13:05.818 INFO:tasks.workunit.client.0.smithi047.stdout: "num_whiteouts": 0, 2024-09-18T14:13:05.818 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read": 0, 2024-09-18T14:13:05.818 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read_kb": 0, 2024-09-18T14:13:05.818 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write": 101, 2024-09-18T14:13:05.818 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write_kb": 101, 2024-09-18T14:13:05.818 INFO:tasks.workunit.client.0.smithi047.stdout: "num_scrub_errors": 0, 2024-09-18T14:13:05.819 INFO:tasks.workunit.client.0.smithi047.stdout: "num_shallow_scrub_errors": 0, 2024-09-18T14:13:05.819 INFO:tasks.workunit.client.0.smithi047.stdout: "num_deep_scrub_errors": 0, 2024-09-18T14:13:05.819 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_recovered": 0, 2024-09-18T14:13:05.819 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_recovered": 0, 2024-09-18T14:13:05.819 INFO:tasks.workunit.client.0.smithi047.stdout: "num_keys_recovered": 0, 2024-09-18T14:13:05.819 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_omap": 0, 2024-09-18T14:13:05.819 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_hit_set_archive": 0, 2024-09-18T14:13:05.819 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_hit_set_archive": 0, 2024-09-18T14:13:05.819 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush": 0, 2024-09-18T14:13:05.819 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_kb": 0, 2024-09-18T14:13:05.819 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict": 0, 2024-09-18T14:13:05.819 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_kb": 0, 2024-09-18T14:13:05.819 INFO:tasks.workunit.client.0.smithi047.stdout: "num_promote": 0, 2024-09-18T14:13:05.819 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_high": 0, 2024-09-18T14:13:05.819 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_low": 0, 2024-09-18T14:13:05.820 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_some": 0, 2024-09-18T14:13:05.820 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_full": 0, 2024-09-18T14:13:05.820 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_pinned": 0, 2024-09-18T14:13:05.820 INFO:tasks.workunit.client.0.smithi047.stdout: "num_legacy_snapsets": 0, 2024-09-18T14:13:05.820 INFO:tasks.workunit.client.0.smithi047.stdout: "num_large_omap_objects": 0, 2024-09-18T14:13:05.820 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_manifest": 0, 2024-09-18T14:13:05.820 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_bytes": 0, 2024-09-18T14:13:05.820 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_keys": 0, 2024-09-18T14:13:05.820 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_repaired": 0 2024-09-18T14:13:05.820 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.820 INFO:tasks.workunit.client.0.smithi047.stdout: "store_stats": { 2024-09-18T14:13:05.820 INFO:tasks.workunit.client.0.smithi047.stdout: "total": 0, 2024-09-18T14:13:05.820 INFO:tasks.workunit.client.0.smithi047.stdout: "available": 0, 2024-09-18T14:13:05.820 INFO:tasks.workunit.client.0.smithi047.stdout: "internally_reserved": 0, 2024-09-18T14:13:05.820 INFO:tasks.workunit.client.0.smithi047.stdout: "allocated": 0, 2024-09-18T14:13:05.821 INFO:tasks.workunit.client.0.smithi047.stdout: "data_stored": 0, 2024-09-18T14:13:05.821 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed": 0, 2024-09-18T14:13:05.821 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_allocated": 0, 2024-09-18T14:13:05.821 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_original": 0, 2024-09-18T14:13:05.821 INFO:tasks.workunit.client.0.smithi047.stdout: "omap_allocated": 0, 2024-09-18T14:13:05.821 INFO:tasks.workunit.client.0.smithi047.stdout: "internal_metadata": 0 2024-09-18T14:13:05.821 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.821 INFO:tasks.workunit.client.0.smithi047.stdout: "log_size": 18, 2024-09-18T14:13:05.821 INFO:tasks.workunit.client.0.smithi047.stdout: "ondisk_log_size": 18, 2024-09-18T14:13:05.821 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 3, 2024-09-18T14:13:05.822 INFO:tasks.workunit.client.0.smithi047.stdout: "acting": 3, 2024-09-18T14:13:05.822 INFO:tasks.workunit.client.0.smithi047.stdout: "num_store_stats": 0 2024-09-18T14:13:05.822 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.822 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_sum": { 2024-09-18T14:13:05.822 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 0, 2024-09-18T14:13:05.822 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 0, 2024-09-18T14:13:05.822 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pgs": 3, 2024-09-18T14:13:05.822 INFO:tasks.workunit.client.0.smithi047.stdout: "num_osds": 3, 2024-09-18T14:13:05.822 INFO:tasks.workunit.client.0.smithi047.stdout: "num_per_pool_osds": 3, 2024-09-18T14:13:05.822 INFO:tasks.workunit.client.0.smithi047.stdout: "num_per_pool_omap_osds": 3, 2024-09-18T14:13:05.822 INFO:tasks.workunit.client.0.smithi047.stdout: "kb": 314572800, 2024-09-18T14:13:05.822 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used": 894640, 2024-09-18T14:13:05.822 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_data": 2124, 2024-09-18T14:13:05.822 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_omap": 10, 2024-09-18T14:13:05.822 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_meta": 892469, 2024-09-18T14:13:05.823 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_avail": 313678160, 2024-09-18T14:13:05.823 INFO:tasks.workunit.client.0.smithi047.stdout: "statfs": { 2024-09-18T14:13:05.823 INFO:tasks.workunit.client.0.smithi047.stdout: "total": 322122547200, 2024-09-18T14:13:05.823 INFO:tasks.workunit.client.0.smithi047.stdout: "available": 321206435840, 2024-09-18T14:13:05.823 INFO:tasks.workunit.client.0.smithi047.stdout: "internally_reserved": 0, 2024-09-18T14:13:05.823 INFO:tasks.workunit.client.0.smithi047.stdout: "allocated": 2174976, 2024-09-18T14:13:05.823 INFO:tasks.workunit.client.0.smithi047.stdout: "data_stored": 1528936, 2024-09-18T14:13:05.823 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed": 0, 2024-09-18T14:13:05.823 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_allocated": 0, 2024-09-18T14:13:05.823 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_original": 0, 2024-09-18T14:13:05.823 INFO:tasks.workunit.client.0.smithi047.stdout: "omap_allocated": 10790, 2024-09-18T14:13:05.823 INFO:tasks.workunit.client.0.smithi047.stdout: "internal_metadata": 913888730 2024-09-18T14:13:05.823 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.823 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_peers": [], 2024-09-18T14:13:05.823 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_trim_queue_len": 0, 2024-09-18T14:13:05.824 INFO:tasks.workunit.client.0.smithi047.stdout: "num_snap_trimming": 0, 2024-09-18T14:13:05.824 INFO:tasks.workunit.client.0.smithi047.stdout: "num_shards_repaired": 0, 2024-09-18T14:13:05.824 INFO:tasks.workunit.client.0.smithi047.stdout: "op_queue_age_hist": { 2024-09-18T14:13:05.824 INFO:tasks.workunit.client.0.smithi047.stdout: "histogram": [], 2024-09-18T14:13:05.824 INFO:tasks.workunit.client.0.smithi047.stdout: "upper_bound": 1 2024-09-18T14:13:05.824 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.824 INFO:tasks.workunit.client.0.smithi047.stdout: "perf_stat": { 2024-09-18T14:13:05.824 INFO:tasks.workunit.client.0.smithi047.stdout: "commit_latency_ms": 0, 2024-09-18T14:13:05.824 INFO:tasks.workunit.client.0.smithi047.stdout: "apply_latency_ms": 0, 2024-09-18T14:13:05.824 INFO:tasks.workunit.client.0.smithi047.stdout: "commit_latency_ns": 0, 2024-09-18T14:13:05.824 INFO:tasks.workunit.client.0.smithi047.stdout: "apply_latency_ns": 0 2024-09-18T14:13:05.824 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.824 INFO:tasks.workunit.client.0.smithi047.stdout: "alerts": [], 2024-09-18T14:13:05.824 INFO:tasks.workunit.client.0.smithi047.stdout: "network_ping_times": [] 2024-09-18T14:13:05.824 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.825 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_sum_by_class": { 2024-09-18T14:13:05.825 INFO:tasks.workunit.client.0.smithi047.stdout: "hdd": { 2024-09-18T14:13:05.825 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 0, 2024-09-18T14:13:05.825 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 0, 2024-09-18T14:13:05.825 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pgs": 3, 2024-09-18T14:13:05.825 INFO:tasks.workunit.client.0.smithi047.stdout: "num_osds": 3, 2024-09-18T14:13:05.825 INFO:tasks.workunit.client.0.smithi047.stdout: "num_per_pool_osds": 3, 2024-09-18T14:13:05.825 INFO:tasks.workunit.client.0.smithi047.stdout: "num_per_pool_omap_osds": 3, 2024-09-18T14:13:05.825 INFO:tasks.workunit.client.0.smithi047.stdout: "kb": 314572800, 2024-09-18T14:13:05.825 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used": 894640, 2024-09-18T14:13:05.825 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_data": 2124, 2024-09-18T14:13:05.825 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_omap": 10, 2024-09-18T14:13:05.825 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_meta": 892469, 2024-09-18T14:13:05.825 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_avail": 313678160, 2024-09-18T14:13:05.825 INFO:tasks.workunit.client.0.smithi047.stdout: "statfs": { 2024-09-18T14:13:05.825 INFO:tasks.workunit.client.0.smithi047.stdout: "total": 322122547200, 2024-09-18T14:13:05.826 INFO:tasks.workunit.client.0.smithi047.stdout: "available": 321206435840, 2024-09-18T14:13:05.826 INFO:tasks.workunit.client.0.smithi047.stdout: "internally_reserved": 0, 2024-09-18T14:13:05.826 INFO:tasks.workunit.client.0.smithi047.stdout: "allocated": 2174976, 2024-09-18T14:13:05.826 INFO:tasks.workunit.client.0.smithi047.stdout: "data_stored": 1528936, 2024-09-18T14:13:05.826 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed": 0, 2024-09-18T14:13:05.826 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_allocated": 0, 2024-09-18T14:13:05.826 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_original": 0, 2024-09-18T14:13:05.826 INFO:tasks.workunit.client.0.smithi047.stdout: "omap_allocated": 10790, 2024-09-18T14:13:05.826 INFO:tasks.workunit.client.0.smithi047.stdout: "internal_metadata": 913888730 2024-09-18T14:13:05.826 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.826 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_peers": [], 2024-09-18T14:13:05.826 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_trim_queue_len": 0, 2024-09-18T14:13:05.826 INFO:tasks.workunit.client.0.smithi047.stdout: "num_snap_trimming": 0, 2024-09-18T14:13:05.826 INFO:tasks.workunit.client.0.smithi047.stdout: "num_shards_repaired": 0, 2024-09-18T14:13:05.826 INFO:tasks.workunit.client.0.smithi047.stdout: "op_queue_age_hist": { 2024-09-18T14:13:05.827 INFO:tasks.workunit.client.0.smithi047.stdout: "histogram": [], 2024-09-18T14:13:05.827 INFO:tasks.workunit.client.0.smithi047.stdout: "upper_bound": 1 2024-09-18T14:13:05.827 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.827 INFO:tasks.workunit.client.0.smithi047.stdout: "perf_stat": { 2024-09-18T14:13:05.827 INFO:tasks.workunit.client.0.smithi047.stdout: "commit_latency_ms": 0, 2024-09-18T14:13:05.827 INFO:tasks.workunit.client.0.smithi047.stdout: "apply_latency_ms": 0, 2024-09-18T14:13:05.827 INFO:tasks.workunit.client.0.smithi047.stdout: "commit_latency_ns": 0, 2024-09-18T14:13:05.827 INFO:tasks.workunit.client.0.smithi047.stdout: "apply_latency_ns": 0 2024-09-18T14:13:05.827 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.827 INFO:tasks.workunit.client.0.smithi047.stdout: "alerts": [], 2024-09-18T14:13:05.827 INFO:tasks.workunit.client.0.smithi047.stdout: "network_ping_times": [] 2024-09-18T14:13:05.827 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.827 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.827 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_stats": [ 2024-09-18T14:13:05.827 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.828 INFO:tasks.workunit.client.0.smithi047.stdout: "poolid": 1, 2024-09-18T14:13:05.828 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg": 1, 2024-09-18T14:13:05.828 INFO:tasks.workunit.client.0.smithi047.stdout: "stat_sum": { 2024-09-18T14:13:05.828 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes": 48454, 2024-09-18T14:13:05.828 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects": 101, 2024-09-18T14:13:05.828 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_clones": 1, 2024-09-18T14:13:05.828 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_copies": 303, 2024-09-18T14:13:05.828 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing_on_primary": 0, 2024-09-18T14:13:05.828 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing": 0, 2024-09-18T14:13:05.828 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_degraded": 101, 2024-09-18T14:13:05.828 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_misplaced": 0, 2024-09-18T14:13:05.828 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_unfound": 0, 2024-09-18T14:13:05.828 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_dirty": 101, 2024-09-18T14:13:05.828 INFO:tasks.workunit.client.0.smithi047.stdout: "num_whiteouts": 0, 2024-09-18T14:13:05.828 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read": 0, 2024-09-18T14:13:05.829 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read_kb": 0, 2024-09-18T14:13:05.829 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write": 101, 2024-09-18T14:13:05.829 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write_kb": 101, 2024-09-18T14:13:05.829 INFO:tasks.workunit.client.0.smithi047.stdout: "num_scrub_errors": 0, 2024-09-18T14:13:05.829 INFO:tasks.workunit.client.0.smithi047.stdout: "num_shallow_scrub_errors": 0, 2024-09-18T14:13:05.829 INFO:tasks.workunit.client.0.smithi047.stdout: "num_deep_scrub_errors": 0, 2024-09-18T14:13:05.829 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_recovered": 0, 2024-09-18T14:13:05.829 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_recovered": 0, 2024-09-18T14:13:05.829 INFO:tasks.workunit.client.0.smithi047.stdout: "num_keys_recovered": 0, 2024-09-18T14:13:05.829 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_omap": 0, 2024-09-18T14:13:05.829 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_hit_set_archive": 0, 2024-09-18T14:13:05.829 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_hit_set_archive": 0, 2024-09-18T14:13:05.829 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush": 0, 2024-09-18T14:13:05.829 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_kb": 0, 2024-09-18T14:13:05.829 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict": 0, 2024-09-18T14:13:05.830 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_kb": 0, 2024-09-18T14:13:05.830 INFO:tasks.workunit.client.0.smithi047.stdout: "num_promote": 0, 2024-09-18T14:13:05.830 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_high": 0, 2024-09-18T14:13:05.830 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_low": 0, 2024-09-18T14:13:05.830 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_some": 0, 2024-09-18T14:13:05.830 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_full": 0, 2024-09-18T14:13:05.830 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_pinned": 0, 2024-09-18T14:13:05.830 INFO:tasks.workunit.client.0.smithi047.stdout: "num_legacy_snapsets": 0, 2024-09-18T14:13:05.830 INFO:tasks.workunit.client.0.smithi047.stdout: "num_large_omap_objects": 0, 2024-09-18T14:13:05.830 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_manifest": 0, 2024-09-18T14:13:05.830 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_bytes": 0, 2024-09-18T14:13:05.830 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_keys": 0, 2024-09-18T14:13:05.830 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_repaired": 0 2024-09-18T14:13:05.830 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.830 INFO:tasks.workunit.client.0.smithi047.stdout: "store_stats": { 2024-09-18T14:13:05.830 INFO:tasks.workunit.client.0.smithi047.stdout: "total": 0, 2024-09-18T14:13:05.831 INFO:tasks.workunit.client.0.smithi047.stdout: "available": 0, 2024-09-18T14:13:05.831 INFO:tasks.workunit.client.0.smithi047.stdout: "internally_reserved": 0, 2024-09-18T14:13:05.831 INFO:tasks.workunit.client.0.smithi047.stdout: "allocated": 1241088, 2024-09-18T14:13:05.831 INFO:tasks.workunit.client.0.smithi047.stdout: "data_stored": 1241088, 2024-09-18T14:13:05.831 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed": 0, 2024-09-18T14:13:05.831 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_allocated": 0, 2024-09-18T14:13:05.831 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_original": 0, 2024-09-18T14:13:05.831 INFO:tasks.workunit.client.0.smithi047.stdout: "omap_allocated": 0, 2024-09-18T14:13:05.831 INFO:tasks.workunit.client.0.smithi047.stdout: "internal_metadata": 0 2024-09-18T14:13:05.831 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.831 INFO:tasks.workunit.client.0.smithi047.stdout: "log_size": 18, 2024-09-18T14:13:05.831 INFO:tasks.workunit.client.0.smithi047.stdout: "ondisk_log_size": 18, 2024-09-18T14:13:05.831 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 3, 2024-09-18T14:13:05.831 INFO:tasks.workunit.client.0.smithi047.stdout: "acting": 3, 2024-09-18T14:13:05.831 INFO:tasks.workunit.client.0.smithi047.stdout: "num_store_stats": 3 2024-09-18T14:13:05.832 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.832 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:05.832 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_stats": [ 2024-09-18T14:13:05.832 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.832 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 0, 2024-09-18T14:13:05.832 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 154618822668 2024-09-18T14:13:05.832 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.832 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.832 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 1, 2024-09-18T14:13:05.832 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 42949672988 2024-09-18T14:13:05.832 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.832 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.832 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 2, 2024-09-18T14:13:05.832 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 163208757259 2024-09-18T14:13:05.832 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.833 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:05.833 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg_by_state": [ 2024-09-18T14:13:05.833 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.833 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "active+undersized+degraded", 2024-09-18T14:13:05.833 INFO:tasks.workunit.client.0.smithi047.stdout: "num": 1 2024-09-18T14:13:05.833 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.833 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:05.833 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg_by_osd": [ 2024-09-18T14:13:05.833 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.833 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 2147483647, 2024-09-18T14:13:05.833 INFO:tasks.workunit.client.0.smithi047.stdout: "num_primary_pg": 0, 2024-09-18T14:13:05.833 INFO:tasks.workunit.client.0.smithi047.stdout: "num_acting_pg": 1, 2024-09-18T14:13:05.833 INFO:tasks.workunit.client.0.smithi047.stdout: "num_up_not_acting_pg": 0 2024-09-18T14:13:05.833 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.833 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.833 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 2, 2024-09-18T14:13:05.834 INFO:tasks.workunit.client.0.smithi047.stdout: "num_primary_pg": 0, 2024-09-18T14:13:05.834 INFO:tasks.workunit.client.0.smithi047.stdout: "num_acting_pg": 1, 2024-09-18T14:13:05.834 INFO:tasks.workunit.client.0.smithi047.stdout: "num_up_not_acting_pg": 0 2024-09-18T14:13:05.834 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.834 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.834 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 0, 2024-09-18T14:13:05.834 INFO:tasks.workunit.client.0.smithi047.stdout: "num_primary_pg": 1, 2024-09-18T14:13:05.834 INFO:tasks.workunit.client.0.smithi047.stdout: "num_acting_pg": 1, 2024-09-18T14:13:05.834 INFO:tasks.workunit.client.0.smithi047.stdout: "num_up_not_acting_pg": 0 2024-09-18T14:13:05.834 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.834 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.834 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 1, 2024-09-18T14:13:05.834 INFO:tasks.workunit.client.0.smithi047.stdout: "num_primary_pg": 0, 2024-09-18T14:13:05.834 INFO:tasks.workunit.client.0.smithi047.stdout: "num_acting_pg": 0, 2024-09-18T14:13:05.834 INFO:tasks.workunit.client.0.smithi047.stdout: "num_up_not_acting_pg": 0 2024-09-18T14:13:05.835 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.835 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:05.835 INFO:tasks.workunit.client.0.smithi047.stdout: "purged_snaps": [ 2024-09-18T14:13:05.835 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:05.835 INFO:tasks.workunit.client.0.smithi047.stdout: "pool": 1, 2024-09-18T14:13:05.835 INFO:tasks.workunit.client.0.smithi047.stdout: "purged_snaps": {} 2024-09-18T14:13:05.835 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.835 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:05.835 INFO:tasks.workunit.client.0.smithi047.stdout: "servicemap": { 2024-09-18T14:13:05.835 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 5, 2024-09-18T14:13:05.835 INFO:tasks.workunit.client.0.smithi047.stdout: "modified": "2024-09-18T14:12:12.047447+0000", 2024-09-18T14:13:05.835 INFO:tasks.workunit.client.0.smithi047.stdout: "services": {} 2024-09-18T14:13:05.835 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:05.835 INFO:tasks.workunit.client.0.smithi047.stdout: "paxos": { 2024-09-18T14:13:05.835 INFO:tasks.workunit.client.0.smithi047.stdout: "first_committed": 1, 2024-09-18T14:13:05.835 INFO:tasks.workunit.client.0.smithi047.stdout: "last_committed": 539, 2024-09-18T14:13:05.836 INFO:tasks.workunit.client.0.smithi047.stdout: "last_pn": 0, 2024-09-18T14:13:05.836 INFO:tasks.workunit.client.0.smithi047.stdout: "accepted_pn": 0 2024-09-18T14:13:05.836 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:05.836 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T14:13:05.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-18T14:13:05.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2024-09-18T14:13:05.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.0 2024-09-18T14:13:05.837 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:13:05.837 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:13:05.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:13:05.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:13:05.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:13:05.862 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:13:05.862 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 0 --op log --pgid 1.0 2024-09-18T14:13:05.862 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-18T14:13:05.862 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T14:13:05.862 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-18T14:13:05.862 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T14:13:05.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2024-09-18T14:13:05.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op log --pgid 1.0 2024-09-18T14:13:07.288 INFO:tasks.workunit.client.0.smithi047.stderr:PG '1.0' not found 2024-09-18T14:13:07.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-09-18T14:13:07.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 0 2024-09-18T14:13:07.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-18T14:13:07.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:13:07.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T14:13:07.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:13:07.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-09-18T14:13:07.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T14:13:07.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:13:07.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:13:07.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:13:07.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-18T14:13:07.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-18T14:13:07.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:13:07.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:13:07.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-18T14:13:07.812 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:13:07.813 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:13:07.813 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:13:07.813 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:13:07.813 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:13:07.813 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:13:07.813 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:13:07.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:13:07.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:13:07.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-18T14:13:07.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-18T14:13:07.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:13:07.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:13:07.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:13:07.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:13:07.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:13:07.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:13:07.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-09-18T14:13:07.815 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T14:13:07.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T14:13:07.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:13:07.816 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-09-18T14:13:07.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T14:13:07.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:13:07.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:13:07.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:13:07.844 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:13:07.843+0000 7f1549422440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:13:07.846 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:13:07.846+0000 7f1549422440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:13:07.848 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:13:07.847+0000 7f1549422440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:13:08.197 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T14:13:08.197 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:13:08.197 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:13:08.197 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:13:08.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:13:08.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:13:08.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:13:08.198 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:13:08.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:13:08.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:13:08.359 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:13:08.359+0000 7f1549422440 -1 Falling back to public interface 2024-09-18T14:13:08.570 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 36 up_thru 40 down_at 30 last_clean_interval [5,28) [v2:127.0.0.1:6802/83233,v1:127.0.0.1:6803/83233] [v2:127.0.0.1:6804/83233,v1:127.0.0.1:6805/83233] exists,up 50835fc1-8964-4c12-b447-46f109dbb8e4 2024-09-18T14:13:08.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:13:08.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:13:08.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:13:08.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2024-09-18T14:13:08.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.2 2024-09-18T14:13:08.572 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:13:08.572 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:13:08.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:13:08.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:13:08.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:13:08.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:13:08.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 2 --op log --pgid 1.0 2024-09-18T14:13:08.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-18T14:13:08.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T14:13:08.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-09-18T14:13:08.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T14:13:08.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/2 2024-09-18T14:13:08.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/2 --op log --pgid 1.0 2024-09-18T14:13:10.019 INFO:tasks.workunit.client.0.smithi047.stderr:PG '1.0' not found 2024-09-18T14:13:10.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-09-18T14:13:10.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 2 2024-09-18T14:13:10.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-18T14:13:10.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:13:10.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T14:13:10.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:13:10.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-09-18T14:13:10.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T14:13:10.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:13:10.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:13:10.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:13:10.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-18T14:13:10.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-18T14:13:10.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:13:10.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:13:10.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-18T14:13:10.540 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:13:10.540 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:13:10.540 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:13:10.540 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:13:10.540 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:13:10.540 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:13:10.540 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:13:10.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:13:10.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:13:10.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-18T14:13:10.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-18T14:13:10.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:13:10.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:13:10.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:13:10.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:13:10.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:13:10.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:13:10.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-09-18T14:13:10.541 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T14:13:10.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T14:13:10.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:13:10.542 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-09-18T14:13:10.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T14:13:10.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:13:10.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:13:10.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:13:10.569 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:13:10.569+0000 7f3cad209440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:13:10.629 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:13:10.629+0000 7f3cad209440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:13:10.630 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:13:10.630+0000 7f3cad209440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:13:10.881 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:13:10.879+0000 7f1549422440 -1 osd.0 41 log_to_monitors true 2024-09-18T14:13:10.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T14:13:10.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:13:10.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T14:13:10.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:13:10.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:13:10.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:13:10.923 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:13:10.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:13:10.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:13:10.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:13:11.144 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:13:11.144+0000 7f3cad209440 -1 Falling back to public interface 2024-09-18T14:13:11.329 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,28) [v2:127.0.0.1:6810/83811,v1:127.0.0.1:6811/83811] [v2:127.0.0.1:6812/83811,v1:127.0.0.1:6813/83811] exists,up 2d53c532-9e0f-4778-a2d9-277af85698d4 2024-09-18T14:13:11.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:13:11.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:13:11.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:13:11.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:336: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 1 --op log --pgid 1.0 2024-09-18T14:13:11.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-18T14:13:11.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T14:13:11.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T14:13:11.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T14:13:11.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-18T14:13:11.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op log --pgid 1.0 2024-09-18T14:13:12.948 INFO:tasks.workunit.client.0.smithi047.stderr:PG '1.0' not found 2024-09-18T14:13:13.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-09-18T14:13:13.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-18T14:13:13.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: wait_for_clean 2024-09-18T14:13:13.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:13:13.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:13:13.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:13:13.220 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-18T14:13:13.220 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:13:13.221 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:13:13.221 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:13:13.221 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:13:13.221 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:13:13.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-18T14:13:13.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:13:13.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:13:13.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:13:13.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:13:13.277 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:13:13.618 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:13:13.617+0000 7f3cad209440 -1 osd.2 41 log_to_monitors true 2024-09-18T14:13:13.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:13:13.656 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T14:13:13.656 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T14:13:13.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:13:13.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:13:13.657 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:13:13.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=188978561027 2024-09-18T14:13:13.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 188978561027 2024-09-18T14:13:13.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-188978561027' 2024-09-18T14:13:13.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:13:13.786 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:13:13.873 INFO:tasks.workunit.client.0.smithi047.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-18T14:13:13.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq= 2024-09-18T14:13:13.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z '' 2024-09-18T14:13:13.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: flush_pg_stats: continue 2024-09-18T14:13:13.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:13:13.875 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:13:18.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=206158430212 2024-09-18T14:13:18.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 206158430212 2024-09-18T14:13:18.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-188978561027 2-206158430212' 2024-09-18T14:13:18.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:13:18.997 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-188978561027 2024-09-18T14:13:18.997 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:13:18.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:13:18.999 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-188978561027 2024-09-18T14:13:18.999 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:13:19.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=188978561027 2024-09-18T14:13:19.000 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 188978561027 2024-09-18T14:13:19.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 188978561027' 2024-09-18T14:13:19.000 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:13:19.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 188978561028 -lt 188978561027 2024-09-18T14:13:19.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:13:19.422 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-206158430212 2024-09-18T14:13:19.422 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:13:19.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:13:19.423 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-206158430212 2024-09-18T14:13:19.423 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:13:19.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=206158430212 2024-09-18T14:13:19.425 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 206158430212 2024-09-18T14:13:19.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 206158430212' 2024-09-18T14:13:19.425 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:13:19.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 206158430212 2024-09-18T14:13:19.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:13:20.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:13:20.841 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:13:21.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 206158430212 -lt 206158430212 2024-09-18T14:13:21.268 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:13:21.268 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:13:21.268 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:13:21.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T14:13:21.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:13:21.809 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:13:21.809 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:13:21.809 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:13:21.809 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:13:21.809 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:13:21.809 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:13:22.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:13:22.184 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:13:22.185 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:13:22.185 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:13:22.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:13:22.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-18T14:13:22.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T14:13:22.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-18T14:13:22.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:13:22.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:13:22.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:13:22.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:13:22.833 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:13:22.833 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:13:22.833 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:13:22.833 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:13:22.833 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:13:22.833 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:13:23.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:13:23.205 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:13:23.205 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:13:23.205 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:13:23.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:13:23.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:13:23.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:13:23.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:13:23.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:13:23.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:13:23.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:13:23.710 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:13:23.710 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:13:24.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:13:24.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:13:24.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-09-18T14:13:24.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:13:24.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-18T14:13:24.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:13:24.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:13:24.415 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:13:24.415 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:13:24.415 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:13:24.415 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:13:24.415 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:13:24.415 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:13:24.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:13:24.782 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:13:24.782 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:13:24.783 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:13:25.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:13:25.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:13:25.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:13:25.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:13:25.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:13:25.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:13:25.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:13:25.283 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:13:25.283 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:13:25.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:13:25.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:13:25.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-09-18T14:13:25.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:13:25.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-18T14:13:26.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:13:26.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:13:26.203 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:13:26.203 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:13:26.203 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:13:26.203 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:13:26.203 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:13:26.203 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:13:26.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:13:26.576 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:13:26.576 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:13:26.576 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:13:27.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:13:27.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:13:27.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:13:27.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:13:27.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:13:27.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:13:27.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:13:27.078 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:13:27.078 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:13:27.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:13:27.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:13:27.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-09-18T14:13:27.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:13:27.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-09-18T14:13:28.406 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:13:28.406 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:13:28.407 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:13:28.407 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:13:28.407 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:13:28.407 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:13:28.408 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:13:28.408 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:13:28.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:13:28.784 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:13:28.784 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:13:28.784 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:13:29.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:13:29.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:13:29.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:13:29.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:13:29.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:13:29.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:13:29.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:13:29.292 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:13:29.292 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:13:29.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:13:29.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:13:29.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-09-18T14:13:29.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:13:29.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-09-18T14:13:31.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:13:31.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:13:31.437 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:13:31.437 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:13:31.437 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:13:31.437 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:13:31.437 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:13:31.438 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:13:31.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:13:31.815 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:13:31.816 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:13:31.816 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:13:32.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:13:32.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:13:32.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:13:32.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:13:32.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:13:32.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:13:32.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:13:32.318 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:13:32.318 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:13:32.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:13:32.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:13:32.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-09-18T14:13:32.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:13:32.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-09-18T14:13:36.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:13:36.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:13:36.060 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:13:36.060 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:13:36.060 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:13:36.060 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:13:36.060 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:13:36.061 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:13:36.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:13:36.437 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:13:36.437 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:13:36.437 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:13:36.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:13:36.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:13:36.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:13:36.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:13:36.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:13:36.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:13:36.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:13:36.982 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:13:36.982 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:13:37.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:13:37.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:13:37.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-09-18T14:13:37.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:13:37.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-09-18T14:13:43.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:13:43.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:13:43.911 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:13:43.912 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:13:43.912 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:13:43.912 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:13:43.912 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:13:43.912 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:13:44.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:13:44.298 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:13:44.299 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:13:44.299 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:13:44.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:13:44.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:13:44.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:13:44.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:13:44.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:13:44.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:13:44.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:13:44.806 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:13:44.806 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:13:45.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:13:45.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:13:45.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-09-18T14:13:45.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:13:45.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-09-18T14:13:52.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:13:52.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:13:52.641 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:13:52.641 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:13:52.641 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:13:52.641 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:13:52.642 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:13:52.642 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:13:53.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:13:53.015 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:13:53.016 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:13:53.016 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:13:53.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T14:13:53.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:13:53.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:13:53.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:13:53.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:13:53.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:13:53.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:13:53.523 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:13:53.523 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:13:54.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:13:54.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:13:54.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-09-18T14:13:54.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-18T14:13:54.590 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T14:13:54.590 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_fingerprint": "182cb370-8208-468f-b53f-d6c2f80a20b3", 2024-09-18T14:13:54.590 INFO:tasks.workunit.client.0.smithi047.stdout: "version": "17.2.7-1645-g7061ea6c", 2024-09-18T14:13:54.591 INFO:tasks.workunit.client.0.smithi047.stdout: "commit": "7061ea6ca50908d104e9f3979dc615e118d9e153", 2024-09-18T14:13:54.591 INFO:tasks.workunit.client.0.smithi047.stdout: "timestamp": "2024-09-18T14:13:54.588869+0000", 2024-09-18T14:13:54.591 INFO:tasks.workunit.client.0.smithi047.stdout: "tag": "", 2024-09-18T14:13:54.591 INFO:tasks.workunit.client.0.smithi047.stdout: "health": { 2024-09-18T14:13:54.591 INFO:tasks.workunit.client.0.smithi047.stdout: "status": "HEALTH_WARN", 2024-09-18T14:13:54.591 INFO:tasks.workunit.client.0.smithi047.stdout: "checks": { 2024-09-18T14:13:54.592 INFO:tasks.workunit.client.0.smithi047.stdout: "OSDMAP_FLAGS": { 2024-09-18T14:13:54.592 INFO:tasks.workunit.client.0.smithi047.stdout: "severity": "HEALTH_WARN", 2024-09-18T14:13:54.592 INFO:tasks.workunit.client.0.smithi047.stdout: "summary": { 2024-09-18T14:13:54.592 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-18T14:13:54.592 INFO:tasks.workunit.client.0.smithi047.stdout: "count": 17 2024-09-18T14:13:54.593 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.593 INFO:tasks.workunit.client.0.smithi047.stdout: "detail": [], 2024-09-18T14:13:54.593 INFO:tasks.workunit.client.0.smithi047.stdout: "muted": false 2024-09-18T14:13:54.593 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.593 INFO:tasks.workunit.client.0.smithi047.stdout: "OSD_DOWN": { 2024-09-18T14:13:54.594 INFO:tasks.workunit.client.0.smithi047.stdout: "severity": "HEALTH_WARN", 2024-09-18T14:13:54.594 INFO:tasks.workunit.client.0.smithi047.stdout: "summary": { 2024-09-18T14:13:54.594 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "1 osds down", 2024-09-18T14:13:54.594 INFO:tasks.workunit.client.0.smithi047.stdout: "count": 1 2024-09-18T14:13:54.594 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.594 INFO:tasks.workunit.client.0.smithi047.stdout: "detail": [ 2024-09-18T14:13:54.594 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.595 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "osd.1 (root=default,host=smithi047) is down" 2024-09-18T14:13:54.595 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.595 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:54.595 INFO:tasks.workunit.client.0.smithi047.stdout: "muted": false 2024-09-18T14:13:54.595 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.595 INFO:tasks.workunit.client.0.smithi047.stdout: "PG_DEGRADED": { 2024-09-18T14:13:54.595 INFO:tasks.workunit.client.0.smithi047.stdout: "severity": "HEALTH_WARN", 2024-09-18T14:13:54.595 INFO:tasks.workunit.client.0.smithi047.stdout: "summary": { 2024-09-18T14:13:54.595 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2024-09-18T14:13:54.595 INFO:tasks.workunit.client.0.smithi047.stdout: "count": 1 2024-09-18T14:13:54.595 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.595 INFO:tasks.workunit.client.0.smithi047.stdout: "detail": [ 2024-09-18T14:13:54.595 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.596 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2024-09-18T14:13:54.596 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.596 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:54.596 INFO:tasks.workunit.client.0.smithi047.stdout: "muted": false 2024-09-18T14:13:54.596 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.596 INFO:tasks.workunit.client.0.smithi047.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-18T14:13:54.596 INFO:tasks.workunit.client.0.smithi047.stdout: "severity": "HEALTH_WARN", 2024-09-18T14:13:54.596 INFO:tasks.workunit.client.0.smithi047.stdout: "summary": { 2024-09-18T14:13:54.596 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-18T14:13:54.596 INFO:tasks.workunit.client.0.smithi047.stdout: "count": 1 2024-09-18T14:13:54.596 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.596 INFO:tasks.workunit.client.0.smithi047.stdout: "detail": [ 2024-09-18T14:13:54.597 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.597 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "application not enabled on pool 'test'" 2024-09-18T14:13:54.597 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.597 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.597 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-18T14:13:54.597 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.597 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:54.597 INFO:tasks.workunit.client.0.smithi047.stdout: "muted": false 2024-09-18T14:13:54.597 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.597 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.597 INFO:tasks.workunit.client.0.smithi047.stdout: "mutes": [] 2024-09-18T14:13:54.597 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.597 INFO:tasks.workunit.client.0.smithi047.stdout: "monmap_first_committed": 1, 2024-09-18T14:13:54.597 INFO:tasks.workunit.client.0.smithi047.stdout: "monmap_last_committed": 1, 2024-09-18T14:13:54.598 INFO:tasks.workunit.client.0.smithi047.stdout: "monmap": { 2024-09-18T14:13:54.598 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 1, 2024-09-18T14:13:54.598 INFO:tasks.workunit.client.0.smithi047.stdout: "fsid": "d7330bd7-8062-424a-bbd8-8c587903c456", 2024-09-18T14:13:54.598 INFO:tasks.workunit.client.0.smithi047.stdout: "modified": "2024-09-18T14:09:28.759690Z", 2024-09-18T14:13:54.598 INFO:tasks.workunit.client.0.smithi047.stdout: "created": "2024-09-18T14:09:28.759690Z", 2024-09-18T14:13:54.598 INFO:tasks.workunit.client.0.smithi047.stdout: "min_mon_release": 17, 2024-09-18T14:13:54.598 INFO:tasks.workunit.client.0.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-18T14:13:54.598 INFO:tasks.workunit.client.0.smithi047.stdout: "election_strategy": 1, 2024-09-18T14:13:54.598 INFO:tasks.workunit.client.0.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-18T14:13:54.598 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_mode": false, 2024-09-18T14:13:54.598 INFO:tasks.workunit.client.0.smithi047.stdout: "tiebreaker_mon": "", 2024-09-18T14:13:54.598 INFO:tasks.workunit.client.0.smithi047.stdout: "removed_ranks: ": "", 2024-09-18T14:13:54.598 INFO:tasks.workunit.client.0.smithi047.stdout: "features": { 2024-09-18T14:13:54.598 INFO:tasks.workunit.client.0.smithi047.stdout: "persistent": [ 2024-09-18T14:13:54.599 INFO:tasks.workunit.client.0.smithi047.stdout: "kraken", 2024-09-18T14:13:54.599 INFO:tasks.workunit.client.0.smithi047.stdout: "luminous", 2024-09-18T14:13:54.599 INFO:tasks.workunit.client.0.smithi047.stdout: "mimic", 2024-09-18T14:13:54.599 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap-prune", 2024-09-18T14:13:54.599 INFO:tasks.workunit.client.0.smithi047.stdout: "nautilus", 2024-09-18T14:13:54.599 INFO:tasks.workunit.client.0.smithi047.stdout: "octopus", 2024-09-18T14:13:54.599 INFO:tasks.workunit.client.0.smithi047.stdout: "pacific", 2024-09-18T14:13:54.599 INFO:tasks.workunit.client.0.smithi047.stdout: "elector-pinging", 2024-09-18T14:13:54.599 INFO:tasks.workunit.client.0.smithi047.stdout: "quincy" 2024-09-18T14:13:54.599 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:54.599 INFO:tasks.workunit.client.0.smithi047.stdout: "optional": [] 2024-09-18T14:13:54.599 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.599 INFO:tasks.workunit.client.0.smithi047.stdout: "mons": [ 2024-09-18T14:13:54.600 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.600 INFO:tasks.workunit.client.0.smithi047.stdout: "rank": 0, 2024-09-18T14:13:54.600 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "a", 2024-09-18T14:13:54.600 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addrs": { 2024-09-18T14:13:54.600 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:13:54.600 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.600 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:13:54.600 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:7115", 2024-09-18T14:13:54.600 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 0 2024-09-18T14:13:54.600 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.600 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:54.600 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.600 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:7115/0", 2024-09-18T14:13:54.601 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-18T14:13:54.601 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 0, 2024-09-18T14:13:54.601 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 0, 2024-09-18T14:13:54.601 INFO:tasks.workunit.client.0.smithi047.stdout: "crush_location": "{}" 2024-09-18T14:13:54.601 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.601 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:54.601 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.601 INFO:tasks.workunit.client.0.smithi047.stdout: "quorum": [ 2024-09-18T14:13:54.601 INFO:tasks.workunit.client.0.smithi047.stdout: 0 2024-09-18T14:13:54.601 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:54.601 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap": { 2024-09-18T14:13:54.601 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 49, 2024-09-18T14:13:54.601 INFO:tasks.workunit.client.0.smithi047.stdout: "fsid": "d7330bd7-8062-424a-bbd8-8c587903c456", 2024-09-18T14:13:54.601 INFO:tasks.workunit.client.0.smithi047.stdout: "created": "2024-09-18T14:09:29.543271+0000", 2024-09-18T14:13:54.602 INFO:tasks.workunit.client.0.smithi047.stdout: "modified": "2024-09-18T14:13:14.496955+0000", 2024-09-18T14:13:54.602 INFO:tasks.workunit.client.0.smithi047.stdout: "last_up_change": "2024-09-18T14:13:14.077659+0000", 2024-09-18T14:13:54.602 INFO:tasks.workunit.client.0.smithi047.stdout: "last_in_change": "2024-09-18T14:12:23.898951+0000", 2024-09-18T14:13:54.602 INFO:tasks.workunit.client.0.smithi047.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-18T14:13:54.602 INFO:tasks.workunit.client.0.smithi047.stdout: "flags_num": 5800384, 2024-09-18T14:13:54.602 INFO:tasks.workunit.client.0.smithi047.stdout: "flags_set": [ 2024-09-18T14:13:54.602 INFO:tasks.workunit.client.0.smithi047.stdout: "nodown", 2024-09-18T14:13:54.602 INFO:tasks.workunit.client.0.smithi047.stdout: "noin", 2024-09-18T14:13:54.602 INFO:tasks.workunit.client.0.smithi047.stdout: "noout", 2024-09-18T14:13:54.602 INFO:tasks.workunit.client.0.smithi047.stdout: "pglog_hardlimit", 2024-09-18T14:13:54.602 INFO:tasks.workunit.client.0.smithi047.stdout: "purged_snapdirs", 2024-09-18T14:13:54.602 INFO:tasks.workunit.client.0.smithi047.stdout: "recovery_deletes", 2024-09-18T14:13:54.602 INFO:tasks.workunit.client.0.smithi047.stdout: "sortbitwise" 2024-09-18T14:13:54.602 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:54.603 INFO:tasks.workunit.client.0.smithi047.stdout: "crush_version": 12, 2024-09-18T14:13:54.603 INFO:tasks.workunit.client.0.smithi047.stdout: "full_ratio": 0.99000000953674316, 2024-09-18T14:13:54.603 INFO:tasks.workunit.client.0.smithi047.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-18T14:13:54.603 INFO:tasks.workunit.client.0.smithi047.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-18T14:13:54.603 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_snapshot": "", 2024-09-18T14:13:54.603 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_max": 1, 2024-09-18T14:13:54.603 INFO:tasks.workunit.client.0.smithi047.stdout: "max_osd": 3, 2024-09-18T14:13:54.603 INFO:tasks.workunit.client.0.smithi047.stdout: "require_min_compat_client": "luminous", 2024-09-18T14:13:54.603 INFO:tasks.workunit.client.0.smithi047.stdout: "min_compat_client": "jewel", 2024-09-18T14:13:54.603 INFO:tasks.workunit.client.0.smithi047.stdout: "require_osd_release": "quincy", 2024-09-18T14:13:54.603 INFO:tasks.workunit.client.0.smithi047.stdout: "pools": [ 2024-09-18T14:13:54.603 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.603 INFO:tasks.workunit.client.0.smithi047.stdout: "pool": 1, 2024-09-18T14:13:54.603 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_name": "test", 2024-09-18T14:13:54.604 INFO:tasks.workunit.client.0.smithi047.stdout: "create_time": "2024-09-18T14:11:14.265067+0000", 2024-09-18T14:13:54.604 INFO:tasks.workunit.client.0.smithi047.stdout: "flags": 16389, 2024-09-18T14:13:54.604 INFO:tasks.workunit.client.0.smithi047.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2024-09-18T14:13:54.604 INFO:tasks.workunit.client.0.smithi047.stdout: "type": 3, 2024-09-18T14:13:54.604 INFO:tasks.workunit.client.0.smithi047.stdout: "size": 3, 2024-09-18T14:13:54.604 INFO:tasks.workunit.client.0.smithi047.stdout: "min_size": 2, 2024-09-18T14:13:54.604 INFO:tasks.workunit.client.0.smithi047.stdout: "crush_rule": 1, 2024-09-18T14:13:54.604 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_count": 0, 2024-09-18T14:13:54.604 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_target": 0, 2024-09-18T14:13:54.604 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T14:13:54.604 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T14:13:54.604 INFO:tasks.workunit.client.0.smithi047.stdout: "object_hash": 2, 2024-09-18T14:13:54.604 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_autoscale_mode": "off", 2024-09-18T14:13:54.605 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_num": 1, 2024-09-18T14:13:54.605 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_placement_num": 1, 2024-09-18T14:13:54.605 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_placement_num_target": 1, 2024-09-18T14:13:54.605 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_num_target": 1, 2024-09-18T14:13:54.605 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_num_pending": 1, 2024-09-18T14:13:54.605 INFO:tasks.workunit.client.0.smithi047.stdout: "last_pg_merge_meta": { 2024-09-18T14:13:54.605 INFO:tasks.workunit.client.0.smithi047.stdout: "source_pgid": "0.0", 2024-09-18T14:13:54.605 INFO:tasks.workunit.client.0.smithi047.stdout: "ready_epoch": 0, 2024-09-18T14:13:54.605 INFO:tasks.workunit.client.0.smithi047.stdout: "last_epoch_started": 0, 2024-09-18T14:13:54.605 INFO:tasks.workunit.client.0.smithi047.stdout: "last_epoch_clean": 0, 2024-09-18T14:13:54.605 INFO:tasks.workunit.client.0.smithi047.stdout: "source_version": "0'0", 2024-09-18T14:13:54.605 INFO:tasks.workunit.client.0.smithi047.stdout: "target_version": "0'0" 2024-09-18T14:13:54.605 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.606 INFO:tasks.workunit.client.0.smithi047.stdout: "last_change": "29", 2024-09-18T14:13:54.606 INFO:tasks.workunit.client.0.smithi047.stdout: "last_force_op_resend": "0", 2024-09-18T14:13:54.606 INFO:tasks.workunit.client.0.smithi047.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T14:13:54.606 INFO:tasks.workunit.client.0.smithi047.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T14:13:54.606 INFO:tasks.workunit.client.0.smithi047.stdout: "auid": 0, 2024-09-18T14:13:54.606 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_mode": "pool", 2024-09-18T14:13:54.606 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_seq": 1, 2024-09-18T14:13:54.606 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_epoch": 29, 2024-09-18T14:13:54.606 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_snaps": [ 2024-09-18T14:13:54.606 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.606 INFO:tasks.workunit.client.0.smithi047.stdout: "snapid": 1, 2024-09-18T14:13:54.606 INFO:tasks.workunit.client.0.smithi047.stdout: "stamp": "2024-09-18T14:11:58.955067+0000", 2024-09-18T14:13:54.606 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "snap1" 2024-09-18T14:13:54.606 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.606 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:54.607 INFO:tasks.workunit.client.0.smithi047.stdout: "removed_snaps": "[]", 2024-09-18T14:13:54.607 INFO:tasks.workunit.client.0.smithi047.stdout: "quota_max_bytes": 0, 2024-09-18T14:13:54.607 INFO:tasks.workunit.client.0.smithi047.stdout: "quota_max_objects": 0, 2024-09-18T14:13:54.607 INFO:tasks.workunit.client.0.smithi047.stdout: "tiers": [], 2024-09-18T14:13:54.607 INFO:tasks.workunit.client.0.smithi047.stdout: "tier_of": -1, 2024-09-18T14:13:54.607 INFO:tasks.workunit.client.0.smithi047.stdout: "read_tier": -1, 2024-09-18T14:13:54.607 INFO:tasks.workunit.client.0.smithi047.stdout: "write_tier": -1, 2024-09-18T14:13:54.607 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_mode": "none", 2024-09-18T14:13:54.607 INFO:tasks.workunit.client.0.smithi047.stdout: "target_max_bytes": 0, 2024-09-18T14:13:54.607 INFO:tasks.workunit.client.0.smithi047.stdout: "target_max_objects": 0, 2024-09-18T14:13:54.607 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T14:13:54.607 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T14:13:54.607 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T14:13:54.607 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_min_flush_age": 0, 2024-09-18T14:13:54.608 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_min_evict_age": 0, 2024-09-18T14:13:54.608 INFO:tasks.workunit.client.0.smithi047.stdout: "erasure_code_profile": "myprofile", 2024-09-18T14:13:54.608 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_params": { 2024-09-18T14:13:54.608 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "none" 2024-09-18T14:13:54.608 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.608 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_period": 0, 2024-09-18T14:13:54.608 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_count": 0, 2024-09-18T14:13:54.608 INFO:tasks.workunit.client.0.smithi047.stdout: "use_gmt_hitset": true, 2024-09-18T14:13:54.608 INFO:tasks.workunit.client.0.smithi047.stdout: "min_read_recency_for_promote": 0, 2024-09-18T14:13:54.608 INFO:tasks.workunit.client.0.smithi047.stdout: "min_write_recency_for_promote": 0, 2024-09-18T14:13:54.608 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T14:13:54.608 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_search_last_n": 0, 2024-09-18T14:13:54.608 INFO:tasks.workunit.client.0.smithi047.stdout: "grade_table": [], 2024-09-18T14:13:54.608 INFO:tasks.workunit.client.0.smithi047.stdout: "stripe_width": 8192, 2024-09-18T14:13:54.609 INFO:tasks.workunit.client.0.smithi047.stdout: "expected_num_objects": 0, 2024-09-18T14:13:54.609 INFO:tasks.workunit.client.0.smithi047.stdout: "fast_read": false, 2024-09-18T14:13:54.609 INFO:tasks.workunit.client.0.smithi047.stdout: "options": {}, 2024-09-18T14:13:54.609 INFO:tasks.workunit.client.0.smithi047.stdout: "application_metadata": {} 2024-09-18T14:13:54.609 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.609 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:54.609 INFO:tasks.workunit.client.0.smithi047.stdout: "osds": [ 2024-09-18T14:13:54.609 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.609 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 0, 2024-09-18T14:13:54.609 INFO:tasks.workunit.client.0.smithi047.stdout: "uuid": "50835fc1-8964-4c12-b447-46f109dbb8e4", 2024-09-18T14:13:54.609 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 1, 2024-09-18T14:13:54.609 INFO:tasks.workunit.client.0.smithi047.stdout: "in": 1, 2024-09-18T14:13:54.609 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 1, 2024-09-18T14:13:54.609 INFO:tasks.workunit.client.0.smithi047.stdout: "primary_affinity": 1, 2024-09-18T14:13:54.610 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_begin": 36, 2024-09-18T14:13:54.610 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_end": 41, 2024-09-18T14:13:54.610 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 44, 2024-09-18T14:13:54.610 INFO:tasks.workunit.client.0.smithi047.stdout: "up_thru": 48, 2024-09-18T14:13:54.610 INFO:tasks.workunit.client.0.smithi047.stdout: "down_at": 43, 2024-09-18T14:13:54.610 INFO:tasks.workunit.client.0.smithi047.stdout: "lost_at": 0, 2024-09-18T14:13:54.610 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addrs": { 2024-09-18T14:13:54.610 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:13:54.610 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.610 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:13:54.610 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6802", 2024-09-18T14:13:54.610 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 85784 2024-09-18T14:13:54.610 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.611 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.611 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:13:54.611 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6803", 2024-09-18T14:13:54.611 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 85784 2024-09-18T14:13:54.611 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.611 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:54.611 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.611 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addrs": { 2024-09-18T14:13:54.611 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:13:54.611 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.611 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:13:54.611 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6804", 2024-09-18T14:13:54.611 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 85784 2024-09-18T14:13:54.611 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.611 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.612 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:13:54.612 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6805", 2024-09-18T14:13:54.612 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 85784 2024-09-18T14:13:54.612 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.612 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:54.612 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.612 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addrs": { 2024-09-18T14:13:54.612 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:13:54.612 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.612 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:13:54.612 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6808", 2024-09-18T14:13:54.612 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 85784 2024-09-18T14:13:54.612 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.612 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.613 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:13:54.613 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6809", 2024-09-18T14:13:54.613 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 85784 2024-09-18T14:13:54.613 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.613 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:54.613 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.613 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addrs": { 2024-09-18T14:13:54.613 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:13:54.613 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.613 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:13:54.613 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6806", 2024-09-18T14:13:54.613 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 85784 2024-09-18T14:13:54.613 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.613 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.614 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:13:54.614 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6807", 2024-09-18T14:13:54.614 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 85784 2024-09-18T14:13:54.614 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.614 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:54.614 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.614 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addr": "127.0.0.1:6803/85784", 2024-09-18T14:13:54.614 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addr": "127.0.0.1:6805/85784", 2024-09-18T14:13:54.614 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addr": "127.0.0.1:6809/85784", 2024-09-18T14:13:54.614 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addr": "127.0.0.1:6807/85784", 2024-09-18T14:13:54.614 INFO:tasks.workunit.client.0.smithi047.stdout: "state": [ 2024-09-18T14:13:54.614 INFO:tasks.workunit.client.0.smithi047.stdout: "exists", 2024-09-18T14:13:54.614 INFO:tasks.workunit.client.0.smithi047.stdout: "up" 2024-09-18T14:13:54.614 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:54.614 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.615 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.615 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 1, 2024-09-18T14:13:54.615 INFO:tasks.workunit.client.0.smithi047.stdout: "uuid": "f4d67a04-7819-45b5-bc8c-088b7d24eed7", 2024-09-18T14:13:54.615 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 0, 2024-09-18T14:13:54.615 INFO:tasks.workunit.client.0.smithi047.stdout: "in": 1, 2024-09-18T14:13:54.615 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 1, 2024-09-18T14:13:54.615 INFO:tasks.workunit.client.0.smithi047.stdout: "primary_affinity": 1, 2024-09-18T14:13:54.615 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_begin": 0, 2024-09-18T14:13:54.615 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_end": 0, 2024-09-18T14:13:54.615 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 10, 2024-09-18T14:13:54.615 INFO:tasks.workunit.client.0.smithi047.stdout: "up_thru": 25, 2024-09-18T14:13:54.615 INFO:tasks.workunit.client.0.smithi047.stdout: "down_at": 31, 2024-09-18T14:13:54.615 INFO:tasks.workunit.client.0.smithi047.stdout: "lost_at": 0, 2024-09-18T14:13:54.615 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addrs": { 2024-09-18T14:13:54.616 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:13:54.616 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.616 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:13:54.616 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6810", 2024-09-18T14:13:54.616 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 76864 2024-09-18T14:13:54.616 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.616 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.616 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:13:54.616 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6811", 2024-09-18T14:13:54.616 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 76864 2024-09-18T14:13:54.616 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.616 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:54.616 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.616 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addrs": { 2024-09-18T14:13:54.617 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:13:54.617 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.617 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:13:54.617 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6812", 2024-09-18T14:13:54.617 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 76864 2024-09-18T14:13:54.617 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.617 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.617 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:13:54.617 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6813", 2024-09-18T14:13:54.617 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 76864 2024-09-18T14:13:54.617 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.617 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:54.617 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.617 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addrs": { 2024-09-18T14:13:54.617 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:13:54.618 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.618 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:13:54.618 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6816", 2024-09-18T14:13:54.618 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 76864 2024-09-18T14:13:54.618 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.618 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.618 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:13:54.618 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6817", 2024-09-18T14:13:54.618 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 76864 2024-09-18T14:13:54.618 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.618 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:54.618 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.618 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addrs": { 2024-09-18T14:13:54.618 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:13:54.619 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.619 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:13:54.619 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6814", 2024-09-18T14:13:54.619 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 76864 2024-09-18T14:13:54.619 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.619 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.619 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:13:54.619 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6815", 2024-09-18T14:13:54.619 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 76864 2024-09-18T14:13:54.619 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.619 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:54.619 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.619 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addr": "127.0.0.1:6811/76864", 2024-09-18T14:13:54.619 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addr": "127.0.0.1:6813/76864", 2024-09-18T14:13:54.620 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addr": "127.0.0.1:6817/76864", 2024-09-18T14:13:54.620 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addr": "127.0.0.1:6815/76864", 2024-09-18T14:13:54.620 INFO:tasks.workunit.client.0.smithi047.stdout: "state": [ 2024-09-18T14:13:54.620 INFO:tasks.workunit.client.0.smithi047.stdout: "exists" 2024-09-18T14:13:54.620 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:54.620 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.620 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.620 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 2, 2024-09-18T14:13:54.620 INFO:tasks.workunit.client.0.smithi047.stdout: "uuid": "2d53c532-9e0f-4778-a2d9-277af85698d4", 2024-09-18T14:13:54.620 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 1, 2024-09-18T14:13:54.620 INFO:tasks.workunit.client.0.smithi047.stdout: "in": 1, 2024-09-18T14:13:54.620 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 1, 2024-09-18T14:13:54.620 INFO:tasks.workunit.client.0.smithi047.stdout: "primary_affinity": 1, 2024-09-18T14:13:54.620 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_begin": 38, 2024-09-18T14:13:54.620 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_end": 41, 2024-09-18T14:13:54.620 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 48, 2024-09-18T14:13:54.621 INFO:tasks.workunit.client.0.smithi047.stdout: "up_thru": 0, 2024-09-18T14:13:54.621 INFO:tasks.workunit.client.0.smithi047.stdout: "down_at": 47, 2024-09-18T14:13:54.621 INFO:tasks.workunit.client.0.smithi047.stdout: "lost_at": 0, 2024-09-18T14:13:54.621 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addrs": { 2024-09-18T14:13:54.621 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:13:54.621 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.621 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:13:54.621 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6810", 2024-09-18T14:13:54.621 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 86664 2024-09-18T14:13:54.621 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.621 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.621 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:13:54.621 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6811", 2024-09-18T14:13:54.621 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 86664 2024-09-18T14:13:54.622 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.622 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:54.622 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.622 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addrs": { 2024-09-18T14:13:54.622 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:13:54.622 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.622 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:13:54.622 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6812", 2024-09-18T14:13:54.622 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 86664 2024-09-18T14:13:54.622 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.622 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.622 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:13:54.622 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6813", 2024-09-18T14:13:54.622 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 86664 2024-09-18T14:13:54.622 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.623 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:54.623 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.623 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addrs": { 2024-09-18T14:13:54.623 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:13:54.623 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.623 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:13:54.623 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6816", 2024-09-18T14:13:54.623 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 86664 2024-09-18T14:13:54.623 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.623 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.623 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:13:54.623 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6817", 2024-09-18T14:13:54.623 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 86664 2024-09-18T14:13:54.623 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.624 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:54.624 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.624 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addrs": { 2024-09-18T14:13:54.624 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T14:13:54.624 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.624 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T14:13:54.624 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6814", 2024-09-18T14:13:54.624 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 86664 2024-09-18T14:13:54.624 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.624 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.624 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T14:13:54.624 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6815", 2024-09-18T14:13:54.624 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 86664 2024-09-18T14:13:54.624 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.625 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:54.625 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.625 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addr": "127.0.0.1:6811/86664", 2024-09-18T14:13:54.625 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addr": "127.0.0.1:6813/86664", 2024-09-18T14:13:54.625 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addr": "127.0.0.1:6817/86664", 2024-09-18T14:13:54.625 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addr": "127.0.0.1:6815/86664", 2024-09-18T14:13:54.625 INFO:tasks.workunit.client.0.smithi047.stdout: "state": [ 2024-09-18T14:13:54.625 INFO:tasks.workunit.client.0.smithi047.stdout: "exists", 2024-09-18T14:13:54.625 INFO:tasks.workunit.client.0.smithi047.stdout: "up" 2024-09-18T14:13:54.625 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:54.625 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.625 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:54.625 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_xinfo": [ 2024-09-18T14:13:54.626 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.626 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 0, 2024-09-18T14:13:54.626 INFO:tasks.workunit.client.0.smithi047.stdout: "down_stamp": "2024-09-18T14:13:11.197705+0000", 2024-09-18T14:13:54.626 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_probability": 0, 2024-09-18T14:13:54.626 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_interval": 0, 2024-09-18T14:13:54.626 INFO:tasks.workunit.client.0.smithi047.stdout: "features": 4540138320759226367, 2024-09-18T14:13:54.626 INFO:tasks.workunit.client.0.smithi047.stdout: "old_weight": 0, 2024-09-18T14:13:54.626 INFO:tasks.workunit.client.0.smithi047.stdout: "last_purged_snaps_scrub": "2024-09-18T14:10:01.318019+0000", 2024-09-18T14:13:54.626 INFO:tasks.workunit.client.0.smithi047.stdout: "dead_epoch": 0 2024-09-18T14:13:54.626 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.626 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.626 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 1, 2024-09-18T14:13:54.626 INFO:tasks.workunit.client.0.smithi047.stdout: "down_stamp": "2024-09-18T14:12:10.481921+0000", 2024-09-18T14:13:54.626 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_probability": 0, 2024-09-18T14:13:54.626 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_interval": 0, 2024-09-18T14:13:54.627 INFO:tasks.workunit.client.0.smithi047.stdout: "features": 4540138320759226367, 2024-09-18T14:13:54.627 INFO:tasks.workunit.client.0.smithi047.stdout: "old_weight": 0, 2024-09-18T14:13:54.627 INFO:tasks.workunit.client.0.smithi047.stdout: "last_purged_snaps_scrub": "2024-09-18T14:10:35.227876+0000", 2024-09-18T14:13:54.627 INFO:tasks.workunit.client.0.smithi047.stdout: "dead_epoch": 0 2024-09-18T14:13:54.627 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.627 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.627 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 2, 2024-09-18T14:13:54.627 INFO:tasks.workunit.client.0.smithi047.stdout: "down_stamp": "2024-09-18T14:13:13.876336+0000", 2024-09-18T14:13:54.627 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_probability": 0, 2024-09-18T14:13:54.627 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_interval": 0, 2024-09-18T14:13:54.627 INFO:tasks.workunit.client.0.smithi047.stdout: "features": 4540138320759226367, 2024-09-18T14:13:54.627 INFO:tasks.workunit.client.0.smithi047.stdout: "old_weight": 0, 2024-09-18T14:13:54.628 INFO:tasks.workunit.client.0.smithi047.stdout: "last_purged_snaps_scrub": "2024-09-18T14:11:08.904096+0000", 2024-09-18T14:13:54.628 INFO:tasks.workunit.client.0.smithi047.stdout: "dead_epoch": 0 2024-09-18T14:13:54.628 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.628 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:54.628 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_upmap": [], 2024-09-18T14:13:54.628 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_upmap_items": [], 2024-09-18T14:13:54.628 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_temp": [], 2024-09-18T14:13:54.628 INFO:tasks.workunit.client.0.smithi047.stdout: "primary_temp": [], 2024-09-18T14:13:54.628 INFO:tasks.workunit.client.0.smithi047.stdout: "blocklist": {}, 2024-09-18T14:13:54.628 INFO:tasks.workunit.client.0.smithi047.stdout: "range_blocklist": {}, 2024-09-18T14:13:54.628 INFO:tasks.workunit.client.0.smithi047.stdout: "erasure_code_profiles": { 2024-09-18T14:13:54.628 INFO:tasks.workunit.client.0.smithi047.stdout: "default": { 2024-09-18T14:13:54.628 INFO:tasks.workunit.client.0.smithi047.stdout: "k": "2", 2024-09-18T14:13:54.629 INFO:tasks.workunit.client.0.smithi047.stdout: "m": "2", 2024-09-18T14:13:54.629 INFO:tasks.workunit.client.0.smithi047.stdout: "plugin": "jerasure", 2024-09-18T14:13:54.629 INFO:tasks.workunit.client.0.smithi047.stdout: "technique": "reed_sol_van" 2024-09-18T14:13:54.629 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.629 INFO:tasks.workunit.client.0.smithi047.stdout: "myprofile": { 2024-09-18T14:13:54.629 INFO:tasks.workunit.client.0.smithi047.stdout: "crush-device-class": "", 2024-09-18T14:13:54.629 INFO:tasks.workunit.client.0.smithi047.stdout: "crush-failure-domain": "osd", 2024-09-18T14:13:54.629 INFO:tasks.workunit.client.0.smithi047.stdout: "crush-root": "default", 2024-09-18T14:13:54.629 INFO:tasks.workunit.client.0.smithi047.stdout: "jerasure-per-chunk-alignment": "false", 2024-09-18T14:13:54.629 INFO:tasks.workunit.client.0.smithi047.stdout: "k": "2", 2024-09-18T14:13:54.629 INFO:tasks.workunit.client.0.smithi047.stdout: "m": "1", 2024-09-18T14:13:54.629 INFO:tasks.workunit.client.0.smithi047.stdout: "plugin": "jerasure", 2024-09-18T14:13:54.629 INFO:tasks.workunit.client.0.smithi047.stdout: "technique": "reed_sol_van", 2024-09-18T14:13:54.630 INFO:tasks.workunit.client.0.smithi047.stdout: "w": "8" 2024-09-18T14:13:54.630 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.630 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.630 INFO:tasks.workunit.client.0.smithi047.stdout: "removed_snaps_queue": [], 2024-09-18T14:13:54.630 INFO:tasks.workunit.client.0.smithi047.stdout: "new_removed_snaps": [], 2024-09-18T14:13:54.630 INFO:tasks.workunit.client.0.smithi047.stdout: "new_purged_snaps": [], 2024-09-18T14:13:54.630 INFO:tasks.workunit.client.0.smithi047.stdout: "crush_node_flags": {}, 2024-09-18T14:13:54.630 INFO:tasks.workunit.client.0.smithi047.stdout: "device_class_flags": {}, 2024-09-18T14:13:54.630 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_mode": { 2024-09-18T14:13:54.630 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_mode_enabled": false, 2024-09-18T14:13:54.630 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_bucket_count": 0, 2024-09-18T14:13:54.630 INFO:tasks.workunit.client.0.smithi047.stdout: "degraded_stretch_mode": 0, 2024-09-18T14:13:54.630 INFO:tasks.workunit.client.0.smithi047.stdout: "recovering_stretch_mode": 0, 2024-09-18T14:13:54.630 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_mode_bucket": 0 2024-09-18T14:13:54.631 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.631 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.631 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_metadata": [ 2024-09-18T14:13:54.631 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.631 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T14:13:54.631 INFO:tasks.workunit.client.0.smithi047.stdout: "arch": "x86_64", 2024-09-18T14:13:54.631 INFO:tasks.workunit.client.0.smithi047.stdout: "back_addr": "[v2:127.0.0.1:6804/85784,v1:127.0.0.1:6805/85784]", 2024-09-18T14:13:54.631 INFO:tasks.workunit.client.0.smithi047.stdout: "back_iface": "", 2024-09-18T14:13:54.631 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs": "1", 2024-09-18T14:13:54.631 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_db": "0", 2024-09-18T14:13:54.631 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_wal": "0", 2024-09-18T14:13:54.631 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_single_shared_device": "1", 2024-09-18T14:13:54.631 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_allocation_from_file": "0", 2024-09-18T14:13:54.631 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_access_mode": "file", 2024-09-18T14:13:54.632 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_block_size": "4096", 2024-09-18T14:13:54.632 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_devices": "sda", 2024-09-18T14:13:54.632 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-18T14:13:54.632 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-18T14:13:54.632 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-18T14:13:54.632 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_rotational": "1", 2024-09-18T14:13:54.632 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_size": "107374182400", 2024-09-18T14:13:54.632 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_support_discard": "1", 2024-09-18T14:13:54.632 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_type": "hdd", 2024-09-18T14:13:54.632 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_min_alloc_size": "4096", 2024-09-18T14:13:54.632 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_release": "quincy", 2024-09-18T14:13:54.632 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:13:54.632 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_short": "17.2.7-1645-g7061ea6c", 2024-09-18T14:13:54.632 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_when_created": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:13:54.633 INFO:tasks.workunit.client.0.smithi047.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-18T14:13:54.633 INFO:tasks.workunit.client.0.smithi047.stdout: "created_at": "2024-09-18T14:09:35.435983Z", 2024-09-18T14:13:54.633 INFO:tasks.workunit.client.0.smithi047.stdout: "default_device_class": "hdd", 2024-09-18T14:13:54.633 INFO:tasks.workunit.client.0.smithi047.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4HN80", 2024-09-18T14:13:54.633 INFO:tasks.workunit.client.0.smithi047.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-18T14:13:54.633 INFO:tasks.workunit.client.0.smithi047.stdout: "devices": "sda", 2024-09-18T14:13:54.633 INFO:tasks.workunit.client.0.smithi047.stdout: "distro": "centos", 2024-09-18T14:13:54.633 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_description": "CentOS Stream 9", 2024-09-18T14:13:54.633 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_version": "9", 2024-09-18T14:13:54.633 INFO:tasks.workunit.client.0.smithi047.stdout: "front_addr": "[v2:127.0.0.1:6802/85784,v1:127.0.0.1:6803/85784]", 2024-09-18T14:13:54.633 INFO:tasks.workunit.client.0.smithi047.stdout: "front_iface": "", 2024-09-18T14:13:54.633 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/85784,v1:127.0.0.1:6809/85784]", 2024-09-18T14:13:54.633 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/85784,v1:127.0.0.1:6807/85784]", 2024-09-18T14:13:54.633 INFO:tasks.workunit.client.0.smithi047.stdout: "hostname": "smithi047", 2024-09-18T14:13:54.634 INFO:tasks.workunit.client.0.smithi047.stdout: "journal_rotational": "1", 2024-09-18T14:13:54.634 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-18T14:13:54.634 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-18T14:13:54.634 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_swap_kb": "0", 2024-09-18T14:13:54.634 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_total_kb": "32493492", 2024-09-18T14:13:54.634 INFO:tasks.workunit.client.0.smithi047.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-18T14:13:54.634 INFO:tasks.workunit.client.0.smithi047.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-18T14:13:54.634 INFO:tasks.workunit.client.0.smithi047.stdout: "os": "Linux", 2024-09-18T14:13:54.634 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_data": "td/divergent-priors/0", 2024-09-18T14:13:54.634 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_objectstore": "bluestore", 2024-09-18T14:13:54.634 INFO:tasks.workunit.client.0.smithi047.stdout: "osdspec_affinity": "", 2024-09-18T14:13:54.634 INFO:tasks.workunit.client.0.smithi047.stdout: "rotational": "1" 2024-09-18T14:13:54.634 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.634 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.635 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 1, 2024-09-18T14:13:54.635 INFO:tasks.workunit.client.0.smithi047.stdout: "arch": "x86_64", 2024-09-18T14:13:54.635 INFO:tasks.workunit.client.0.smithi047.stdout: "back_addr": "[v2:127.0.0.1:6812/76864,v1:127.0.0.1:6813/76864]", 2024-09-18T14:13:54.635 INFO:tasks.workunit.client.0.smithi047.stdout: "back_iface": "", 2024-09-18T14:13:54.635 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs": "1", 2024-09-18T14:13:54.635 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_db": "0", 2024-09-18T14:13:54.635 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_wal": "0", 2024-09-18T14:13:54.635 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_single_shared_device": "1", 2024-09-18T14:13:54.635 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_allocation_from_file": "0", 2024-09-18T14:13:54.635 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_access_mode": "file", 2024-09-18T14:13:54.635 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_block_size": "4096", 2024-09-18T14:13:54.635 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_devices": "sda", 2024-09-18T14:13:54.635 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-18T14:13:54.635 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-18T14:13:54.636 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-18T14:13:54.636 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_rotational": "1", 2024-09-18T14:13:54.636 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_size": "107374182400", 2024-09-18T14:13:54.636 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_support_discard": "1", 2024-09-18T14:13:54.636 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_type": "hdd", 2024-09-18T14:13:54.636 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_min_alloc_size": "4096", 2024-09-18T14:13:54.636 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_release": "quincy", 2024-09-18T14:13:54.636 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:13:54.636 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_short": "17.2.7-1645-g7061ea6c", 2024-09-18T14:13:54.636 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_when_created": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:13:54.636 INFO:tasks.workunit.client.0.smithi047.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-18T14:13:54.636 INFO:tasks.workunit.client.0.smithi047.stdout: "created_at": "2024-09-18T14:10:08.245708Z", 2024-09-18T14:13:54.636 INFO:tasks.workunit.client.0.smithi047.stdout: "default_device_class": "hdd", 2024-09-18T14:13:54.636 INFO:tasks.workunit.client.0.smithi047.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4HN80", 2024-09-18T14:13:54.637 INFO:tasks.workunit.client.0.smithi047.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-18T14:13:54.637 INFO:tasks.workunit.client.0.smithi047.stdout: "devices": "sda", 2024-09-18T14:13:54.637 INFO:tasks.workunit.client.0.smithi047.stdout: "distro": "centos", 2024-09-18T14:13:54.637 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_description": "CentOS Stream 9", 2024-09-18T14:13:54.637 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_version": "9", 2024-09-18T14:13:54.637 INFO:tasks.workunit.client.0.smithi047.stdout: "front_addr": "[v2:127.0.0.1:6810/76864,v1:127.0.0.1:6811/76864]", 2024-09-18T14:13:54.637 INFO:tasks.workunit.client.0.smithi047.stdout: "front_iface": "", 2024-09-18T14:13:54.637 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/76864,v1:127.0.0.1:6817/76864]", 2024-09-18T14:13:54.637 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/76864,v1:127.0.0.1:6815/76864]", 2024-09-18T14:13:54.637 INFO:tasks.workunit.client.0.smithi047.stdout: "hostname": "smithi047", 2024-09-18T14:13:54.637 INFO:tasks.workunit.client.0.smithi047.stdout: "journal_rotational": "1", 2024-09-18T14:13:54.637 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-18T14:13:54.637 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-18T14:13:54.637 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_swap_kb": "0", 2024-09-18T14:13:54.638 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_total_kb": "32493492", 2024-09-18T14:13:54.638 INFO:tasks.workunit.client.0.smithi047.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-18T14:13:54.638 INFO:tasks.workunit.client.0.smithi047.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-18T14:13:54.638 INFO:tasks.workunit.client.0.smithi047.stdout: "os": "Linux", 2024-09-18T14:13:54.638 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_data": "td/divergent-priors/1", 2024-09-18T14:13:54.638 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_objectstore": "bluestore", 2024-09-18T14:13:54.638 INFO:tasks.workunit.client.0.smithi047.stdout: "osdspec_affinity": "", 2024-09-18T14:13:54.638 INFO:tasks.workunit.client.0.smithi047.stdout: "rotational": "1" 2024-09-18T14:13:54.638 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.638 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.638 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T14:13:54.638 INFO:tasks.workunit.client.0.smithi047.stdout: "arch": "x86_64", 2024-09-18T14:13:54.638 INFO:tasks.workunit.client.0.smithi047.stdout: "back_addr": "[v2:127.0.0.1:6812/86664,v1:127.0.0.1:6813/86664]", 2024-09-18T14:13:54.638 INFO:tasks.workunit.client.0.smithi047.stdout: "back_iface": "", 2024-09-18T14:13:54.639 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs": "1", 2024-09-18T14:13:54.639 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_db": "0", 2024-09-18T14:13:54.639 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_wal": "0", 2024-09-18T14:13:54.639 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_single_shared_device": "1", 2024-09-18T14:13:54.639 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_allocation_from_file": "0", 2024-09-18T14:13:54.639 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_access_mode": "file", 2024-09-18T14:13:54.639 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_block_size": "4096", 2024-09-18T14:13:54.639 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_devices": "sda", 2024-09-18T14:13:54.639 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-18T14:13:54.639 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-18T14:13:54.639 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-18T14:13:54.639 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_rotational": "1", 2024-09-18T14:13:54.639 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_size": "107374182400", 2024-09-18T14:13:54.639 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_support_discard": "1", 2024-09-18T14:13:54.640 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_type": "hdd", 2024-09-18T14:13:54.640 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_min_alloc_size": "4096", 2024-09-18T14:13:54.640 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_release": "quincy", 2024-09-18T14:13:54.640 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:13:54.640 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_short": "17.2.7-1645-g7061ea6c", 2024-09-18T14:13:54.640 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_when_created": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T14:13:54.640 INFO:tasks.workunit.client.0.smithi047.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-18T14:13:54.640 INFO:tasks.workunit.client.0.smithi047.stdout: "created_at": "2024-09-18T14:10:42.506913Z", 2024-09-18T14:13:54.640 INFO:tasks.workunit.client.0.smithi047.stdout: "default_device_class": "hdd", 2024-09-18T14:13:54.640 INFO:tasks.workunit.client.0.smithi047.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4HN80", 2024-09-18T14:13:54.640 INFO:tasks.workunit.client.0.smithi047.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-18T14:13:54.640 INFO:tasks.workunit.client.0.smithi047.stdout: "devices": "sda", 2024-09-18T14:13:54.640 INFO:tasks.workunit.client.0.smithi047.stdout: "distro": "centos", 2024-09-18T14:13:54.641 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_description": "CentOS Stream 9", 2024-09-18T14:13:54.641 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_version": "9", 2024-09-18T14:13:54.641 INFO:tasks.workunit.client.0.smithi047.stdout: "front_addr": "[v2:127.0.0.1:6810/86664,v1:127.0.0.1:6811/86664]", 2024-09-18T14:13:54.641 INFO:tasks.workunit.client.0.smithi047.stdout: "front_iface": "", 2024-09-18T14:13:54.641 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/86664,v1:127.0.0.1:6817/86664]", 2024-09-18T14:13:54.641 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/86664,v1:127.0.0.1:6815/86664]", 2024-09-18T14:13:54.641 INFO:tasks.workunit.client.0.smithi047.stdout: "hostname": "smithi047", 2024-09-18T14:13:54.641 INFO:tasks.workunit.client.0.smithi047.stdout: "journal_rotational": "1", 2024-09-18T14:13:54.641 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-18T14:13:54.641 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-18T14:13:54.641 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_swap_kb": "0", 2024-09-18T14:13:54.641 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_total_kb": "32493492", 2024-09-18T14:13:54.641 INFO:tasks.workunit.client.0.smithi047.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-18T14:13:54.641 INFO:tasks.workunit.client.0.smithi047.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-18T14:13:54.641 INFO:tasks.workunit.client.0.smithi047.stdout: "os": "Linux", 2024-09-18T14:13:54.642 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_data": "td/divergent-priors/2", 2024-09-18T14:13:54.642 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_objectstore": "bluestore", 2024-09-18T14:13:54.642 INFO:tasks.workunit.client.0.smithi047.stdout: "osdspec_affinity": "", 2024-09-18T14:13:54.642 INFO:tasks.workunit.client.0.smithi047.stdout: "rotational": "1" 2024-09-18T14:13:54.642 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.642 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:54.642 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap_clean_epochs": { 2024-09-18T14:13:54.642 INFO:tasks.workunit.client.0.smithi047.stdout: "min_last_epoch_clean": 0, 2024-09-18T14:13:54.642 INFO:tasks.workunit.client.0.smithi047.stdout: "last_epoch_clean": { 2024-09-18T14:13:54.642 INFO:tasks.workunit.client.0.smithi047.stdout: "per_pool": [] 2024-09-18T14:13:54.642 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.642 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_epochs": [ 2024-09-18T14:13:54.642 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.642 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T14:13:54.642 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 44 2024-09-18T14:13:54.643 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.643 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.643 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T14:13:54.643 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 48 2024-09-18T14:13:54.643 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.643 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:54.643 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.643 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap_first_committed": 1, 2024-09-18T14:13:54.643 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap_last_committed": 49, 2024-09-18T14:13:54.643 INFO:tasks.workunit.client.0.smithi047.stdout: "crushmap": { 2024-09-18T14:13:54.643 INFO:tasks.workunit.client.0.smithi047.stdout: "devices": [ 2024-09-18T14:13:54.643 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.643 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T14:13:54.643 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "osd.0", 2024-09-18T14:13:54.644 INFO:tasks.workunit.client.0.smithi047.stdout: "class": "hdd" 2024-09-18T14:13:54.644 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.644 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.644 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 1, 2024-09-18T14:13:54.644 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "osd.1", 2024-09-18T14:13:54.644 INFO:tasks.workunit.client.0.smithi047.stdout: "class": "hdd" 2024-09-18T14:13:54.644 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.644 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.644 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T14:13:54.644 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "osd.2", 2024-09-18T14:13:54.644 INFO:tasks.workunit.client.0.smithi047.stdout: "class": "hdd" 2024-09-18T14:13:54.644 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.644 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:54.644 INFO:tasks.workunit.client.0.smithi047.stdout: "types": [ 2024-09-18T14:13:54.644 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.645 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 0, 2024-09-18T14:13:54.645 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "osd" 2024-09-18T14:13:54.645 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.645 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.645 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 1, 2024-09-18T14:13:54.645 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "host" 2024-09-18T14:13:54.645 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.645 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.645 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 2, 2024-09-18T14:13:54.645 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "chassis" 2024-09-18T14:13:54.645 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.645 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.645 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 3, 2024-09-18T14:13:54.645 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "rack" 2024-09-18T14:13:54.646 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.646 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.646 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 4, 2024-09-18T14:13:54.646 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "row" 2024-09-18T14:13:54.646 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.646 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.646 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 5, 2024-09-18T14:13:54.646 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "pdu" 2024-09-18T14:13:54.646 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.646 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.647 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 6, 2024-09-18T14:13:54.647 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "pod" 2024-09-18T14:13:54.647 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.647 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.647 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 7, 2024-09-18T14:13:54.647 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "room" 2024-09-18T14:13:54.647 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.647 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.647 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 8, 2024-09-18T14:13:54.647 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "datacenter" 2024-09-18T14:13:54.647 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.647 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.647 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 9, 2024-09-18T14:13:54.647 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "zone" 2024-09-18T14:13:54.648 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.648 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.648 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 10, 2024-09-18T14:13:54.648 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "region" 2024-09-18T14:13:54.648 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.648 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.648 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 11, 2024-09-18T14:13:54.648 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "root" 2024-09-18T14:13:54.648 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.648 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:54.648 INFO:tasks.workunit.client.0.smithi047.stdout: "buckets": [ 2024-09-18T14:13:54.648 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.648 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -1, 2024-09-18T14:13:54.649 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "default", 2024-09-18T14:13:54.649 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 11, 2024-09-18T14:13:54.649 INFO:tasks.workunit.client.0.smithi047.stdout: "type_name": "root", 2024-09-18T14:13:54.649 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:13:54.649 INFO:tasks.workunit.client.0.smithi047.stdout: "alg": "straw2", 2024-09-18T14:13:54.649 INFO:tasks.workunit.client.0.smithi047.stdout: "hash": "rjenkins1", 2024-09-18T14:13:54.649 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T14:13:54.649 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.649 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -3, 2024-09-18T14:13:54.649 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:13:54.649 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 0 2024-09-18T14:13:54.649 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.649 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:54.649 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.650 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.650 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -2, 2024-09-18T14:13:54.650 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "default~hdd", 2024-09-18T14:13:54.650 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 11, 2024-09-18T14:13:54.650 INFO:tasks.workunit.client.0.smithi047.stdout: "type_name": "root", 2024-09-18T14:13:54.650 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:13:54.650 INFO:tasks.workunit.client.0.smithi047.stdout: "alg": "straw2", 2024-09-18T14:13:54.650 INFO:tasks.workunit.client.0.smithi047.stdout: "hash": "rjenkins1", 2024-09-18T14:13:54.650 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T14:13:54.650 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.650 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -4, 2024-09-18T14:13:54.650 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:13:54.650 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 0 2024-09-18T14:13:54.651 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.651 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:54.651 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.651 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.651 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -3, 2024-09-18T14:13:54.651 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "smithi047", 2024-09-18T14:13:54.651 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 1, 2024-09-18T14:13:54.651 INFO:tasks.workunit.client.0.smithi047.stdout: "type_name": "host", 2024-09-18T14:13:54.651 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:13:54.651 INFO:tasks.workunit.client.0.smithi047.stdout: "alg": "straw2", 2024-09-18T14:13:54.651 INFO:tasks.workunit.client.0.smithi047.stdout: "hash": "rjenkins1", 2024-09-18T14:13:54.651 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T14:13:54.651 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.651 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T14:13:54.651 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:13:54.652 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 0 2024-09-18T14:13:54.652 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.652 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.652 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 1, 2024-09-18T14:13:54.652 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:13:54.652 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 1 2024-09-18T14:13:54.652 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.652 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.652 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T14:13:54.652 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:13:54.652 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 2 2024-09-18T14:13:54.652 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.652 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:54.652 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.653 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.653 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -4, 2024-09-18T14:13:54.653 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "smithi047~hdd", 2024-09-18T14:13:54.653 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 1, 2024-09-18T14:13:54.653 INFO:tasks.workunit.client.0.smithi047.stdout: "type_name": "host", 2024-09-18T14:13:54.653 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 19206, 2024-09-18T14:13:54.653 INFO:tasks.workunit.client.0.smithi047.stdout: "alg": "straw2", 2024-09-18T14:13:54.653 INFO:tasks.workunit.client.0.smithi047.stdout: "hash": "rjenkins1", 2024-09-18T14:13:54.653 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T14:13:54.653 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.653 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T14:13:54.653 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:13:54.653 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 0 2024-09-18T14:13:54.653 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.654 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.654 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 1, 2024-09-18T14:13:54.654 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:13:54.654 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 1 2024-09-18T14:13:54.654 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.654 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.654 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T14:13:54.654 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T14:13:54.654 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 2 2024-09-18T14:13:54.654 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.654 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:54.654 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.654 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:54.654 INFO:tasks.workunit.client.0.smithi047.stdout: "rules": [ 2024-09-18T14:13:54.655 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.655 INFO:tasks.workunit.client.0.smithi047.stdout: "rule_id": 0, 2024-09-18T14:13:54.655 INFO:tasks.workunit.client.0.smithi047.stdout: "rule_name": "replicated_rule", 2024-09-18T14:13:54.655 INFO:tasks.workunit.client.0.smithi047.stdout: "type": 1, 2024-09-18T14:13:54.655 INFO:tasks.workunit.client.0.smithi047.stdout: "steps": [ 2024-09-18T14:13:54.655 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.655 INFO:tasks.workunit.client.0.smithi047.stdout: "op": "take", 2024-09-18T14:13:54.655 INFO:tasks.workunit.client.0.smithi047.stdout: "item": -1, 2024-09-18T14:13:54.655 INFO:tasks.workunit.client.0.smithi047.stdout: "item_name": "default" 2024-09-18T14:13:54.655 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.655 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.655 INFO:tasks.workunit.client.0.smithi047.stdout: "op": "choose_firstn", 2024-09-18T14:13:54.655 INFO:tasks.workunit.client.0.smithi047.stdout: "num": 0, 2024-09-18T14:13:54.655 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "osd" 2024-09-18T14:13:54.655 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.656 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.656 INFO:tasks.workunit.client.0.smithi047.stdout: "op": "emit" 2024-09-18T14:13:54.656 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.656 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:54.656 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.656 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.656 INFO:tasks.workunit.client.0.smithi047.stdout: "rule_id": 1, 2024-09-18T14:13:54.656 INFO:tasks.workunit.client.0.smithi047.stdout: "rule_name": "test", 2024-09-18T14:13:54.656 INFO:tasks.workunit.client.0.smithi047.stdout: "type": 3, 2024-09-18T14:13:54.656 INFO:tasks.workunit.client.0.smithi047.stdout: "steps": [ 2024-09-18T14:13:54.656 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.656 INFO:tasks.workunit.client.0.smithi047.stdout: "op": "set_chooseleaf_tries", 2024-09-18T14:13:54.656 INFO:tasks.workunit.client.0.smithi047.stdout: "num": 5 2024-09-18T14:13:54.656 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.657 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.657 INFO:tasks.workunit.client.0.smithi047.stdout: "op": "set_choose_tries", 2024-09-18T14:13:54.657 INFO:tasks.workunit.client.0.smithi047.stdout: "num": 100 2024-09-18T14:13:54.657 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.657 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.657 INFO:tasks.workunit.client.0.smithi047.stdout: "op": "take", 2024-09-18T14:13:54.657 INFO:tasks.workunit.client.0.smithi047.stdout: "item": -1, 2024-09-18T14:13:54.657 INFO:tasks.workunit.client.0.smithi047.stdout: "item_name": "default" 2024-09-18T14:13:54.657 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.657 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.657 INFO:tasks.workunit.client.0.smithi047.stdout: "op": "choose_indep", 2024-09-18T14:13:54.657 INFO:tasks.workunit.client.0.smithi047.stdout: "num": 0, 2024-09-18T14:13:54.657 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "osd" 2024-09-18T14:13:54.658 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.658 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.658 INFO:tasks.workunit.client.0.smithi047.stdout: "op": "emit" 2024-09-18T14:13:54.658 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.658 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:13:54.658 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.658 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:54.658 INFO:tasks.workunit.client.0.smithi047.stdout: "tunables": { 2024-09-18T14:13:54.658 INFO:tasks.workunit.client.0.smithi047.stdout: "choose_local_tries": 0, 2024-09-18T14:13:54.658 INFO:tasks.workunit.client.0.smithi047.stdout: "choose_local_fallback_tries": 0, 2024-09-18T14:13:54.658 INFO:tasks.workunit.client.0.smithi047.stdout: "choose_total_tries": 50, 2024-09-18T14:13:54.658 INFO:tasks.workunit.client.0.smithi047.stdout: "chooseleaf_descend_once": 1, 2024-09-18T14:13:54.658 INFO:tasks.workunit.client.0.smithi047.stdout: "chooseleaf_vary_r": 1, 2024-09-18T14:13:54.658 INFO:tasks.workunit.client.0.smithi047.stdout: "chooseleaf_stable": 1, 2024-09-18T14:13:54.658 INFO:tasks.workunit.client.0.smithi047.stdout: "straw_calc_version": 1, 2024-09-18T14:13:54.659 INFO:tasks.workunit.client.0.smithi047.stdout: "allowed_bucket_algs": 54, 2024-09-18T14:13:54.659 INFO:tasks.workunit.client.0.smithi047.stdout: "profile": "jewel", 2024-09-18T14:13:54.659 INFO:tasks.workunit.client.0.smithi047.stdout: "optimal_tunables": 1, 2024-09-18T14:13:54.659 INFO:tasks.workunit.client.0.smithi047.stdout: "legacy_tunables": 0, 2024-09-18T14:13:54.659 INFO:tasks.workunit.client.0.smithi047.stdout: "minimum_required_version": "jewel", 2024-09-18T14:13:54.659 INFO:tasks.workunit.client.0.smithi047.stderr:report 925551411 2024-09-18T14:13:54.660 INFO:tasks.workunit.client.0.smithi047.stdout: "require_feature_tunables": 1, 2024-09-18T14:13:54.660 INFO:tasks.workunit.client.0.smithi047.stdout: "require_feature_tunables2": 1, 2024-09-18T14:13:54.660 INFO:tasks.workunit.client.0.smithi047.stdout: "has_v2_rules": 1, 2024-09-18T14:13:54.660 INFO:tasks.workunit.client.0.smithi047.stdout: "require_feature_tunables3": 1, 2024-09-18T14:13:54.660 INFO:tasks.workunit.client.0.smithi047.stdout: "has_v3_rules": 0, 2024-09-18T14:13:54.660 INFO:tasks.workunit.client.0.smithi047.stdout: "has_v4_buckets": 1, 2024-09-18T14:13:54.660 INFO:tasks.workunit.client.0.smithi047.stdout: "require_feature_tunables5": 1, 2024-09-18T14:13:54.660 INFO:tasks.workunit.client.0.smithi047.stdout: "has_v5_rules": 0 2024-09-18T14:13:54.660 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.660 INFO:tasks.workunit.client.0.smithi047.stdout: "choose_args": {} 2024-09-18T14:13:54.660 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.660 INFO:tasks.workunit.client.0.smithi047.stdout: "fsmap": { 2024-09-18T14:13:54.661 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 1, 2024-09-18T14:13:54.661 INFO:tasks.workunit.client.0.smithi047.stdout: "default_fscid": -1, 2024-09-18T14:13:54.661 INFO:tasks.workunit.client.0.smithi047.stdout: "compat": { 2024-09-18T14:13:54.661 INFO:tasks.workunit.client.0.smithi047.stdout: "compat": {}, 2024-09-18T14:13:54.661 INFO:tasks.workunit.client.0.smithi047.stdout: "ro_compat": {}, 2024-09-18T14:13:54.661 INFO:tasks.workunit.client.0.smithi047.stdout: "incompat": { 2024-09-18T14:13:54.661 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_1": "base v0.20", 2024-09-18T14:13:54.661 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_2": "client writeable ranges", 2024-09-18T14:13:54.661 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_3": "default file layouts on dirs", 2024-09-18T14:13:54.661 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_4": "dir inode in separate object", 2024-09-18T14:13:54.661 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_5": "mds uses versioned encoding", 2024-09-18T14:13:54.661 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-18T14:13:54.661 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_8": "no anchor table", 2024-09-18T14:13:54.661 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_9": "file layout v2", 2024-09-18T14:13:54.662 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_10": "snaprealm v2" 2024-09-18T14:13:54.662 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.662 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.662 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_flags": { 2024-09-18T14:13:54.662 INFO:tasks.workunit.client.0.smithi047.stdout: "enable_multiple": true, 2024-09-18T14:13:54.662 INFO:tasks.workunit.client.0.smithi047.stdout: "ever_enabled_multiple": true 2024-09-18T14:13:54.662 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.662 INFO:tasks.workunit.client.0.smithi047.stdout: "standbys": [], 2024-09-18T14:13:54.662 INFO:tasks.workunit.client.0.smithi047.stdout: "filesystems": [] 2024-09-18T14:13:54.662 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.662 INFO:tasks.workunit.client.0.smithi047.stdout: "mdsmap_first_committed": 1, 2024-09-18T14:13:54.662 INFO:tasks.workunit.client.0.smithi047.stdout: "mdsmap_last_committed": 1, 2024-09-18T14:13:54.662 INFO:tasks.workunit.client.0.smithi047.stdout: "auth": { 2024-09-18T14:13:54.662 INFO:tasks.workunit.client.0.smithi047.stdout: "first_committed": 1, 2024-09-18T14:13:54.663 INFO:tasks.workunit.client.0.smithi047.stdout: "last_committed": 5, 2024-09-18T14:13:54.663 INFO:tasks.workunit.client.0.smithi047.stdout: "num_secrets": 3 2024-09-18T14:13:54.663 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.663 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg": 1, 2024-09-18T14:13:54.663 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg_active": 1, 2024-09-18T14:13:54.663 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg_unknown": 0, 2024-09-18T14:13:54.663 INFO:tasks.workunit.client.0.smithi047.stdout: "num_osd": 3, 2024-09-18T14:13:54.663 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_sum": { 2024-09-18T14:13:54.663 INFO:tasks.workunit.client.0.smithi047.stdout: "stat_sum": { 2024-09-18T14:13:54.663 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes": 48454, 2024-09-18T14:13:54.663 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects": 101, 2024-09-18T14:13:54.663 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_clones": 1, 2024-09-18T14:13:54.663 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_copies": 303, 2024-09-18T14:13:54.664 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing_on_primary": 0, 2024-09-18T14:13:54.664 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing": 0, 2024-09-18T14:13:54.664 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_degraded": 101, 2024-09-18T14:13:54.664 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_misplaced": 0, 2024-09-18T14:13:54.664 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_unfound": 0, 2024-09-18T14:13:54.664 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_dirty": 101, 2024-09-18T14:13:54.664 INFO:tasks.workunit.client.0.smithi047.stdout: "num_whiteouts": 0, 2024-09-18T14:13:54.664 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read": 0, 2024-09-18T14:13:54.664 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read_kb": 0, 2024-09-18T14:13:54.664 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write": 101, 2024-09-18T14:13:54.664 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write_kb": 101, 2024-09-18T14:13:54.664 INFO:tasks.workunit.client.0.smithi047.stdout: "num_scrub_errors": 0, 2024-09-18T14:13:54.664 INFO:tasks.workunit.client.0.smithi047.stdout: "num_shallow_scrub_errors": 0, 2024-09-18T14:13:54.664 INFO:tasks.workunit.client.0.smithi047.stdout: "num_deep_scrub_errors": 0, 2024-09-18T14:13:54.665 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_recovered": 0, 2024-09-18T14:13:54.665 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_recovered": 0, 2024-09-18T14:13:54.665 INFO:tasks.workunit.client.0.smithi047.stdout: "num_keys_recovered": 0, 2024-09-18T14:13:54.665 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_omap": 0, 2024-09-18T14:13:54.665 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_hit_set_archive": 0, 2024-09-18T14:13:54.665 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_hit_set_archive": 0, 2024-09-18T14:13:54.665 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush": 0, 2024-09-18T14:13:54.665 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_kb": 0, 2024-09-18T14:13:54.665 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict": 0, 2024-09-18T14:13:54.665 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_kb": 0, 2024-09-18T14:13:54.665 INFO:tasks.workunit.client.0.smithi047.stdout: "num_promote": 0, 2024-09-18T14:13:54.665 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_high": 0, 2024-09-18T14:13:54.665 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_low": 0, 2024-09-18T14:13:54.665 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_some": 0, 2024-09-18T14:13:54.666 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_full": 0, 2024-09-18T14:13:54.666 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_pinned": 0, 2024-09-18T14:13:54.666 INFO:tasks.workunit.client.0.smithi047.stdout: "num_legacy_snapsets": 0, 2024-09-18T14:13:54.666 INFO:tasks.workunit.client.0.smithi047.stdout: "num_large_omap_objects": 0, 2024-09-18T14:13:54.666 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_manifest": 0, 2024-09-18T14:13:54.666 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_bytes": 0, 2024-09-18T14:13:54.666 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_keys": 0, 2024-09-18T14:13:54.666 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_repaired": 0 2024-09-18T14:13:54.666 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.666 INFO:tasks.workunit.client.0.smithi047.stdout: "store_stats": { 2024-09-18T14:13:54.666 INFO:tasks.workunit.client.0.smithi047.stdout: "total": 0, 2024-09-18T14:13:54.666 INFO:tasks.workunit.client.0.smithi047.stdout: "available": 0, 2024-09-18T14:13:54.666 INFO:tasks.workunit.client.0.smithi047.stdout: "internally_reserved": 0, 2024-09-18T14:13:54.666 INFO:tasks.workunit.client.0.smithi047.stdout: "allocated": 0, 2024-09-18T14:13:54.667 INFO:tasks.workunit.client.0.smithi047.stdout: "data_stored": 0, 2024-09-18T14:13:54.667 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed": 0, 2024-09-18T14:13:54.667 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_allocated": 0, 2024-09-18T14:13:54.667 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_original": 0, 2024-09-18T14:13:54.667 INFO:tasks.workunit.client.0.smithi047.stdout: "omap_allocated": 0, 2024-09-18T14:13:54.667 INFO:tasks.workunit.client.0.smithi047.stdout: "internal_metadata": 0 2024-09-18T14:13:54.667 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.667 INFO:tasks.workunit.client.0.smithi047.stdout: "log_size": 18, 2024-09-18T14:13:54.667 INFO:tasks.workunit.client.0.smithi047.stdout: "ondisk_log_size": 18, 2024-09-18T14:13:54.667 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 3, 2024-09-18T14:13:54.667 INFO:tasks.workunit.client.0.smithi047.stdout: "acting": 3, 2024-09-18T14:13:54.667 INFO:tasks.workunit.client.0.smithi047.stdout: "num_store_stats": 0 2024-09-18T14:13:54.667 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.667 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_sum": { 2024-09-18T14:13:54.667 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 0, 2024-09-18T14:13:54.668 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 0, 2024-09-18T14:13:54.668 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pgs": 3, 2024-09-18T14:13:54.668 INFO:tasks.workunit.client.0.smithi047.stdout: "num_osds": 3, 2024-09-18T14:13:54.668 INFO:tasks.workunit.client.0.smithi047.stdout: "num_per_pool_osds": 3, 2024-09-18T14:13:54.668 INFO:tasks.workunit.client.0.smithi047.stdout: "num_per_pool_omap_osds": 3, 2024-09-18T14:13:54.668 INFO:tasks.workunit.client.0.smithi047.stdout: "kb": 314572800, 2024-09-18T14:13:54.668 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used": 895688, 2024-09-18T14:13:54.668 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_data": 2252, 2024-09-18T14:13:54.668 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_omap": 16, 2024-09-18T14:13:54.668 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_meta": 893359, 2024-09-18T14:13:54.668 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_avail": 313677112, 2024-09-18T14:13:54.668 INFO:tasks.workunit.client.0.smithi047.stdout: "statfs": { 2024-09-18T14:13:54.668 INFO:tasks.workunit.client.0.smithi047.stdout: "total": 322122547200, 2024-09-18T14:13:54.668 INFO:tasks.workunit.client.0.smithi047.stdout: "available": 321205362688, 2024-09-18T14:13:54.668 INFO:tasks.workunit.client.0.smithi047.stdout: "internally_reserved": 0, 2024-09-18T14:13:54.669 INFO:tasks.workunit.client.0.smithi047.stdout: "allocated": 2306048, 2024-09-18T14:13:54.669 INFO:tasks.workunit.client.0.smithi047.stdout: "data_stored": 1582468, 2024-09-18T14:13:54.669 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed": 0, 2024-09-18T14:13:54.669 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_allocated": 0, 2024-09-18T14:13:54.669 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_original": 0, 2024-09-18T14:13:54.669 INFO:tasks.workunit.client.0.smithi047.stdout: "omap_allocated": 17014, 2024-09-18T14:13:54.669 INFO:tasks.workunit.client.0.smithi047.stdout: "internal_metadata": 914800010 2024-09-18T14:13:54.669 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.669 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_peers": [], 2024-09-18T14:13:54.669 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_trim_queue_len": 0, 2024-09-18T14:13:54.669 INFO:tasks.workunit.client.0.smithi047.stdout: "num_snap_trimming": 0, 2024-09-18T14:13:54.669 INFO:tasks.workunit.client.0.smithi047.stdout: "num_shards_repaired": 0, 2024-09-18T14:13:54.669 INFO:tasks.workunit.client.0.smithi047.stdout: "op_queue_age_hist": { 2024-09-18T14:13:54.669 INFO:tasks.workunit.client.0.smithi047.stdout: "histogram": [], 2024-09-18T14:13:54.670 INFO:tasks.workunit.client.0.smithi047.stdout: "upper_bound": 1 2024-09-18T14:13:54.670 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.670 INFO:tasks.workunit.client.0.smithi047.stdout: "perf_stat": { 2024-09-18T14:13:54.670 INFO:tasks.workunit.client.0.smithi047.stdout: "commit_latency_ms": 0, 2024-09-18T14:13:54.670 INFO:tasks.workunit.client.0.smithi047.stdout: "apply_latency_ms": 0, 2024-09-18T14:13:54.670 INFO:tasks.workunit.client.0.smithi047.stdout: "commit_latency_ns": 0, 2024-09-18T14:13:54.670 INFO:tasks.workunit.client.0.smithi047.stdout: "apply_latency_ns": 0 2024-09-18T14:13:54.670 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.670 INFO:tasks.workunit.client.0.smithi047.stdout: "alerts": [], 2024-09-18T14:13:54.670 INFO:tasks.workunit.client.0.smithi047.stdout: "network_ping_times": [] 2024-09-18T14:13:54.670 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.670 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_sum_by_class": { 2024-09-18T14:13:54.670 INFO:tasks.workunit.client.0.smithi047.stdout: "hdd": { 2024-09-18T14:13:54.670 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 0, 2024-09-18T14:13:54.671 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 0, 2024-09-18T14:13:54.671 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pgs": 3, 2024-09-18T14:13:54.671 INFO:tasks.workunit.client.0.smithi047.stdout: "num_osds": 3, 2024-09-18T14:13:54.671 INFO:tasks.workunit.client.0.smithi047.stdout: "num_per_pool_osds": 3, 2024-09-18T14:13:54.671 INFO:tasks.workunit.client.0.smithi047.stdout: "num_per_pool_omap_osds": 3, 2024-09-18T14:13:54.671 INFO:tasks.workunit.client.0.smithi047.stdout: "kb": 314572800, 2024-09-18T14:13:54.671 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used": 895688, 2024-09-18T14:13:54.671 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_data": 2252, 2024-09-18T14:13:54.671 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_omap": 16, 2024-09-18T14:13:54.671 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_meta": 893359, 2024-09-18T14:13:54.671 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_avail": 313677112, 2024-09-18T14:13:54.671 INFO:tasks.workunit.client.0.smithi047.stdout: "statfs": { 2024-09-18T14:13:54.671 INFO:tasks.workunit.client.0.smithi047.stdout: "total": 322122547200, 2024-09-18T14:13:54.671 INFO:tasks.workunit.client.0.smithi047.stdout: "available": 321205362688, 2024-09-18T14:13:54.672 INFO:tasks.workunit.client.0.smithi047.stdout: "internally_reserved": 0, 2024-09-18T14:13:54.672 INFO:tasks.workunit.client.0.smithi047.stdout: "allocated": 2306048, 2024-09-18T14:13:54.672 INFO:tasks.workunit.client.0.smithi047.stdout: "data_stored": 1582468, 2024-09-18T14:13:54.672 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed": 0, 2024-09-18T14:13:54.672 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_allocated": 0, 2024-09-18T14:13:54.672 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_original": 0, 2024-09-18T14:13:54.672 INFO:tasks.workunit.client.0.smithi047.stdout: "omap_allocated": 17014, 2024-09-18T14:13:54.672 INFO:tasks.workunit.client.0.smithi047.stdout: "internal_metadata": 914800010 2024-09-18T14:13:54.672 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.672 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_peers": [], 2024-09-18T14:13:54.672 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_trim_queue_len": 0, 2024-09-18T14:13:54.672 INFO:tasks.workunit.client.0.smithi047.stdout: "num_snap_trimming": 0, 2024-09-18T14:13:54.672 INFO:tasks.workunit.client.0.smithi047.stdout: "num_shards_repaired": 0, 2024-09-18T14:13:54.672 INFO:tasks.workunit.client.0.smithi047.stdout: "op_queue_age_hist": { 2024-09-18T14:13:54.673 INFO:tasks.workunit.client.0.smithi047.stdout: "histogram": [], 2024-09-18T14:13:54.673 INFO:tasks.workunit.client.0.smithi047.stdout: "upper_bound": 1 2024-09-18T14:13:54.673 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.673 INFO:tasks.workunit.client.0.smithi047.stdout: "perf_stat": { 2024-09-18T14:13:54.673 INFO:tasks.workunit.client.0.smithi047.stdout: "commit_latency_ms": 0, 2024-09-18T14:13:54.673 INFO:tasks.workunit.client.0.smithi047.stdout: "apply_latency_ms": 0, 2024-09-18T14:13:54.673 INFO:tasks.workunit.client.0.smithi047.stdout: "commit_latency_ns": 0, 2024-09-18T14:13:54.673 INFO:tasks.workunit.client.0.smithi047.stdout: "apply_latency_ns": 0 2024-09-18T14:13:54.673 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.673 INFO:tasks.workunit.client.0.smithi047.stdout: "alerts": [], 2024-09-18T14:13:54.673 INFO:tasks.workunit.client.0.smithi047.stdout: "network_ping_times": [] 2024-09-18T14:13:54.673 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.673 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.673 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_stats": [ 2024-09-18T14:13:54.674 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.674 INFO:tasks.workunit.client.0.smithi047.stdout: "poolid": 1, 2024-09-18T14:13:54.674 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg": 1, 2024-09-18T14:13:54.674 INFO:tasks.workunit.client.0.smithi047.stdout: "stat_sum": { 2024-09-18T14:13:54.674 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes": 48454, 2024-09-18T14:13:54.674 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects": 101, 2024-09-18T14:13:54.674 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_clones": 1, 2024-09-18T14:13:54.674 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_copies": 303, 2024-09-18T14:13:54.674 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing_on_primary": 0, 2024-09-18T14:13:54.674 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing": 0, 2024-09-18T14:13:54.674 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_degraded": 101, 2024-09-18T14:13:54.674 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_misplaced": 0, 2024-09-18T14:13:54.674 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_unfound": 0, 2024-09-18T14:13:54.674 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_dirty": 101, 2024-09-18T14:13:54.674 INFO:tasks.workunit.client.0.smithi047.stdout: "num_whiteouts": 0, 2024-09-18T14:13:54.675 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read": 0, 2024-09-18T14:13:54.675 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read_kb": 0, 2024-09-18T14:13:54.675 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write": 101, 2024-09-18T14:13:54.675 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write_kb": 101, 2024-09-18T14:13:54.675 INFO:tasks.workunit.client.0.smithi047.stdout: "num_scrub_errors": 0, 2024-09-18T14:13:54.675 INFO:tasks.workunit.client.0.smithi047.stdout: "num_shallow_scrub_errors": 0, 2024-09-18T14:13:54.675 INFO:tasks.workunit.client.0.smithi047.stdout: "num_deep_scrub_errors": 0, 2024-09-18T14:13:54.675 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_recovered": 0, 2024-09-18T14:13:54.675 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_recovered": 0, 2024-09-18T14:13:54.675 INFO:tasks.workunit.client.0.smithi047.stdout: "num_keys_recovered": 0, 2024-09-18T14:13:54.675 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_omap": 0, 2024-09-18T14:13:54.675 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_hit_set_archive": 0, 2024-09-18T14:13:54.675 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_hit_set_archive": 0, 2024-09-18T14:13:54.675 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush": 0, 2024-09-18T14:13:54.675 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_kb": 0, 2024-09-18T14:13:54.676 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict": 0, 2024-09-18T14:13:54.676 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_kb": 0, 2024-09-18T14:13:54.676 INFO:tasks.workunit.client.0.smithi047.stdout: "num_promote": 0, 2024-09-18T14:13:54.676 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_high": 0, 2024-09-18T14:13:54.676 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_low": 0, 2024-09-18T14:13:54.676 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_some": 0, 2024-09-18T14:13:54.676 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_full": 0, 2024-09-18T14:13:54.676 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_pinned": 0, 2024-09-18T14:13:54.676 INFO:tasks.workunit.client.0.smithi047.stdout: "num_legacy_snapsets": 0, 2024-09-18T14:13:54.676 INFO:tasks.workunit.client.0.smithi047.stdout: "num_large_omap_objects": 0, 2024-09-18T14:13:54.676 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_manifest": 0, 2024-09-18T14:13:54.676 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_bytes": 0, 2024-09-18T14:13:54.676 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_keys": 0, 2024-09-18T14:13:54.677 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_repaired": 0 2024-09-18T14:13:54.677 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.677 INFO:tasks.workunit.client.0.smithi047.stdout: "store_stats": { 2024-09-18T14:13:54.677 INFO:tasks.workunit.client.0.smithi047.stdout: "total": 0, 2024-09-18T14:13:54.677 INFO:tasks.workunit.client.0.smithi047.stdout: "available": 0, 2024-09-18T14:13:54.677 INFO:tasks.workunit.client.0.smithi047.stdout: "internally_reserved": 0, 2024-09-18T14:13:54.677 INFO:tasks.workunit.client.0.smithi047.stdout: "allocated": 1241088, 2024-09-18T14:13:54.677 INFO:tasks.workunit.client.0.smithi047.stdout: "data_stored": 1241088, 2024-09-18T14:13:54.677 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed": 0, 2024-09-18T14:13:54.677 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_allocated": 0, 2024-09-18T14:13:54.677 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_original": 0, 2024-09-18T14:13:54.677 INFO:tasks.workunit.client.0.smithi047.stdout: "omap_allocated": 0, 2024-09-18T14:13:54.677 INFO:tasks.workunit.client.0.smithi047.stdout: "internal_metadata": 0 2024-09-18T14:13:54.678 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.678 INFO:tasks.workunit.client.0.smithi047.stdout: "log_size": 18, 2024-09-18T14:13:54.678 INFO:tasks.workunit.client.0.smithi047.stdout: "ondisk_log_size": 18, 2024-09-18T14:13:54.678 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 3, 2024-09-18T14:13:54.678 INFO:tasks.workunit.client.0.smithi047.stdout: "acting": 3, 2024-09-18T14:13:54.678 INFO:tasks.workunit.client.0.smithi047.stdout: "num_store_stats": 3 2024-09-18T14:13:54.678 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.678 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:54.678 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_stats": [ 2024-09-18T14:13:54.678 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.678 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 0, 2024-09-18T14:13:54.678 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 188978561035 2024-09-18T14:13:54.678 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.680 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.680 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 1, 2024-09-18T14:13:54.680 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 42949672988 2024-09-18T14:13:54.680 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.680 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.680 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 2, 2024-09-18T14:13:54.680 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 206158430219 2024-09-18T14:13:54.681 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.681 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:54.681 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg_by_state": [ 2024-09-18T14:13:54.681 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.681 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "active+undersized+degraded", 2024-09-18T14:13:54.681 INFO:tasks.workunit.client.0.smithi047.stdout: "num": 1 2024-09-18T14:13:54.681 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.681 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:54.681 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg_by_osd": [ 2024-09-18T14:13:54.681 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.681 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 2147483647, 2024-09-18T14:13:54.681 INFO:tasks.workunit.client.0.smithi047.stdout: "num_primary_pg": 0, 2024-09-18T14:13:54.681 INFO:tasks.workunit.client.0.smithi047.stdout: "num_acting_pg": 1, 2024-09-18T14:13:54.681 INFO:tasks.workunit.client.0.smithi047.stdout: "num_up_not_acting_pg": 0 2024-09-18T14:13:54.682 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.682 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.682 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 2, 2024-09-18T14:13:54.682 INFO:tasks.workunit.client.0.smithi047.stdout: "num_primary_pg": 0, 2024-09-18T14:13:54.682 INFO:tasks.workunit.client.0.smithi047.stdout: "num_acting_pg": 1, 2024-09-18T14:13:54.682 INFO:tasks.workunit.client.0.smithi047.stdout: "num_up_not_acting_pg": 0 2024-09-18T14:13:54.682 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.682 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.682 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 0, 2024-09-18T14:13:54.682 INFO:tasks.workunit.client.0.smithi047.stdout: "num_primary_pg": 1, 2024-09-18T14:13:54.682 INFO:tasks.workunit.client.0.smithi047.stdout: "num_acting_pg": 1, 2024-09-18T14:13:54.682 INFO:tasks.workunit.client.0.smithi047.stdout: "num_up_not_acting_pg": 0 2024-09-18T14:13:54.682 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.682 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.682 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 1, 2024-09-18T14:13:54.683 INFO:tasks.workunit.client.0.smithi047.stdout: "num_primary_pg": 0, 2024-09-18T14:13:54.683 INFO:tasks.workunit.client.0.smithi047.stdout: "num_acting_pg": 0, 2024-09-18T14:13:54.683 INFO:tasks.workunit.client.0.smithi047.stdout: "num_up_not_acting_pg": 0 2024-09-18T14:13:54.683 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.683 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:54.683 INFO:tasks.workunit.client.0.smithi047.stdout: "purged_snaps": [ 2024-09-18T14:13:54.683 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:13:54.683 INFO:tasks.workunit.client.0.smithi047.stdout: "pool": 1, 2024-09-18T14:13:54.683 INFO:tasks.workunit.client.0.smithi047.stdout: "purged_snaps": {} 2024-09-18T14:13:54.683 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.683 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T14:13:54.683 INFO:tasks.workunit.client.0.smithi047.stdout: "servicemap": { 2024-09-18T14:13:54.683 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 5, 2024-09-18T14:13:54.683 INFO:tasks.workunit.client.0.smithi047.stdout: "modified": "2024-09-18T14:12:12.047447+0000", 2024-09-18T14:13:54.683 INFO:tasks.workunit.client.0.smithi047.stdout: "services": {} 2024-09-18T14:13:54.684 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:13:54.684 INFO:tasks.workunit.client.0.smithi047.stdout: "paxos": { 2024-09-18T14:13:54.684 INFO:tasks.workunit.client.0.smithi047.stdout: "first_committed": 1, 2024-09-18T14:13:54.684 INFO:tasks.workunit.client.0.smithi047.stdout: "last_committed": 657, 2024-09-18T14:13:54.684 INFO:tasks.workunit.client.0.smithi047.stdout: "last_pn": 0, 2024-09-18T14:13:54.684 INFO:tasks.workunit.client.0.smithi047.stdout: "accepted_pn": 0 2024-09-18T14:13:54.684 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:13:54.684 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T14:13:54.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-18T14:13:54.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:341: TEST_divergent_ec: echo 'delay recovery' 2024-09-18T14:13:54.685 INFO:tasks.workunit.client.0.smithi047.stdout:delay recovery 2024-09-18T14:13:54.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:342: TEST_divergent_ec: ceph pg dump pgs 2024-09-18T14:13:54.983 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:13:54.983 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 101 0 101 0 0 48454 0 0 18 84 18 active+undersized+degraded 2024-09-18T14:13:14.749002+0000 41'102 49:140 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-18T14:11:14.315267+0000 0'0 2024-09-18T14:11:14.315267+0000 0 0 periodic scrub scheduled @ 2024-09-19T18:00:48.219702+0000 0 0 2024-09-18T14:13:54.983 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:13:54.983 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:13:54.984 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:13:54.993 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2024-09-18T14:13:54.994 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.0 2024-09-18T14:13:54.994 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:13:54.995 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:13:54.995 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:13:54.995 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:13:54.995 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:13:54.995 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32609/ceph-osd.0.asok 2024-09-18T14:13:54.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2024-09-18T14:13:54.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.32609/ceph-osd.0.asok set_recovery_delay 100000 2024-09-18T14:13:55.074 INFO:tasks.workunit.client.0.smithi047.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-18T14:13:55.080 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2024-09-18T14:13:55.081 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.2 2024-09-18T14:13:55.081 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T14:13:55.081 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T14:13:55.082 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:13:55.082 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:13:55.082 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:13:55.082 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32609/ceph-osd.2.asok 2024-09-18T14:13:55.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2024-09-18T14:13:55.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.32609/ceph-osd.2.asok set_recovery_delay 100000 2024-09-18T14:13:55.160 INFO:tasks.workunit.client.0.smithi047.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-18T14:13:55.166 INFO:tasks.workunit.client.0.smithi047.stdout:revive divergent 1 2024-09-18T14:13:55.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:349: TEST_divergent_ec: echo 'revive divergent 1' 2024-09-18T14:13:55.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:350: TEST_divergent_ec: ceph pg dump pgs 2024-09-18T14:13:55.517 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:13:55.517 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 101 0 101 0 0 48454 0 0 18 84 18 active+undersized+degraded 2024-09-18T14:13:14.749002+0000 41'102 49:140 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-18T14:11:14.315267+0000 0'0 2024-09-18T14:11:14.315267+0000 0 0 periodic scrub scheduled @ 2024-09-19T18:00:48.219702+0000 0 0 2024-09-18T14:13:55.517 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:13:55.517 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:13:55.518 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:13:55.527 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:351: TEST_divergent_ec: ceph osd set noup 2024-09-18T14:13:56.292 INFO:tasks.workunit.client.0.smithi047.stderr:noup is set 2024-09-18T14:13:56.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:352: TEST_divergent_ec: activate_osd td/divergent-priors 1 2024-09-18T14:13:56.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-18T14:13:56.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:13:56.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T14:13:56.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:13:56.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-09-18T14:13:56.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T14:13:56.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:13:56.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:13:56.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:13:56.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-18T14:13:56.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-18T14:13:56.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:13:56.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:13:56.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-18T14:13:56.307 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:13:56.307 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:13:56.307 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:13:56.307 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:13:56.307 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:13:56.308 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:13:56.308 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:13:56.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:13:56.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:13:56.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-18T14:13:56.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-18T14:13:56.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:13:56.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:13:56.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:13:56.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:13:56.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:13:56.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:13:56.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-09-18T14:13:56.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T14:13:56.309 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T14:13:56.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:13:56.310 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-09-18T14:13:56.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T14:13:56.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:13:56.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:13:56.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:13:56.337 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:13:56.336+0000 7fd2a880d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:13:56.339 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:13:56.338+0000 7fd2a880d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:13:56.340 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:13:56.339+0000 7fd2a880d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:13:56.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-09-18T14:13:56.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:353: TEST_divergent_ec: sleep 5 2024-09-18T14:13:56.853 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:13:56.853+0000 7fd2a880d440 -1 Falling back to public interface 2024-09-18T14:13:59.499 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:13:59.498+0000 7fd2a880d440 -1 osd.1 29 log_to_monitors true 2024-09-18T14:14:01.698 INFO:tasks.workunit.client.0.smithi047.stdout:delay recovery divergent 2024-09-18T14:14:01.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:355: TEST_divergent_ec: echo 'delay recovery divergent' 2024-09-18T14:14:01.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:356: TEST_divergent_ec: ceph pg dump pgs 2024-09-18T14:14:02.068 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:14:02.068 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 101 0 101 0 0 48454 0 0 18 84 18 active+undersized+degraded 2024-09-18T14:13:14.749002+0000 41'102 51:142 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-18T14:11:14.315267+0000 0'0 2024-09-18T14:11:14.315267+0000 0 0 periodic scrub scheduled @ 2024-09-19T18:00:48.219702+0000 0 0 2024-09-18T14:14:02.068 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:14:02.068 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:14:02.069 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:14:02.080 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: get_asok_path osd.1 2024-09-18T14:14:02.080 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:14:02.080 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:14:02.080 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:14:02.080 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:14:02.080 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:14:02.080 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32609/ceph-osd.1.asok 2024-09-18T14:14:02.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: CEPH_ARGS= 2024-09-18T14:14:02.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.32609/ceph-osd.1.asok set_recovery_delay 100000 2024-09-18T14:14:02.159 INFO:tasks.workunit.client.0.smithi047.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-18T14:14:02.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:359: TEST_divergent_ec: ceph osd unset noup 2024-09-18T14:14:02.847 INFO:tasks.workunit.client.0.smithi047.stderr:noup is unset 2024-09-18T14:14:02.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:361: TEST_divergent_ec: wait_for_osd up 0 2024-09-18T14:14:02.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:14:02.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:14:02.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:14:02.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:14:02.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:14:02.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:14:02.858 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:14:02.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:14:02.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:14:03.229 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 44 up_thru 48 down_at 43 last_clean_interval [36,41) [v2:127.0.0.1:6802/85784,v1:127.0.0.1:6803/85784] [v2:127.0.0.1:6804/85784,v1:127.0.0.1:6805/85784] exists,up 50835fc1-8964-4c12-b447-46f109dbb8e4 2024-09-18T14:14:03.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:14:03.231 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:14:03.231 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:14:03.231 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:362: TEST_divergent_ec: wait_for_osd up 1 2024-09-18T14:14:03.231 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:14:03.231 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:14:03.231 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:14:03.231 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:14:03.231 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:14:03.231 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:14:03.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:14:03.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:14:03.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:14:03.625 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 54 up_thru 54 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/88552,v1:127.0.0.1:6819/88552] [v2:127.0.0.1:6820/88552,v1:127.0.0.1:6821/88552] exists,up f4d67a04-7819-45b5-bc8c-088b7d24eed7 2024-09-18T14:14:03.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:14:03.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:14:03.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:14:03.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:363: TEST_divergent_ec: wait_for_osd up 2 2024-09-18T14:14:03.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:14:03.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T14:14:03.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:14:03.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:14:03.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:14:03.627 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:14:03.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:14:03.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:14:03.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:14:04.031 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 48 up_thru 0 down_at 47 last_clean_interval [38,41) [v2:127.0.0.1:6810/86664,v1:127.0.0.1:6811/86664] [v2:127.0.0.1:6812/86664,v1:127.0.0.1:6813/86664] exists,up 2d53c532-9e0f-4778-a2d9-277af85698d4 2024-09-18T14:14:04.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:14:04.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:14:04.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:14:04.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:365: TEST_divergent_ec: ceph pg dump pgs 2024-09-18T14:14:04.392 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:14:04.393 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 101 0 101 0 0 48454 0 0 18 84 18 active+undersized+degraded 2024-09-18T14:13:14.749002+0000 41'102 51:142 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-18T14:11:14.315267+0000 0'0 2024-09-18T14:11:14.315267+0000 0 0 periodic scrub scheduled @ 2024-09-19T18:00:48.219702+0000 0 0 2024-09-18T14:14:04.393 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:14:04.393 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:14:04.393 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:14:04.403 INFO:tasks.workunit.client.0.smithi047.stdout:wait for peering 2024-09-18T14:14:04.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:366: TEST_divergent_ec: echo 'wait for peering' 2024-09-18T14:14:04.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:367: TEST_divergent_ec: ceph pg dump pgs 2024-09-18T14:14:04.765 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:14:04.766 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 101 0 101 0 0 48454 0 0 18 84 18 active+undersized+degraded 2024-09-18T14:13:14.749002+0000 41'102 51:142 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-18T14:11:14.315267+0000 0'0 2024-09-18T14:11:14.315267+0000 0 0 periodic scrub scheduled @ 2024-09-19T18:00:48.219702+0000 0 0 2024-09-18T14:14:04.766 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:14:04.766 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:14:04.766 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:14:04.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:368: TEST_divergent_ec: rados -p test put foo /etc/fstab 2024-09-18T14:14:04.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:370: TEST_divergent_ec: echo 'killing divergent 1' 2024-09-18T14:14:04.906 INFO:tasks.workunit.client.0.smithi047.stdout:killing divergent 1 2024-09-18T14:14:04.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:371: TEST_divergent_ec: ceph pg dump pgs 2024-09-18T14:14:05.259 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:14:05.259 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 101 0 101 0 0 48454 0 0 18 84 18 active+undersized+degraded 2024-09-18T14:13:14.749002+0000 41'102 51:142 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-18T14:11:14.315267+0000 0'0 2024-09-18T14:11:14.315267+0000 0 0 periodic scrub scheduled @ 2024-09-19T18:00:48.219702+0000 0 0 2024-09-18T14:14:05.259 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:14:05.259 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:14:05.259 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:14:05.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:372: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.1 2024-09-18T14:14:05.269 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:14:05.269 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:14:05.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:14:05.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:14:05.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:14:05.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:14:05.375 INFO:tasks.workunit.client.0.smithi047.stdout:reviving divergent 1 2024-09-18T14:14:05.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:374: TEST_divergent_ec: echo 'reviving divergent 1' 2024-09-18T14:14:05.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:375: TEST_divergent_ec: ceph pg dump pgs 2024-09-18T14:14:05.737 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:14:05.737 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 101 0 101 0 0 48454 0 0 18 84 18 active+undersized+degraded 2024-09-18T14:13:14.749002+0000 41'102 51:142 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-18T14:11:14.315267+0000 0'0 2024-09-18T14:11:14.315267+0000 0 0 periodic scrub scheduled @ 2024-09-19T18:00:48.219702+0000 0 0 2024-09-18T14:14:05.737 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:14:05.737 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:14:05.737 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:14:05.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:376: TEST_divergent_ec: activate_osd td/divergent-priors 1 2024-09-18T14:14:05.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-18T14:14:05.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:14:05.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T14:14:05.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:14:05.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-09-18T14:14:05.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T14:14:05.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:14:05.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:14:05.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:14:05.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-18T14:14:05.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-18T14:14:05.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:14:05.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:14:05.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-18T14:14:05.751 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:14:05.751 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:14:05.751 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:14:05.751 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:14:05.751 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:14:05.751 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:14:05.751 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:14:05.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' 2024-09-18T14:14:05.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:14:05.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-18T14:14:05.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-18T14:14:05.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:14:05.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:14:05.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:14:05.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:14:05.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:14:05.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:14:05.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-09-18T14:14:05.752 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T14:14:05.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T14:14:05.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d7330bd7-8062-424a-bbd8-8c587903c456 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32609/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:14:05.753 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-09-18T14:14:05.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T14:14:05.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:14:05.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:14:05.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:14:05.780 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:14:05.780+0000 7f0a98030440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:14:05.782 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:14:05.782+0000 7f0a98030440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:14:05.783 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:14:05.783+0000 7f0a98030440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:14:06.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T14:14:06.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:14:06.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:14:06.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:14:06.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:14:06.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:14:06.170 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:14:06.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:14:06.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:14:06.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:14:06.302 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:14:06.302+0000 7f0a98030440 -1 Falling back to public interface 2024-09-18T14:14:06.541 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 54 up_thru 54 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/88552,v1:127.0.0.1:6819/88552] [v2:127.0.0.1:6820/88552,v1:127.0.0.1:6821/88552] exists,up f4d67a04-7819-45b5-bc8c-088b7d24eed7 2024-09-18T14:14:06.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:14:06.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:14:06.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:14:06.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:378: TEST_divergent_ec: sleep 20 2024-09-18T14:14:09.256 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:14:09.255+0000 7f0a98030440 -1 osd.1 55 log_to_monitors true 2024-09-18T14:14:26.542 INFO:tasks.workunit.client.0.smithi047.stdout:allowing recovery 2024-09-18T14:14:26.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:380: TEST_divergent_ec: echo 'allowing recovery' 2024-09-18T14:14:26.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:381: TEST_divergent_ec: ceph pg dump pgs 2024-09-18T14:14:26.909 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:14:26.909 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 102 0 0 0 0 48937 0 0 13 90 13 active+clean 2024-09-18T14:14:10.755550+0000 55'103 59:153 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:11:14.315267+0000 0'0 2024-09-18T14:11:14.315267+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:58:46.853492+0000 0 0 2024-09-18T14:14:26.909 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:14:26.909 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:14:26.909 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:14:26.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-09-18T14:14:26.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-18T14:14:27.026 INFO:tasks.workunit.client.0.smithi047.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-18T14:14:27.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-09-18T14:14:27.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-18T14:14:27.143 INFO:tasks.workunit.client.0.smithi047.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-18T14:14:27.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-09-18T14:14:27.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-18T14:14:27.259 INFO:tasks.workunit.client.0.smithi047.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-18T14:14:27.266 INFO:tasks.workunit.client.0.smithi047.stdout:reading divergent objects 2024-09-18T14:14:27.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:388: TEST_divergent_ec: echo 'reading divergent objects' 2024-09-18T14:14:27.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:389: TEST_divergent_ec: ceph pg dump pgs 2024-09-18T14:14:27.619 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:14:27.619 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 102 0 0 0 0 48937 0 0 13 90 13 active+clean 2024-09-18T14:14:10.755550+0000 55'103 59:153 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T14:11:14.315267+0000 0'0 2024-09-18T14:11:14.315267+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:58:46.853492+0000 0 0 2024-09-18T14:14:27.619 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:14:27.619 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:14:27.619 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:14:27.630 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: expr 2 + 2 2024-09-18T14:14:27.631 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: seq 1 4 2024-09-18T14:14:27.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-18T14:14:27.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_1 td/divergent-priors/existing 2024-09-18T14:14:27.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-18T14:14:27.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_2 td/divergent-priors/existing 2024-09-18T14:14:27.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-18T14:14:27.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_3 td/divergent-priors/existing 2024-09-18T14:14:27.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-18T14:14:27.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_4 td/divergent-priors/existing 2024-09-18T14:14:27.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:394: TEST_divergent_ec: rm -f td/divergent-priors/existing 2024-09-18T14:14:27.776 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2024-09-18T14:14:27.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-18T14:14:27.782 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:14:03.432+0000 7fd28b7c7640 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:1 entries: 29'103 (28'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2024-09-18T14:11:51.702793+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2024-09-18T14:14:27.783 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:14:03.432+0000 7fd28b7c7640 20 _merge_object_divergent_entries: keeping 29'103 (28'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2024-09-18T14:11:51.702793+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2024-09-18T14:14:27.783 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:14:03.432+0000 7fd28b7c7640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-18T14:14:27.783 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:14:03.432+0000 7fd28b7c7640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version: 28'99 first_divergent_update: 29'103 last_divergent_update: 29'103 2024-09-18T14:14:27.783 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:14:03.432+0000 7fd28b7c7640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 has no more recent entries in log 2024-09-18T14:14:27.783 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:14:03.432+0000 7fd28b7c7640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version or op type indicates creation, deleting 2024-09-18T14:14:27.783 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:14:03.432+0000 7fd28b7c7640 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:head entries: 29'104 (28'99) modify 1:7a80c0d5:::existing_99:head by client.4653.0:1 2024-09-18T14:11:59.149958+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-18T14:14:27.783 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:14:03.432+0000 7fd28b7c7640 20 _merge_object_divergent_entries: keeping 29'104 (28'99) modify 1:7a80c0d5:::existing_99:head by client.4653.0:1 2024-09-18T14:11:59.149958+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-18T14:14:27.783 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:14:03.432+0000 7fd28b7c7640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-18T14:14:27.783 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:14:03.432+0000 7fd28b7c7640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head prior_version: 28'99 first_divergent_update: 29'104 last_divergent_update: 29'104 2024-09-18T14:14:27.783 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:14:03.432+0000 7fd28b7c7640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head has no more recent entries in log 2024-09-18T14:14:27.783 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:14:03.432+0000 7fd28b7c7640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head must be rolled back or recovered, attempting to rollback 2024-09-18T14:14:27.783 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:14:03.432+0000 7fd28b7c7640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head olog_can_rollback_to: 29'104 2024-09-18T14:14:27.783 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:14:03.432+0000 7fd28b7c7640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot rollback 29'104 (28'99) modify 1:7a80c0d5:::existing_99:head by client.4653.0:1 2024-09-18T14:11:59.149958+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-18T14:14:27.783 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:14:03.432+0000 7fd28b7c7640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot roll back, removing and adding to missing 2024-09-18T14:14:27.784 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:14:03.432+0000 7fd28b7c7640 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4644.0:1 2024-09-18T14:11:57.949557+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0,28'102 (28'101) modify 1:c156033e:::existing_100:head by client.4650.0:1 2024-09-18T14:11:58.955687+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-18T14:14:27.784 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:14:03.432+0000 7fd28b7c7640 20 _merge_object_divergent_entries: keeping 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4644.0:1 2024-09-18T14:11:57.949557+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-18T14:14:27.784 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:14:03.432+0000 7fd28b7c7640 20 _merge_object_divergent_entries: keeping 28'102 (28'101) modify 1:c156033e:::existing_100:head by client.4650.0:1 2024-09-18T14:11:58.955687+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-18T14:14:27.784 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:14:03.432+0000 7fd28b7c7640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-18T14:14:27.784 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:14:03.432+0000 7fd28b7c7640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 28'100 first_divergent_update: 28'101 last_divergent_update: 28'102 2024-09-18T14:14:27.784 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:14:03.432+0000 7fd28b7c7640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2024-09-18T14:14:27.784 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:14:03.432+0000 7fd28b7c7640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2024-09-18T14:14:27.785 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:14:03.432+0000 7fd28b7c7640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 29'104 2024-09-18T14:14:27.785 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:14:03.432+0000 7fd28b7c7640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 28'102 (28'101) modify 1:c156033e:::existing_100:head by client.4650.0:1 2024-09-18T14:11:58.955687+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-18T14:14:27.785 INFO:tasks.workunit.client.0.smithi047.stdout:td/divergent-priors/osd.1.log:2024-09-18T14:14:03.432+0000 7fd28b7c7640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2024-09-18T14:14:27.785 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2024-09-18T14:14:27.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: grep -q '_merge_object_divergent_entries.*rolled back' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-18T14:14:27.791 INFO:tasks.workunit.client.0.smithi047.stdout:success 2024-09-18T14:14:27.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:410: TEST_divergent_ec: echo success 2024-09-18T14:14:27.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:412: TEST_divergent_ec: delete_pool test 2024-09-18T14:14:27.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-18T14:14:27.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-18T14:14:28.300 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test' does not exist 2024-09-18T14:14:28.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:413: TEST_divergent_ec: kill_daemons td/divergent-priors 2024-09-18T14:14:28.311 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:14:28.311 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:14:28.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:14:28.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:14:28.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:14:33.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:14:33.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-18T14:14:33.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-18T14:14:33.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:14:33.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-18T14:14:33.628 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:14:33.628 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:14:33.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:14:33.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:14:33.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:14:33.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:14:33.634 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:14:33.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:14:33.636 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:14:33.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:14:33.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:14:33.637 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:14:33.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:14:33.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:14:33.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:14:33.639 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:14:33.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:14:33.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:14:33.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-18T14:14:33.667 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:14:33.667 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:14:33.667 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:14:33.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.32609 2024-09-18T14:14:33.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:14:33.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:14:33.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T14:14:33.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/divergent-priors 0 2024-09-18T14:14:33.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-18T14:14:33.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T14:14:33.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-18T14:14:33.669 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:14:33.669 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:14:33.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:14:33.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:14:33.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:14:33.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:14:33.671 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:14:33.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:14:33.673 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:14:33.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:14:33.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:14:33.674 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:14:33.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:14:33.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:14:33.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:14:33.676 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:14:33.676 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:14:33.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T14:14:33.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-18T14:14:33.679 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:14:33.679 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:14:33.679 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32609 2024-09-18T14:14:33.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.32609 2024-09-18T14:14:33.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:14:33.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:14:33.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T14:14:33.682 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T14:14:33.682 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T14:14:33.713 INFO:tasks.workunit:Running workunit osd/ec-error-rollforward.sh... 2024-09-18T14:14:33.713 DEBUG:teuthology.orchestra.run.smithi047:workunit test osd/ec-error-rollforward.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh 2024-09-18T14:14:33.778 INFO:tasks.workunit.client.0.smithi047.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T14:14:33.783 INFO:tasks.workunit.client.0.smithi047.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T14:14:33.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T14:14:33.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T14:14:33.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T14:14:33.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T14:14:33.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T14:14:33.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T14:14:33.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T14:14:33.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T14:14:33.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/ec-error-rollforward 2024-09-18T14:14:33.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:6: run: local dir=td/ec-error-rollforward 2024-09-18T14:14:33.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:7: run: shift 2024-09-18T14:14:33.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: export CEPH_MON=127.0.0.1:7132 2024-09-18T14:14:33.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: CEPH_MON=127.0.0.1:7132 2024-09-18T14:14:33.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:11: run: export CEPH_ARGS 2024-09-18T14:14:33.784 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: uuidgen 2024-09-18T14:14:33.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: CEPH_ARGS+='--fsid=dfb91648-020e-4f79-9bcd-96d66c09eeb2 --auth-supported=none ' 2024-09-18T14:14:33.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:13: run: CEPH_ARGS+='--mon-host=127.0.0.1:7132 ' 2024-09-18T14:14:33.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: export margin=10 2024-09-18T14:14:33.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: margin=10 2024-09-18T14:14:33.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: export objects=200 2024-09-18T14:14:33.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: objects=200 2024-09-18T14:14:33.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: export poolname=test 2024-09-18T14:14:33.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: poolname=test 2024-09-18T14:14:33.786 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: set 2024-09-18T14:14:33.786 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-18T14:14:33.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: local funcs=TEST_ec_error_rollforward 2024-09-18T14:14:33.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:19: run: for func in $funcs 2024-09-18T14:14:33.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:20: run: setup td/ec-error-rollforward 2024-09-18T14:14:33.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ec-error-rollforward 2024-09-18T14:14:33.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ec-error-rollforward 2024-09-18T14:14:33.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2024-09-18T14:14:33.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:14:33.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2024-09-18T14:14:33.788 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:14:33.788 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:14:33.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:14:33.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:14:33.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:14:33.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:14:33.789 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:14:33.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:14:33.789 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:14:33.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:14:33.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:14:33.790 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:14:33.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:14:33.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:14:33.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:14:33.792 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:14:33.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:14:33.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:14:33.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2024-09-18T14:14:33.795 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:14:33.795 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:14:33.795 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90101 2024-09-18T14:14:33.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.90101 2024-09-18T14:14:33.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:14:33.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:14:33.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ec-error-rollforward 2024-09-18T14:14:33.797 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T14:14:33.797 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:14:33.797 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90101 2024-09-18T14:14:33.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.90101 2024-09-18T14:14:33.799 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T14:14:33.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T14:14:33.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T14:14:33.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T14:14:33.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ec-error-rollforward 1' TERM HUP INT 2024-09-18T14:14:33.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:21: run: TEST_ec_error_rollforward td/ec-error-rollforward 2024-09-18T14:14:33.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:27: TEST_ec_error_rollforward: local dir=td/ec-error-rollforward 2024-09-18T14:14:33.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:28: TEST_ec_error_rollforward: run_mon td/ec-error-rollforward a 2024-09-18T14:14:33.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ec-error-rollforward 2024-09-18T14:14:33.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T14:14:33.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T14:14:33.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T14:14:33.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ec-error-rollforward/a 2024-09-18T14:14:33.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ec-error-rollforward/a --run-dir=td/ec-error-rollforward 2024-09-18T14:14:34.157 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T14:14:34.157 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:14:34.157 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:14:34.157 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:14:34.157 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:14:34.157 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90101 2024-09-18T14:14:34.157 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90101/$cluster-$name.asok' 2024-09-18T14:14:34.157 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ec-error-rollforward/a '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.90101/$cluster-$name.asok' --mon-cluster-log-file=td/ec-error-rollforward/log --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T14:14:34.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T14:14:34.418 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T14:14:34.418 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:14:34.418 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:14:34.418 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T14:14:34.418 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T14:14:34.418 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:14:34.418 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:14:34.418 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:14:34.419 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:14:34.419 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:14:34.419 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90101 2024-09-18T14:14:34.419 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.90101/ceph-mon.a.asok 2024-09-18T14:14:34.419 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:14:34.420 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.90101/ceph-mon.a.asok config get fsid 2024-09-18T14:14:34.506 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T14:14:34.506 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:14:34.506 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:14:34.506 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T14:14:34.506 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T14:14:34.506 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:14:34.506 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:14:34.506 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:14:34.506 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:14:34.506 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:14:34.507 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90101 2024-09-18T14:14:34.507 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.90101/ceph-mon.a.asok 2024-09-18T14:14:34.507 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:14:34.507 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.90101/ceph-mon.a.asok config get mon_host 2024-09-18T14:14:34.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:29: TEST_ec_error_rollforward: run_mgr td/ec-error-rollforward x 2024-09-18T14:14:34.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ec-error-rollforward 2024-09-18T14:14:34.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T14:14:34.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T14:14:34.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T14:14:34.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ec-error-rollforward/x 2024-09-18T14:14:34.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T14:14:35.431 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T14:14:35.431 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:14:35.431 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:14:35.431 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:14:35.431 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:14:35.431 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90101 2024-09-18T14:14:35.431 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90101/$cluster-$name.asok' 2024-09-18T14:14:35.431 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:14:35.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ec-error-rollforward/x '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.90101/$cluster-$name.asok' --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:14:35.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:30: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 0 2024-09-18T14:14:35.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-09-18T14:14:35.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:14:35.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T14:14:35.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:14:35.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/0 2024-09-18T14:14:35.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=dfb91648-020e-4f79-9bcd-96d66c09eeb2 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-18T14:14:35.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:14:35.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:14:35.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:14:35.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2024-09-18T14:14:35.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2024-09-18T14:14:35.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:14:35.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:14:35.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-18T14:14:35.458 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:14:35.458 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:14:35.458 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:14:35.458 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:14:35.458 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:14:35.458 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90101 2024-09-18T14:14:35.458 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90101/$cluster-$name.asok' 2024-09-18T14:14:35.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90101/$cluster-$name.asok' 2024-09-18T14:14:35.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:14:35.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:14:35.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:14:35.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-18T14:14:35.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-18T14:14:35.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:14:35.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:14:35.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:14:35.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:14:35.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:14:35.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:14:35.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/0 2024-09-18T14:14:35.460 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:14:35.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=79086c0e-ae91-44f5-a2e8-1fbd63d8f95e 2024-09-18T14:14:35.460 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 79086c0e-ae91-44f5-a2e8-1fbd63d8f95e 2024-09-18T14:14:35.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 79086c0e-ae91-44f5-a2e8-1fbd63d8f95e' 2024-09-18T14:14:35.461 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:14:35.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBL4OpmnBIUHBAAZOBuHMb8NjbW7qsOg1dUbg== 2024-09-18T14:14:35.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBL4OpmnBIUHBAAZOBuHMb8NjbW7qsOg1dUbg=="}' 2024-09-18T14:14:35.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 79086c0e-ae91-44f5-a2e8-1fbd63d8f95e -i td/ec-error-rollforward/0/new.json 2024-09-18T14:14:35.764 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:14:35.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/0/new.json 2024-09-18T14:14:35.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=dfb91648-020e-4f79-9bcd-96d66c09eeb2 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90101/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBL4OpmnBIUHBAAZOBuHMb8NjbW7qsOg1dUbg== --osd-uuid 79086c0e-ae91-44f5-a2e8-1fbd63d8f95e 2024-09-18T14:14:35.799 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:14:35.797+0000 7fa6b6a1a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:14:35.809 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:14:35.808+0000 7fa6b6a1a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:14:35.809 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:14:35.809+0000 7fa6b6a1a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:14:35.810 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:14:35.809+0000 7fa6b6a1a440 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2024-09-18T14:14:35.810 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:14:35.810+0000 7fa6b6a1a440 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2024-09-18T14:14:35.810 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:14:35.810+0000 7fa6b6a1a440 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2024-09-18T14:14:35.849 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:14:35.848+0000 7fa6b6a1a440 -1 bluestore(td/ec-error-rollforward/0) _read_fsid unparsable uuid 2024-09-18T14:14:41.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/0/keyring 2024-09-18T14:14:41.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:14:41.230 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T14:14:41.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T14:14:41.231 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:14:41.733 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T14:14:41.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T14:14:41.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=dfb91648-020e-4f79-9bcd-96d66c09eeb2 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90101/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:14:41.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:14:41.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:14:41.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:14:41.755 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:14:41.754+0000 7f91ad80e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:14:41.781 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:14:41.781+0000 7f91ad80e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:14:41.782 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:14:41.782+0000 7f91ad80e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:14:42.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T14:14:42.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:14:42.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:14:42.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:14:42.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:14:42.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:14:42.178 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:14:42.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:14:42.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:14:42.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:14:42.305 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:14:42.305+0000 7f91ad80e440 -1 Falling back to public interface 2024-09-18T14:14:42.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:14:43.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:14:43.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:14:43.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:14:43.588 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:14:43.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:14:43.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:14:43.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:14:44.818 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:14:44.817+0000 7f91ad80e440 -1 osd.0 0 log_to_monitors true 2024-09-18T14:14:44.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:14:44.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:14:44.959 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:14:44.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:14:44.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:14:44.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:14:45.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:14:46.420 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:14:46.420 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:14:46.420 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:14:46.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:14:46.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:14:46.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:14:46.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:14:47.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:14:47.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:14:47.794 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:14:47.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:14:47.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:14:47.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:14:48.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:14:49.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:14:49.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:14:49.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:14:49.181 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:14:49.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:14:49.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:14:49.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:14:50.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:14:50.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:14:50.703 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:14:50.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:14:50.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:14:50.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:14:51.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:14:52.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:14:52.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:14:52.109 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:14:52.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:14:52.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:14:52.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:14:52.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:14:53.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:14:53.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:14:53.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:14:53.511 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:14:53.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:14:53.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:14:53.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:14:54.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:14:54.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:14:54.909 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:14:54.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:14:54.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:14:54.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:14:55.299 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:14:56.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:14:56.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:14:56.301 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:14:56.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:14:56.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:14:56.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:14:57.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:14:58.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:14:58.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:14:58.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:14:58.213 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:14:58.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:14:58.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:14:59.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:15:00.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:15:00.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:15:00.291 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:15:00.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:15:00.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:15:00.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:15:01.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:15:02.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:15:02.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:15:02.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:15:02.061 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:15:02.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:15:02.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:15:02.861 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:15:03.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:15:03.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:15:03.863 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:15:03.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:15:03.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:15:03.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:15:04.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:15:05.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:15:05.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:15:05.726 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:15:05.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:15:05.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:15:05.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:15:06.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:15:07.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:15:07.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:15:07.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:15:07.462 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:15:07.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:15:07.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:15:07.873 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/91192,v1:127.0.0.1:6803/91192] [v2:127.0.0.1:6804/91192,v1:127.0.0.1:6805/91192] exists,up 79086c0e-ae91-44f5-a2e8-1fbd63d8f95e 2024-09-18T14:15:07.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:15:07.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:15:07.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:15:07.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:31: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 1 2024-09-18T14:15:07.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-09-18T14:15:07.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:15:07.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T14:15:07.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:15:07.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/1 2024-09-18T14:15:07.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=dfb91648-020e-4f79-9bcd-96d66c09eeb2 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-18T14:15:07.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:15:07.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:15:07.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:15:07.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2024-09-18T14:15:07.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2024-09-18T14:15:07.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:15:07.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:15:07.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-18T14:15:07.878 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:15:07.878 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:15:07.878 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:15:07.879 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:15:07.879 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:15:07.879 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90101 2024-09-18T14:15:07.879 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90101/$cluster-$name.asok' 2024-09-18T14:15:07.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90101/$cluster-$name.asok' 2024-09-18T14:15:07.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:15:07.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:15:07.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:15:07.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-18T14:15:07.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-18T14:15:07.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:15:07.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:15:07.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:15:07.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:15:07.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:15:07.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:15:07.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/1 2024-09-18T14:15:07.880 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:15:07.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f4836393-f170-421f-91ac-56ac8b10c3b4 2024-09-18T14:15:07.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f4836393-f170-421f-91ac-56ac8b10c3b4' 2024-09-18T14:15:07.880 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 f4836393-f170-421f-91ac-56ac8b10c3b4 2024-09-18T14:15:07.880 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:15:07.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBr4Opm3TQPNRAAdERvXwBHIACqgLHeYA6hBQ== 2024-09-18T14:15:07.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBr4Opm3TQPNRAAdERvXwBHIACqgLHeYA6hBQ=="}' 2024-09-18T14:15:07.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f4836393-f170-421f-91ac-56ac8b10c3b4 -i td/ec-error-rollforward/1/new.json 2024-09-18T14:15:08.381 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:15:08.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/1/new.json 2024-09-18T14:15:08.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=dfb91648-020e-4f79-9bcd-96d66c09eeb2 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90101/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBr4Opm3TQPNRAAdERvXwBHIACqgLHeYA6hBQ== --osd-uuid f4836393-f170-421f-91ac-56ac8b10c3b4 2024-09-18T14:15:08.416 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:15:08.415+0000 7f3a1d923440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:15:08.418 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:15:08.417+0000 7f3a1d923440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:15:08.419 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:15:08.418+0000 7f3a1d923440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:15:08.419 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:15:08.419+0000 7f3a1d923440 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2024-09-18T14:15:08.419 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:15:08.419+0000 7f3a1d923440 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2024-09-18T14:15:08.419 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:15:08.419+0000 7f3a1d923440 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2024-09-18T14:15:08.471 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:15:08.471+0000 7f3a1d923440 -1 bluestore(td/ec-error-rollforward/1) _read_fsid unparsable uuid 2024-09-18T14:15:14.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/1/keyring 2024-09-18T14:15:14.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:15:14.026 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T14:15:14.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T14:15:14.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:15:14.532 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T14:15:14.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T14:15:14.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=dfb91648-020e-4f79-9bcd-96d66c09eeb2 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90101/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:15:14.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:15:14.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:15:14.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:15:14.553 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:15:14.553+0000 7efd45721440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:15:14.555 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:15:14.555+0000 7efd45721440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:15:14.556 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:15:14.556+0000 7efd45721440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:15:14.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T14:15:14.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:15:14.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:15:14.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:15:14.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:15:14.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:15:14.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:15:14.958 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:15:14.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:15:14.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:15:15.073 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:15:15.073+0000 7efd45721440 -1 Falling back to public interface 2024-09-18T14:15:15.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:15:16.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:15:16.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:15:16.349 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:15:16.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:15:16.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:15:16.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:15:16.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:15:17.691 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:15:17.690+0000 7efd45721440 -1 osd.1 0 log_to_monitors true 2024-09-18T14:15:17.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:15:17.725 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:15:17.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:15:17.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:15:17.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:15:17.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:15:18.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:15:19.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:15:19.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:15:19.352 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:15:19.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:15:19.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:15:19.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:15:19.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:15:20.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:15:20.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:15:20.753 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:15:20.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:15:20.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:15:20.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:15:21.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:15:22.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:15:22.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:15:22.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:15:22.162 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:15:22.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:15:22.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:15:22.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:15:23.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:15:23.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:15:23.585 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:15:23.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:15:23.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:15:23.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:15:24.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:15:25.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:15:25.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:15:25.042 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:15:25.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:15:25.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:15:25.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:15:25.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:15:26.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:15:26.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:15:26.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:15:26.558 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:15:26.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:15:26.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:15:26.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:15:27.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:15:27.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:15:27.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:15:27.963 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:15:27.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:15:27.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:15:28.469 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:15:29.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:15:29.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:15:29.471 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:15:29.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:15:29.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:15:29.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:15:30.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:15:31.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:15:31.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:15:31.273 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:15:31.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:15:31.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:15:31.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:15:31.746 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:15:32.746 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:15:32.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:15:32.747 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:15:32.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:15:32.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:15:32.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:15:33.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:15:34.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:15:34.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:15:34.410 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:15:34.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:15:34.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:15:34.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:15:35.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:15:36.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:15:36.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:15:36.124 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:15:36.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:15:36.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:15:36.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:15:37.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:15:38.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:15:38.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:15:38.036 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:15:38.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:15:38.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:15:38.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:15:38.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:15:39.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:15:39.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:15:39.853 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:15:39.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:15:39.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:15:39.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:15:40.266 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/92922,v1:127.0.0.1:6811/92922] [v2:127.0.0.1:6812/92922,v1:127.0.0.1:6813/92922] exists,up f4836393-f170-421f-91ac-56ac8b10c3b4 2024-09-18T14:15:40.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:15:40.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:15:40.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:15:40.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:32: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 2 2024-09-18T14:15:40.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-09-18T14:15:40.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:15:40.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T14:15:40.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:15:40.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/2 2024-09-18T14:15:40.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=dfb91648-020e-4f79-9bcd-96d66c09eeb2 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-18T14:15:40.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:15:40.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:15:40.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:15:40.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2024-09-18T14:15:40.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2024-09-18T14:15:40.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:15:40.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:15:40.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-18T14:15:40.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:15:40.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:15:40.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:15:40.272 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:15:40.272 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:15:40.272 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90101 2024-09-18T14:15:40.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90101/$cluster-$name.asok' 2024-09-18T14:15:40.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90101/$cluster-$name.asok' 2024-09-18T14:15:40.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:15:40.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:15:40.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:15:40.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-18T14:15:40.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-18T14:15:40.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:15:40.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:15:40.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:15:40.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:15:40.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:15:40.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:15:40.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/2 2024-09-18T14:15:40.274 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:15:40.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=16cb8f48-7b74-43b4-a380-a84baae6fa15 2024-09-18T14:15:40.274 INFO:tasks.workunit.client.0.smithi047.stdout:add osd2 16cb8f48-7b74-43b4-a380-a84baae6fa15 2024-09-18T14:15:40.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 16cb8f48-7b74-43b4-a380-a84baae6fa15' 2024-09-18T14:15:40.275 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:15:40.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCM4OpmW7LjEBAAPaaT3Z8YjxipP7mhwERSAQ== 2024-09-18T14:15:40.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCM4OpmW7LjEBAAPaaT3Z8YjxipP7mhwERSAQ=="}' 2024-09-18T14:15:40.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 16cb8f48-7b74-43b4-a380-a84baae6fa15 -i td/ec-error-rollforward/2/new.json 2024-09-18T14:15:40.766 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:15:40.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/2/new.json 2024-09-18T14:15:40.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=dfb91648-020e-4f79-9bcd-96d66c09eeb2 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90101/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCM4OpmW7LjEBAAPaaT3Z8YjxipP7mhwERSAQ== --osd-uuid 16cb8f48-7b74-43b4-a380-a84baae6fa15 2024-09-18T14:15:40.801 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:15:40.800+0000 7f26af551440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:15:40.803 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:15:40.802+0000 7f26af551440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:15:40.804 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:15:40.803+0000 7f26af551440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:15:40.804 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:15:40.803+0000 7f26af551440 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2024-09-18T14:15:40.804 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:15:40.804+0000 7f26af551440 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2024-09-18T14:15:40.804 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:15:40.804+0000 7f26af551440 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2024-09-18T14:15:40.869 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:15:40.870+0000 7f26af551440 -1 bluestore(td/ec-error-rollforward/2) _read_fsid unparsable uuid 2024-09-18T14:15:46.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/2/keyring 2024-09-18T14:15:46.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:15:46.729 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd2 key to auth repository 2024-09-18T14:15:46.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T14:15:46.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:15:47.233 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T14:15:47.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T14:15:47.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=dfb91648-020e-4f79-9bcd-96d66c09eeb2 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90101/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:15:47.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:15:47.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:15:47.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:15:47.254 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:15:47.253+0000 7fd4a011f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:15:47.256 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:15:47.255+0000 7fd4a011f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:15:47.257 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:15:47.257+0000 7fd4a011f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:15:47.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T14:15:47.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:15:47.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T14:15:47.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:15:47.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:15:47.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:15:47.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:15:47.636 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:15:47.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:15:47.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:15:47.771 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:15:47.771+0000 7fd4a011f440 -1 Falling back to public interface 2024-09-18T14:15:48.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:15:49.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:15:49.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:15:49.038 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:15:49.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:15:49.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:15:49.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:15:49.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:15:50.284 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:15:50.284+0000 7fd4a011f440 -1 osd.2 0 log_to_monitors true 2024-09-18T14:15:50.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:15:50.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:15:50.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:15:50.439 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:15:50.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:15:50.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:15:50.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:15:51.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:15:51.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:15:51.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:15:51.963 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:15:51.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:15:51.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:15:52.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:15:53.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:15:53.365 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:15:53.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:15:53.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:15:53.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:15:53.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:15:53.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:15:54.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:15:54.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:15:54.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:15:54.777 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:15:54.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:15:54.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:15:55.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:15:56.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:15:56.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:15:56.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:15:56.164 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:15:56.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:15:56.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:15:56.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:15:57.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:15:57.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:15:57.592 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:15:57.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:15:57.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:15:57.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:15:58.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:15:59.095 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:15:59.095 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:15:59.095 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:15:59.096 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:15:59.096 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:15:59.096 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:15:59.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:16:00.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:16:00.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:16:00.510 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:16:00.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:16:00.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:16:00.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:16:00.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:16:01.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:16:01.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:16:01.885 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:16:01.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:16:01.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:16:01.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:16:02.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:16:03.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:16:03.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:16:03.728 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:16:03.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:16:03.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:16:03.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:16:04.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:16:05.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:16:05.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:16:05.526 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:16:05.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:16:05.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:16:05.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:16:06.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:16:07.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:16:07.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:16:07.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:16:07.352 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:16:07.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:16:07.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:16:07.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:16:08.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:16:08.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:16:08.979 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:16:08.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:16:08.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:16:08.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:16:09.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:16:10.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:16:10.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:16:10.689 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:16:10.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:16:10.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:16:10.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:16:11.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:16:12.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:16:12.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:16:12.378 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:16:12.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:16:12.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:16:12.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:16:12.961 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/94651,v1:127.0.0.1:6819/94651] [v2:127.0.0.1:6820/94651,v1:127.0.0.1:6821/94651] exists,up 16cb8f48-7b74-43b4-a380-a84baae6fa15 2024-09-18T14:16:12.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:16:12.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:16:12.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:16:12.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:33: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 3 2024-09-18T14:16:12.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-09-18T14:16:12.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:16:12.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-18T14:16:12.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:16:12.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/3 2024-09-18T14:16:12.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=dfb91648-020e-4f79-9bcd-96d66c09eeb2 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-18T14:16:12.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:16:12.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:16:12.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:16:12.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2024-09-18T14:16:12.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2024-09-18T14:16:12.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:16:12.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:16:12.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-18T14:16:12.967 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:16:12.967 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:16:12.967 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:16:12.967 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:16:12.967 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:16:12.967 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90101 2024-09-18T14:16:12.967 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90101/$cluster-$name.asok' 2024-09-18T14:16:12.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90101/$cluster-$name.asok' 2024-09-18T14:16:12.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:16:12.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:16:12.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:16:12.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-18T14:16:12.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-18T14:16:12.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:16:12.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:16:12.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:16:12.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:16:12.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:16:12.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:16:12.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/3 2024-09-18T14:16:12.968 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:16:12.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=66643cfd-8b90-4fb9-a00a-2df620999e6d 2024-09-18T14:16:12.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 66643cfd-8b90-4fb9-a00a-2df620999e6d' 2024-09-18T14:16:12.969 INFO:tasks.workunit.client.0.smithi047.stdout:add osd3 66643cfd-8b90-4fb9-a00a-2df620999e6d 2024-09-18T14:16:12.969 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:16:12.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCs4OpmlsdVOhAAyeY0jJBLqTz/qGetAT3RXg== 2024-09-18T14:16:12.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCs4OpmlsdVOhAAyeY0jJBLqTz/qGetAT3RXg=="}' 2024-09-18T14:16:12.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 66643cfd-8b90-4fb9-a00a-2df620999e6d -i td/ec-error-rollforward/3/new.json 2024-09-18T14:16:13.434 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:16:13.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/3/new.json 2024-09-18T14:16:13.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=dfb91648-020e-4f79-9bcd-96d66c09eeb2 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90101/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCs4OpmlsdVOhAAyeY0jJBLqTz/qGetAT3RXg== --osd-uuid 66643cfd-8b90-4fb9-a00a-2df620999e6d 2024-09-18T14:16:13.468 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:16:13.467+0000 7f25b9b29440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:16:13.495 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:16:13.495+0000 7f25b9b29440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:16:13.497 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:16:13.496+0000 7f25b9b29440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:16:13.497 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:16:13.496+0000 7f25b9b29440 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2024-09-18T14:16:13.497 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:16:13.496+0000 7f25b9b29440 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2024-09-18T14:16:13.497 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:16:13.496+0000 7f25b9b29440 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2024-09-18T14:16:13.599 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:16:13.598+0000 7f25b9b29440 -1 bluestore(td/ec-error-rollforward/3) _read_fsid unparsable uuid 2024-09-18T14:16:19.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/3/keyring 2024-09-18T14:16:19.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:16:19.214 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd3 key to auth repository 2024-09-18T14:16:19.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-18T14:16:19.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:16:19.741 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.3 2024-09-18T14:16:19.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-18T14:16:19.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=dfb91648-020e-4f79-9bcd-96d66c09eeb2 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90101/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:16:19.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:16:19.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:16:19.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:16:19.763 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:16:19.762+0000 7fdbfc008440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:16:19.779 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:16:19.779+0000 7fdbfc008440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:16:19.780 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:16:19.780+0000 7fdbfc008440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:16:20.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-18T14:16:20.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:16:20.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T14:16:20.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:16:20.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:16:20.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:16:20.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:16:20.143 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:16:20.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:16:20.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:16:20.299 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:16:20.299+0000 7fdbfc008440 -1 Falling back to public interface 2024-09-18T14:16:20.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:16:21.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:16:21.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:16:21.504 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:16:21.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:16:21.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:16:21.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:16:21.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:16:22.800 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:16:22.799+0000 7fdbfc008440 -1 osd.3 0 log_to_monitors true 2024-09-18T14:16:22.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:16:22.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:16:22.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:16:22.901 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:16:22.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:16:22.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:16:23.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:16:24.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:16:24.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:16:24.427 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:16:24.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:16:24.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:16:24.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:16:24.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:16:25.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:16:25.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:16:25.874 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:16:25.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:16:25.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:16:25.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:16:26.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:16:27.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:16:27.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:16:27.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:16:27.347 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:16:27.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:16:27.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:16:27.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:16:28.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:16:28.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:16:28.754 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:16:28.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:16:28.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:16:28.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:16:29.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:16:30.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:16:30.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:16:30.151 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:16:30.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:16:30.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:16:30.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:16:30.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:16:31.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:16:31.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:16:31.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:16:31.568 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:16:31.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:16:31.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:16:32.079 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:16:33.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:16:33.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:16:33.081 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:16:33.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:16:33.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:16:33.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:16:33.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:16:34.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:16:34.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:16:34.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:16:34.573 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:16:34.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:16:34.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:16:35.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:16:36.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:16:36.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:16:36.459 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:16:36.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:16:36.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:16:36.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:16:37.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:16:38.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:16:38.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:16:38.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:16:38.262 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:16:38.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:16:38.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:16:39.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:16:40.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:16:40.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:16:40.091 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:16:40.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:16:40.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:16:40.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:16:40.946 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:16:41.947 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:16:41.948 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:16:41.948 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:16:41.948 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:16:41.949 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:16:41.949 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:16:42.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:16:43.714 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:16:43.714 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:16:43.714 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:16:43.715 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:16:43.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:16:43.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:16:44.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:16:45.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:16:45.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:16:45.369 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:16:45.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:16:45.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:16:45.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:16:45.780 INFO:tasks.workunit.client.0.smithi047.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/96380,v1:127.0.0.1:6827/96380] [v2:127.0.0.1:6828/96380,v1:127.0.0.1:6829/96380] exists,up 66643cfd-8b90-4fb9-a00a-2df620999e6d 2024-09-18T14:16:45.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:16:45.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:16:45.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:16:45.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:35: TEST_ec_error_rollforward: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2024-09-18T14:16:46.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:36: TEST_ec_error_rollforward: ceph osd pool create ec 1 1 erasure ec-profile 2024-09-18T14:16:47.418 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'ec' already exists 2024-09-18T14:16:47.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:38: TEST_ec_error_rollforward: rados -p ec put foo /etc/passwd 2024-09-18T14:16:49.008 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: cat td/ec-error-rollforward/osd.2.pid 2024-09-18T14:16:49.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: kill -STOP 94651 2024-09-18T14:16:49.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:43: TEST_ec_error_rollforward: pids=97336 2024-09-18T14:16:49.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:44: TEST_ec_error_rollforward: sleep 1 2024-09-18T14:16:49.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:42: TEST_ec_error_rollforward: rados -p ec rm foo 2024-09-18T14:16:50.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:46: TEST_ec_error_rollforward: pids+=' 97357' 2024-09-18T14:16:50.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:45: TEST_ec_error_rollforward: rados -p ec rm a 2024-09-18T14:16:50.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:48: TEST_ec_error_rollforward: pids+=' 97358' 2024-09-18T14:16:50.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:47: TEST_ec_error_rollforward: rados -p ec rm b 2024-09-18T14:16:50.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:50: TEST_ec_error_rollforward: pids+=' 97359' 2024-09-18T14:16:50.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:51: TEST_ec_error_rollforward: sleep 1 2024-09-18T14:16:50.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:49: TEST_ec_error_rollforward: rados -p ec rm c 2024-09-18T14:16:51.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:54: TEST_ec_error_rollforward: kill_daemons td/ec-error-rollforward KILL osd 2024-09-18T14:16:51.014 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:16:51.014 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:16:51.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:16:51.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:16:51.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:16:51.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:16:51.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:55: TEST_ec_error_rollforward: kill 97336 97357 97358 97359 2024-09-18T14:16:51.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:56: TEST_ec_error_rollforward: wait 2024-09-18T14:16:51.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:58: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 0 2024-09-18T14:16:51.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-09-18T14:16:51.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:16:51.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T14:16:51.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:16:51.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/0 2024-09-18T14:16:51.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=dfb91648-020e-4f79-9bcd-96d66c09eeb2 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-18T14:16:51.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:16:51.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:16:51.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:16:51.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2024-09-18T14:16:51.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2024-09-18T14:16:51.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:16:51.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:16:51.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-18T14:16:51.129 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:16:51.129 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:16:51.129 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:16:51.129 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:16:51.129 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:16:51.129 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90101 2024-09-18T14:16:51.129 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90101/$cluster-$name.asok' 2024-09-18T14:16:51.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90101/$cluster-$name.asok' 2024-09-18T14:16:51.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:16:51.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-18T14:16:51.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-18T14:16:51.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:16:51.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:16:51.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:16:51.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:16:51.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:16:51.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:16:51.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/0 2024-09-18T14:16:51.131 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T14:16:51.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T14:16:51.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=dfb91648-020e-4f79-9bcd-96d66c09eeb2 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90101/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:16:51.132 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/0/whoami 2024-09-18T14:16:51.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T14:16:51.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:16:51.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:16:51.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:16:51.151 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:16:51.151+0000 7f76e0129440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:16:51.153 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:16:51.153+0000 7f76e0129440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:16:51.154 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:16:51.154+0000 7f76e0129440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:16:51.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T14:16:51.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:16:51.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:16:51.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:16:51.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:16:51.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:16:51.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:16:51.544 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:16:51.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:16:51.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:16:51.672 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:16:51.672+0000 7f76e0129440 -1 Falling back to public interface 2024-09-18T14:16:51.922 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/91192,v1:127.0.0.1:6803/91192] [v2:127.0.0.1:6804/91192,v1:127.0.0.1:6805/91192] exists,up 79086c0e-ae91-44f5-a2e8-1fbd63d8f95e 2024-09-18T14:16:51.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:16:51.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:16:51.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:16:51.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:59: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 1 2024-09-18T14:16:51.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-09-18T14:16:51.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:16:51.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T14:16:51.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:16:51.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/1 2024-09-18T14:16:51.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=dfb91648-020e-4f79-9bcd-96d66c09eeb2 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-18T14:16:51.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:16:51.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:16:51.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:16:51.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2024-09-18T14:16:51.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2024-09-18T14:16:51.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:16:51.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:16:51.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-18T14:16:51.927 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:16:51.927 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:16:51.927 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:16:51.927 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:16:51.928 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:16:51.928 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90101 2024-09-18T14:16:51.928 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90101/$cluster-$name.asok' 2024-09-18T14:16:51.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90101/$cluster-$name.asok' 2024-09-18T14:16:51.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:16:51.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-18T14:16:51.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-18T14:16:51.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:16:51.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:16:51.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:16:51.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:16:51.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:16:51.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:16:51.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/1 2024-09-18T14:16:51.929 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T14:16:51.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T14:16:51.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=dfb91648-020e-4f79-9bcd-96d66c09eeb2 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90101/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:16:51.930 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/1/whoami 2024-09-18T14:16:51.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T14:16:51.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:16:51.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:16:51.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:16:51.945 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:16:51.945+0000 7f9e8e413440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:16:51.950 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:16:51.949+0000 7f9e8e413440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:16:51.951 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:16:51.951+0000 7f9e8e413440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:16:52.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T14:16:52.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:16:52.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:16:52.292 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:16:52.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:16:52.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:16:52.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:16:52.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:16:52.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:16:52.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:16:52.468 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:16:52.467+0000 7f9e8e413440 -1 Falling back to public interface 2024-09-18T14:16:52.696 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 10 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/92922,v1:127.0.0.1:6811/92922] [v2:127.0.0.1:6812/92922,v1:127.0.0.1:6813/92922] exists,up f4836393-f170-421f-91ac-56ac8b10c3b4 2024-09-18T14:16:52.699 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:16:52.699 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:16:52.699 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:16:52.699 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:60: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 2 2024-09-18T14:16:52.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-09-18T14:16:52.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:16:52.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T14:16:52.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:16:52.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/2 2024-09-18T14:16:52.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=dfb91648-020e-4f79-9bcd-96d66c09eeb2 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-18T14:16:52.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:16:52.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:16:52.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:16:52.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2024-09-18T14:16:52.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2024-09-18T14:16:52.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:16:52.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:16:52.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-18T14:16:52.701 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:16:52.701 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:16:52.701 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:16:52.701 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:16:52.701 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:16:52.701 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90101 2024-09-18T14:16:52.701 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90101/$cluster-$name.asok' 2024-09-18T14:16:52.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90101/$cluster-$name.asok' 2024-09-18T14:16:52.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:16:52.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-18T14:16:52.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-18T14:16:52.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:16:52.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:16:52.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:16:52.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:16:52.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:16:52.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:16:52.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/2 2024-09-18T14:16:52.702 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T14:16:52.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T14:16:52.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=dfb91648-020e-4f79-9bcd-96d66c09eeb2 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90101/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:16:52.703 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/2/whoami 2024-09-18T14:16:52.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T14:16:52.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:16:52.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:16:52.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:16:52.719 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:16:52.719+0000 7fc06b32a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:16:52.721 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:16:52.721+0000 7fc06b32a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:16:52.722 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:16:52.722+0000 7fc06b32a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:16:53.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T14:16:53.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:16:53.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T14:16:53.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:16:53.099 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:16:53.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:16:53.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:16:53.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:16:53.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:16:53.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:16:53.237 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:16:53.236+0000 7fc06b32a440 -1 Falling back to public interface 2024-09-18T14:16:53.490 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/94651,v1:127.0.0.1:6819/94651] [v2:127.0.0.1:6820/94651,v1:127.0.0.1:6821/94651] exists,up 16cb8f48-7b74-43b4-a380-a84baae6fa15 2024-09-18T14:16:53.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:16:53.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:16:53.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:16:53.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:61: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 3 2024-09-18T14:16:53.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-09-18T14:16:53.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:16:53.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-18T14:16:53.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:16:53.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/3 2024-09-18T14:16:53.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=dfb91648-020e-4f79-9bcd-96d66c09eeb2 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-18T14:16:53.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:16:53.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:16:53.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:16:53.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2024-09-18T14:16:53.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2024-09-18T14:16:53.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:16:53.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:16:53.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-18T14:16:53.495 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:16:53.495 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:16:53.495 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:16:53.495 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:16:53.495 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:16:53.495 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90101 2024-09-18T14:16:53.495 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90101/$cluster-$name.asok' 2024-09-18T14:16:53.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90101/$cluster-$name.asok' 2024-09-18T14:16:53.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:16:53.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-18T14:16:53.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-18T14:16:53.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:16:53.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:16:53.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:16:53.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:16:53.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:16:53.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:16:53.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/3 2024-09-18T14:16:53.496 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.3 2024-09-18T14:16:53.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-18T14:16:53.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=dfb91648-020e-4f79-9bcd-96d66c09eeb2 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90101/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:16:53.497 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/3/whoami 2024-09-18T14:16:53.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-18T14:16:53.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:16:53.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:16:53.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:16:53.514 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:16:53.513+0000 7fb4b5c39440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:16:53.516 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:16:53.516+0000 7fb4b5c39440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:16:53.517 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:16:53.517+0000 7fb4b5c39440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:16:53.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-18T14:16:53.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:16:53.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T14:16:53.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:16:53.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:16:53.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:16:53.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:16:53.864 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:16:53.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:16:53.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:16:54.055 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:16:54.055+0000 7fb4b5c39440 -1 Falling back to public interface 2024-09-18T14:16:54.372 INFO:tasks.workunit.client.0.smithi047.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/96380,v1:127.0.0.1:6827/96380] [v2:127.0.0.1:6828/96380,v1:127.0.0.1:6829/96380] exists,up 66643cfd-8b90-4fb9-a00a-2df620999e6d 2024-09-18T14:16:54.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:16:54.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:16:54.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:16:54.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:63: TEST_ec_error_rollforward: wait_for_clean 2024-09-18T14:16:54.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:16:54.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:16:54.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:16:54.374 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:16:54.374 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:16:54.374 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:16:54.374 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:16:54.374 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:16:54.375 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:16:54.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T14:16:54.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:16:54.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:16:54.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:16:54.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:16:54.461 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:16:54.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:16:54.890 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T14:16:54.890 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T14:16:54.890 INFO:tasks.workunit.client.0.smithi047.stderr:3' 2024-09-18T14:16:54.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:16:54.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:16:54.891 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:16:55.265 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:16:55.264+0000 7f76e0129440 -1 osd.0 26 log_to_monitors true 2024-09-18T14:16:56.589 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:16:56.589+0000 7f9e8e413440 -1 osd.1 26 log_to_monitors true 2024-09-18T14:16:57.472 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:16:57.471+0000 7fc06b32a440 -1 osd.2 25 log_to_monitors true 2024-09-18T14:16:58.157 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:16:58.156+0000 7fb4b5c39440 -1 osd.3 26 log_to_monitors true 2024-09-18T14:17:00.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=124554051587 2024-09-18T14:17:00.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 124554051587 2024-09-18T14:17:00.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-124554051587' 2024-09-18T14:17:00.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:17:00.088 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:17:00.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=137438953475 2024-09-18T14:17:00.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 137438953475 2024-09-18T14:17:00.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-124554051587 1-137438953475' 2024-09-18T14:17:00.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:17:00.201 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:17:00.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=146028888067 2024-09-18T14:17:00.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 146028888067 2024-09-18T14:17:00.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-124554051587 1-137438953475 2-146028888067' 2024-09-18T14:17:00.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:17:00.308 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T14:17:00.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=154618822659 2024-09-18T14:17:00.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 154618822659 2024-09-18T14:17:00.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-124554051587 1-137438953475 2-146028888067 3-154618822659' 2024-09-18T14:17:00.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:17:00.414 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-124554051587 2024-09-18T14:17:00.414 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:17:00.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:17:00.416 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-124554051587 2024-09-18T14:17:00.416 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:17:00.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=124554051587 2024-09-18T14:17:00.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 124554051587' 2024-09-18T14:17:00.417 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 124554051587 2024-09-18T14:17:00.417 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:17:00.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 124554051587 2024-09-18T14:17:00.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:17:01.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:17:01.816 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:17:02.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051588 -lt 124554051587 2024-09-18T14:17:02.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:17:02.234 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-137438953475 2024-09-18T14:17:02.234 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:17:02.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:17:02.235 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-137438953475 2024-09-18T14:17:02.235 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:17:02.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=137438953475 2024-09-18T14:17:02.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 137438953475' 2024-09-18T14:17:02.237 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 137438953475 2024-09-18T14:17:02.237 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:17:02.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 137438953475 -lt 137438953475 2024-09-18T14:17:02.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:17:02.598 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-146028888067 2024-09-18T14:17:02.598 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:17:02.599 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:17:02.600 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-146028888067 2024-09-18T14:17:02.600 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:17:02.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=146028888067 2024-09-18T14:17:02.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 146028888067' 2024-09-18T14:17:02.601 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 146028888067 2024-09-18T14:17:02.601 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:17:02.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 146028888067 -lt 146028888067 2024-09-18T14:17:02.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:17:02.965 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-154618822659 2024-09-18T14:17:02.965 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:17:02.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T14:17:02.966 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-154618822659 2024-09-18T14:17:02.967 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:17:02.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=154618822659 2024-09-18T14:17:02.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 154618822659' 2024-09-18T14:17:02.968 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 154618822659 2024-09-18T14:17:02.968 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T14:17:03.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 154618822659 -lt 154618822659 2024-09-18T14:17:03.387 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:17:03.387 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:17:03.388 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:17:03.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T14:17:03.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:17:03.887 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:17:03.887 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:17:03.888 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:17:03.888 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:17:03.888 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:17:03.888 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:17:04.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T14:17:04.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:17:04.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:17:04.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:17:04.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T14:17:04.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:17:04.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:17:04.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:22: run: teardown td/ec-error-rollforward 2024-09-18T14:17:04.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2024-09-18T14:17:04.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:17:04.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2024-09-18T14:17:04.806 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:17:04.806 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:17:04.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:17:04.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:17:04.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:17:04.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:17:04.914 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:17:04.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:17:04.915 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:17:04.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:17:04.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:17:04.916 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:17:04.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:17:04.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:17:04.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:17:04.918 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:17:04.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:17:04.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:17:04.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2024-09-18T14:17:04.937 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:17:04.937 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:17:04.937 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90101 2024-09-18T14:17:04.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.90101 2024-09-18T14:17:04.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:17:04.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:17:04.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T14:17:04.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/ec-error-rollforward 0 2024-09-18T14:17:04.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2024-09-18T14:17:04.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T14:17:04.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2024-09-18T14:17:04.939 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:17:04.939 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:17:04.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:17:04.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:17:04.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:17:04.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:17:04.940 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:17:04.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:17:04.941 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:17:04.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:17:04.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:17:04.943 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:17:04.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:17:04.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:17:04.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:17:04.944 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:17:04.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:17:04.946 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T14:17:04.946 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2024-09-18T14:17:04.947 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:17:04.947 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:17:04.947 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90101 2024-09-18T14:17:04.947 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.90101 2024-09-18T14:17:04.948 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:17:04.949 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:17:04.949 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T14:17:04.949 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T14:17:04.949 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T14:17:05.014 INFO:tasks.workunit:Running workunit osd/osd-bench.sh... 2024-09-18T14:17:05.015 DEBUG:teuthology.orchestra.run.smithi047:workunit test osd/osd-bench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh 2024-09-18T14:17:05.077 INFO:tasks.workunit.client.0.smithi047.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T14:17:05.082 INFO:tasks.workunit.client.0.smithi047.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T14:17:05.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T14:17:05.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T14:17:05.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T14:17:05.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T14:17:05.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T14:17:05.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T14:17:05.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T14:17:05.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T14:17:05.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/osd-bench 2024-09-18T14:17:05.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:22: run: local dir=td/osd-bench 2024-09-18T14:17:05.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:23: run: shift 2024-09-18T14:17:05.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: export CEPH_MON=127.0.0.1:7106 2024-09-18T14:17:05.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: CEPH_MON=127.0.0.1:7106 2024-09-18T14:17:05.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:26: run: export CEPH_ARGS 2024-09-18T14:17:05.083 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: uuidgen 2024-09-18T14:17:05.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: CEPH_ARGS+='--fsid=c5445131-2652-4f32-ae28-bede72de4615 --auth-supported=none ' 2024-09-18T14:17:05.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7106 ' 2024-09-18T14:17:05.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:29: run: CEPH_ARGS+='--debug-bluestore 20 ' 2024-09-18T14:17:05.084 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: set 2024-09-18T14:17:05.084 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-18T14:17:05.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: local funcs=TEST_bench 2024-09-18T14:17:05.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:32: run: for func in $funcs 2024-09-18T14:17:05.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:33: run: setup td/osd-bench 2024-09-18T14:17:05.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bench 2024-09-18T14:17:05.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bench 2024-09-18T14:17:05.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2024-09-18T14:17:05.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:17:05.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2024-09-18T14:17:05.086 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:17:05.086 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:17:05.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:17:05.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:17:05.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:17:05.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:17:05.086 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:17:05.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:17:05.087 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:17:05.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:17:05.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:17:05.088 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:17:05.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:17:05.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:17:05.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:17:05.090 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:17:05.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:17:05.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:17:05.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2024-09-18T14:17:05.092 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:17:05.092 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:17:05.092 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99929 2024-09-18T14:17:05.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.99929 2024-09-18T14:17:05.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:17:05.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:17:05.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bench 2024-09-18T14:17:05.094 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T14:17:05.094 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:17:05.095 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99929 2024-09-18T14:17:05.095 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.99929 2024-09-18T14:17:05.096 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T14:17:05.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T14:17:05.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T14:17:05.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T14:17:05.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bench 1' TERM HUP INT 2024-09-18T14:17:05.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:34: run: TEST_bench td/osd-bench 2024-09-18T14:17:05.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:40: TEST_bench: local dir=td/osd-bench 2024-09-18T14:17:05.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:42: TEST_bench: run_mon td/osd-bench a 2024-09-18T14:17:05.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bench 2024-09-18T14:17:05.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T14:17:05.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T14:17:05.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T14:17:05.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bench/a 2024-09-18T14:17:05.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bench/a --run-dir=td/osd-bench 2024-09-18T14:17:05.455 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T14:17:05.455 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:17:05.456 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:17:05.456 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:17:05.456 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:17:05.456 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99929 2024-09-18T14:17:05.456 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.99929/$cluster-$name.asok' 2024-09-18T14:17:05.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bench/a '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.99929/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bench/log --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T14:17:05.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T14:17:05.709 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T14:17:05.709 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:17:05.709 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:17:05.709 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T14:17:05.709 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T14:17:05.709 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:17:05.709 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:17:05.709 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:17:05.710 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:17:05.710 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:17:05.710 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99929 2024-09-18T14:17:05.710 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.99929/ceph-mon.a.asok 2024-09-18T14:17:05.710 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:17:05.710 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.99929/ceph-mon.a.asok config get fsid 2024-09-18T14:17:05.791 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T14:17:05.791 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:17:05.792 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:17:05.792 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T14:17:05.792 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T14:17:05.792 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:17:05.792 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:17:05.792 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:17:05.792 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:17:05.792 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:17:05.792 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99929 2024-09-18T14:17:05.792 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.99929/ceph-mon.a.asok 2024-09-18T14:17:05.792 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:17:05.792 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.99929/ceph-mon.a.asok config get mon_host 2024-09-18T14:17:05.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:43: TEST_bench: run_mgr td/osd-bench x 2024-09-18T14:17:05.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bench 2024-09-18T14:17:05.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T14:17:05.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T14:17:05.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T14:17:05.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bench/x 2024-09-18T14:17:05.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T14:17:06.703 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T14:17:06.704 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:17:06.704 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:17:06.704 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:17:06.704 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:17:06.704 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99929 2024-09-18T14:17:06.704 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.99929/$cluster-$name.asok' 2024-09-18T14:17:06.704 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:17:06.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bench/x '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.99929/$cluster-$name.asok' --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:17:06.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:44: TEST_bench: run_osd td/osd-bench 0 2024-09-18T14:17:06.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bench 2024-09-18T14:17:06.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:17:06.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T14:17:06.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:17:06.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bench/0 2024-09-18T14:17:06.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c5445131-2652-4f32-ae28-bede72de4615 --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 ' 2024-09-18T14:17:06.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:17:06.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:17:06.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:17:06.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bench/0' 2024-09-18T14:17:06.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bench/0/journal' 2024-09-18T14:17:06.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:17:06.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:17:06.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bench' 2024-09-18T14:17:06.732 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:17:06.732 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:17:06.733 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:17:06.733 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:17:06.733 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:17:06.733 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99929 2024-09-18T14:17:06.733 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.99929/$cluster-$name.asok' 2024-09-18T14:17:06.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.99929/$cluster-$name.asok' 2024-09-18T14:17:06.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:17:06.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:17:06.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:17:06.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bench/$name.log' 2024-09-18T14:17:06.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bench/$name.pid' 2024-09-18T14:17:06.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:17:06.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:17:06.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:17:06.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:17:06.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:17:06.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:17:06.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bench/0 2024-09-18T14:17:06.734 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:17:06.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cf2e4397-e2cd-484d-9d35-89033ff1ef13 2024-09-18T14:17:06.735 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 cf2e4397-e2cd-484d-9d35-89033ff1ef13 2024-09-18T14:17:06.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 cf2e4397-e2cd-484d-9d35-89033ff1ef13' 2024-09-18T14:17:06.735 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:17:06.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDi4OpmuqFtLBAAXBZhceaXis5KUg7zecnrJw== 2024-09-18T14:17:06.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDi4OpmuqFtLBAAXBZhceaXis5KUg7zecnrJw=="}' 2024-09-18T14:17:06.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cf2e4397-e2cd-484d-9d35-89033ff1ef13 -i td/osd-bench/0/new.json 2024-09-18T14:17:07.047 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:17:07.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bench/0/new.json 2024-09-18T14:17:07.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=c5445131-2652-4f32-ae28-bede72de4615 --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.99929/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDi4OpmuqFtLBAAXBZhceaXis5KUg7zecnrJw== --osd-uuid cf2e4397-e2cd-484d-9d35-89033ff1ef13 2024-09-18T14:17:07.080 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:17:07.080+0000 7f543f525440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:17:07.091 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:17:07.090+0000 7f543f525440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:17:07.092 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:17:07.092+0000 7f543f525440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:17:07.092 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:17:07.092+0000 7f543f525440 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2024-09-18T14:17:07.092 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:17:07.092+0000 7f543f525440 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2024-09-18T14:17:07.093 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:17:07.092+0000 7f543f525440 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2024-09-18T14:17:07.131 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:17:07.131+0000 7f543f525440 -1 bluestore(td/osd-bench/0) _read_fsid unparsable uuid 2024-09-18T14:17:12.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bench/0/keyring 2024-09-18T14:17:12.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:17:12.533 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T14:17:12.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T14:17:12.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bench/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:17:13.060 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T14:17:13.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T14:17:13.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=c5445131-2652-4f32-ae28-bede72de4615 --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.99929/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:17:13.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:17:13.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:17:13.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:17:13.082 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:17:13.081+0000 7fa3d0a6e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:17:13.098 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:17:13.098+0000 7fa3d0a6e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:17:13.100 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:17:13.099+0000 7fa3d0a6e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:17:13.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T14:17:13.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:17:13.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:17:13.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:17:13.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:17:13.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:17:13.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:17:13.503 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:17:13.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:17:13.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:17:13.629 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:17:13.629+0000 7fa3d0a6e440 -1 Falling back to public interface 2024-09-18T14:17:13.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:17:14.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:17:14.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:17:14.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:17:14.923 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:17:14.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:17:14.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:17:15.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:17:16.112 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:17:16.111+0000 7fa3d0a6e440 -1 osd.0 0 log_to_monitors true 2024-09-18T14:17:16.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:17:16.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:17:16.349 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:17:16.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:17:16.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:17:16.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:17:16.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:17:17.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:17:17.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:17:17.795 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:17:17.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:17:17.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:17:17.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:17:18.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:17:19.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:17:19.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:17:19.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:17:19.168 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:17:19.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:17:19.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:17:19.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:17:20.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:17:20.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:17:20.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:17:20.567 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:17:20.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:17:20.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:17:20.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:17:21.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:17:21.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:17:21.945 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:17:21.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:17:21.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:17:21.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:17:22.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:17:23.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:17:23.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:17:23.347 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:17:23.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:17:23.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:17:23.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:17:23.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:17:24.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:17:24.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:17:24.789 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:17:24.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:17:24.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:17:24.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:17:25.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:17:26.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:17:26.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:17:26.189 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:17:26.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:17:26.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:17:26.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:17:26.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:17:27.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:17:27.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:17:27.574 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:17:27.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:17:27.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:17:27.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:17:28.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:17:29.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:17:29.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:17:29.531 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:17:29.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:17:29.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:17:29.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:17:30.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:17:31.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:17:31.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:17:31.008 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:17:31.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:17:31.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:17:31.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:17:31.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:17:32.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:17:32.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:17:32.786 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:17:32.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:17:32.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:17:32.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:17:33.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:17:34.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:17:34.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:17:34.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:17:34.739 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:17:34.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:17:34.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:17:35.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:17:36.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:17:36.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:17:36.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:17:36.367 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:17:36.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:17:36.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:17:36.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:17:37.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:17:37.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:17:37.899 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:17:37.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:17:37.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:17:37.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:17:38.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:17:39.299 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:17:39.299 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:17:39.299 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:17:39.300 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T14:17:39.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:17:39.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:17:39.728 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/101020,v1:127.0.0.1:6803/101020] [v2:127.0.0.1:6804/101020,v1:127.0.0.1:6805/101020] exists,up cf2e4397-e2cd-484d-9d35-89033ff1ef13 2024-09-18T14:17:39.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:17:39.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:17:39.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:17:39.729 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: CEPH_ARGS= 2024-09-18T14:17:39.729 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: ceph-conf --show-config-value osd_bench_small_size_max_iops 2024-09-18T14:17:39.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: local osd_bench_small_size_max_iops=100 2024-09-18T14:17:39.739 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: CEPH_ARGS= 2024-09-18T14:17:39.739 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: ceph-conf --show-config-value osd_bench_large_size_max_throughput 2024-09-18T14:17:39.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: local osd_bench_large_size_max_throughput=104857600 2024-09-18T14:17:39.750 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: CEPH_ARGS= 2024-09-18T14:17:39.750 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: ceph-conf --show-config-value osd_bench_max_block_size 2024-09-18T14:17:39.761 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: local osd_bench_max_block_size=67108864 2024-09-18T14:17:39.761 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: CEPH_ARGS= 2024-09-18T14:17:39.761 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: ceph-conf --show-config-value osd_bench_duration 2024-09-18T14:17:39.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: local osd_bench_duration=30 2024-09-18T14:17:39.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:58: TEST_bench: expect_failure td/osd-bench osd_bench_max_block_size ceph tell osd.0 bench 1024 67108865 2024-09-18T14:17:39.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: expect_failure: local dir=td/osd-bench 2024-09-18T14:17:39.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: expect_failure: shift 2024-09-18T14:17:39.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: expect_failure: local expected=osd_bench_max_block_size 2024-09-18T14:17:39.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: expect_failure: shift 2024-09-18T14:17:39.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: expect_failure: local success 2024-09-18T14:17:39.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1939: expect_failure: ceph tell osd.0 bench 1024 67108865 2024-09-18T14:17:39.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1942: expect_failure: success=false 2024-09-18T14:17:39.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: false 2024-09-18T14:17:39.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: grep --quiet osd_bench_max_block_size td/osd-bench/out 2024-09-18T14:17:39.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: expect_failure: return 0 2024-09-18T14:17:39.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:64: TEST_bench: local bsize=1024 2024-09-18T14:17:39.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:65: TEST_bench: local max_count=3072000 2024-09-18T14:17:39.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:66: TEST_bench: expect_failure td/osd-bench bench_small_size_max_iops ceph tell osd.0 bench 3072001 1024 2024-09-18T14:17:39.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: expect_failure: local dir=td/osd-bench 2024-09-18T14:17:39.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: expect_failure: shift 2024-09-18T14:17:39.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: expect_failure: local expected=bench_small_size_max_iops 2024-09-18T14:17:39.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: expect_failure: shift 2024-09-18T14:17:39.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: expect_failure: local success 2024-09-18T14:17:39.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1939: expect_failure: ceph tell osd.0 bench 3072001 1024 2024-09-18T14:17:39.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1942: expect_failure: success=false 2024-09-18T14:17:39.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: false 2024-09-18T14:17:39.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: grep --quiet bench_small_size_max_iops td/osd-bench/out 2024-09-18T14:17:39.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: expect_failure: return 0 2024-09-18T14:17:39.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:72: TEST_bench: local bsize=1048577 2024-09-18T14:17:39.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:73: TEST_bench: local max_count=3145728000 2024-09-18T14:17:39.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:74: TEST_bench: expect_failure td/osd-bench osd_bench_large_size_max_throughput ceph tell osd.0 bench 3145728001 1048577 2024-09-18T14:17:39.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: expect_failure: local dir=td/osd-bench 2024-09-18T14:17:39.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: expect_failure: shift 2024-09-18T14:17:39.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: expect_failure: local expected=osd_bench_large_size_max_throughput 2024-09-18T14:17:39.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: expect_failure: shift 2024-09-18T14:17:39.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: expect_failure: local success 2024-09-18T14:17:39.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1939: expect_failure: ceph tell osd.0 bench 3145728001 1048577 2024-09-18T14:17:40.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1942: expect_failure: success=false 2024-09-18T14:17:40.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: false 2024-09-18T14:17:40.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: grep --quiet osd_bench_large_size_max_throughput td/osd-bench/out 2024-09-18T14:17:40.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: expect_failure: return 0 2024-09-18T14:17:40.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:80: TEST_bench: ceph tell osd.0 bench 2024-09-18T14:18:28.573 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T14:18:28.574 INFO:tasks.workunit.client.0.smithi047.stdout: "bytes_written": 1073741824, 2024-09-18T14:18:28.574 INFO:tasks.workunit.client.0.smithi047.stdout: "blocksize": 4194304, 2024-09-18T14:18:28.574 INFO:tasks.workunit.client.0.smithi047.stdout: "elapsed_sec": 23.429895727999998, 2024-09-18T14:18:28.574 INFO:tasks.workunit.client.0.smithi047.stdout: "bytes_per_sec": 45827853.289027669, 2024-09-18T14:18:28.574 INFO:tasks.workunit.client.0.smithi047.stdout: "iops": 10.926211664444844 2024-09-18T14:18:28.574 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T14:18:28.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:84: TEST_bench: ceph tell osd.0 bench 10 14456 4444 3 2024-09-18T14:18:29.125 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T14:18:29.125 INFO:tasks.workunit.client.0.smithi047.stdout: "bytes_written": 10, 2024-09-18T14:18:29.125 INFO:tasks.workunit.client.0.smithi047.stdout: "blocksize": 14456, 2024-09-18T14:18:29.125 INFO:tasks.workunit.client.0.smithi047.stdout: "elapsed_sec": 0.050512733999999997, 2024-09-18T14:18:29.126 INFO:tasks.workunit.client.0.smithi047.stdout: "bytes_per_sec": 197.96988220831605, 2024-09-18T14:18:29.126 INFO:tasks.workunit.client.0.smithi047.stdout: "iops": 0.013694651508599616 2024-09-18T14:18:29.126 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T14:18:29.135 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:90: TEST_bench: ceph tell osd.0 bench 1 14456 2024-09-18T14:18:29.426 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T14:18:29.426 INFO:tasks.workunit.client.0.smithi047.stdout: "bytes_written": 1, 2024-09-18T14:18:29.426 INFO:tasks.workunit.client.0.smithi047.stdout: "blocksize": 14456, 2024-09-18T14:18:29.426 INFO:tasks.workunit.client.0.smithi047.stdout: "elapsed_sec": 0.051826116999999998, 2024-09-18T14:18:29.427 INFO:tasks.workunit.client.0.smithi047.stdout: "bytes_per_sec": 19.295290827981574, 2024-09-18T14:18:29.427 INFO:tasks.workunit.client.0.smithi047.stdout: "iops": 0.0013347600185377403 2024-09-18T14:18:29.427 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T14:18:29.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:35: run: teardown td/osd-bench 2024-09-18T14:18:29.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2024-09-18T14:18:29.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:18:29.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2024-09-18T14:18:29.436 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:18:29.436 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:18:29.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:18:29.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:18:29.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:18:29.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:18:29.542 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:18:29.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:18:29.543 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:18:29.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:18:29.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:18:29.544 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:18:29.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:18:29.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:18:29.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:18:29.545 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:18:29.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:18:29.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:18:29.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2024-09-18T14:18:29.573 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:18:29.573 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:18:29.573 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99929 2024-09-18T14:18:29.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.99929 2024-09-18T14:18:29.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:18:29.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:18:29.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T14:18:29.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/osd-bench 0 2024-09-18T14:18:29.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2024-09-18T14:18:29.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T14:18:29.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2024-09-18T14:18:29.575 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:18:29.575 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:18:29.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:18:29.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:18:29.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:18:29.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:18:29.576 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:18:29.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:18:29.577 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:18:29.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:18:29.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:18:29.578 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:18:29.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:18:29.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:18:29.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:18:29.580 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:18:29.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:18:29.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T14:18:29.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2024-09-18T14:18:29.582 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:18:29.582 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:18:29.582 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.99929 2024-09-18T14:18:29.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.99929 2024-09-18T14:18:29.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:18:29.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:18:29.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T14:18:29.585 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T14:18:29.585 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T14:18:29.650 INFO:tasks.workunit:Running workunit osd/osd-bluefs-volume-ops.sh... 2024-09-18T14:18:29.650 DEBUG:teuthology.orchestra.run.smithi047:workunit test osd/osd-bluefs-volume-ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh 2024-09-18T14:18:29.713 INFO:tasks.workunit.client.0.smithi047.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T14:18:29.720 INFO:tasks.workunit.client.0.smithi047.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T14:18:29.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T14:18:29.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T14:18:29.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T14:18:29.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T14:18:29.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T14:18:29.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T14:18:29.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T14:18:29.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T14:18:29.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/osd-bluefs-volume-ops 2024-09-18T14:18:29.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:8: run: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:18:29.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:9: run: shift 2024-09-18T14:18:29.721 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: set 2024-09-18T14:18:29.721 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-18T14:18:29.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: local 'funcs=TEST_bluestore 2024-09-18T14:18:29.723 INFO:tasks.workunit.client.0.smithi047.stderr:TEST_bluestore2 2024-09-18T14:18:29.723 INFO:tasks.workunit.client.0.smithi047.stderr:TEST_bluestore_expand' 2024-09-18T14:18:29.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-09-18T14:18:29.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-09-18T14:18:29.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:18:29.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2024-09-18T14:18:29.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:18:29.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:18:29.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-18T14:18:29.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:18:29.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:18:29.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:18:29.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:18:29.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:18:29.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:18:29.725 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:18:29.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:18:29.726 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:18:29.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:18:29.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:18:29.726 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:18:29.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:18:29.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:18:29.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:18:29.727 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:18:29.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:18:29.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:18:29.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-18T14:18:29.730 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:18:29.730 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:18:29.730 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:18:29.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.102152 2024-09-18T14:18:29.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:18:29.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:18:29.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2024-09-18T14:18:29.732 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T14:18:29.732 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:18:29.732 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:18:29.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.102152 2024-09-18T14:18:29.735 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T14:18:29.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T14:18:29.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T14:18:29.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T14:18:29.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-09-18T14:18:29.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore td/osd-bluefs-volume-ops 2024-09-18T14:18:29.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:20: TEST_bluestore: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:18:29.736 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: TEST_bluestore: ulimit -n 2024-09-18T14:18:29.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: TEST_bluestore: local flimit=4096 2024-09-18T14:18:29.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:23: TEST_bluestore: '[' 4096 -lt 1536 ']' 2024-09-18T14:18:29.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:26: TEST_bluestore: export CEPH_MON=127.0.0.1:7146 2024-09-18T14:18:29.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:26: TEST_bluestore: CEPH_MON=127.0.0.1:7146 2024-09-18T14:18:29.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:27: TEST_bluestore: export CEPH_ARGS 2024-09-18T14:18:29.736 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:28: TEST_bluestore: uuidgen 2024-09-18T14:18:29.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:28: TEST_bluestore: CEPH_ARGS+='--fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none ' 2024-09-18T14:18:29.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:29: TEST_bluestore: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-18T14:18:29.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:30: TEST_bluestore: CEPH_ARGS+='--bluestore_block_size=2147483648 ' 2024-09-18T14:18:29.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:31: TEST_bluestore: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-09-18T14:18:29.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:32: TEST_bluestore: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-09-18T14:18:29.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:33: TEST_bluestore: CEPH_ARGS+='--bluestore_block_wal_size=536870912 ' 2024-09-18T14:18:29.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:34: TEST_bluestore: CEPH_ARGS+='--bluestore_block_wal_create=true ' 2024-09-18T14:18:29.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:35: TEST_bluestore: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-09-18T14:18:29.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:37: TEST_bluestore: run_mon td/osd-bluefs-volume-ops a 2024-09-18T14:18:29.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:18:29.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T14:18:29.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T14:18:29.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T14:18:29.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-09-18T14:18:29.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-09-18T14:18:29.978 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T14:18:29.978 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:18:29.978 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:18:29.979 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:18:29.979 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:18:29.979 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:18:29.979 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:18:29.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T14:18:30.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T14:18:30.212 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T14:18:30.212 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:18:30.212 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:18:30.212 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T14:18:30.212 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T14:18:30.212 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:18:30.212 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:18:30.212 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:18:30.213 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:18:30.213 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:18:30.213 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:18:30.213 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.102152/ceph-mon.a.asok 2024-09-18T14:18:30.213 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:18:30.213 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.102152/ceph-mon.a.asok config get fsid 2024-09-18T14:18:30.289 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T14:18:30.290 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:18:30.290 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:18:30.290 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T14:18:30.290 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T14:18:30.290 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:18:30.290 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:18:30.290 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:18:30.290 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:18:30.290 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:18:30.290 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:18:30.291 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.102152/ceph-mon.a.asok 2024-09-18T14:18:30.291 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:18:30.291 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.102152/ceph-mon.a.asok config get mon_host 2024-09-18T14:18:30.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:38: TEST_bluestore: run_mgr td/osd-bluefs-volume-ops x 2024-09-18T14:18:30.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:18:30.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T14:18:30.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T14:18:30.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T14:18:30.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-09-18T14:18:30.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T14:18:31.199 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T14:18:31.199 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:18:31.199 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:18:31.199 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:18:31.199 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:18:31.200 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:18:31.200 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:18:31.200 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:18:31.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:18:31.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:39: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 0 2024-09-18T14:18:31.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:18:31.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:18:31.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T14:18:31.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:18:31.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-18T14:18:31.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-18T14:18:31.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:18:31.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:18:31.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:18:31.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-18T14:18:31.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-18T14:18:31.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:18:31.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:18:31.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-18T14:18:31.226 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:18:31.226 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:18:31.226 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:18:31.226 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:18:31.226 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:18:31.226 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:18:31.226 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:18:31.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:18:31.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:18:31.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:18:31.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:18:31.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-18T14:18:31.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-18T14:18:31.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:18:31.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:18:31.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:18:31.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:18:31.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:18:31.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:18:31.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-18T14:18:31.227 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:18:31.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=77e514a7-b7cf-4967-acb8-f65677560a17 2024-09-18T14:18:31.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 77e514a7-b7cf-4967-acb8-f65677560a17' 2024-09-18T14:18:31.228 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 77e514a7-b7cf-4967-acb8-f65677560a17 2024-09-18T14:18:31.228 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:18:31.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA34epmlBk3DhAALHEPlGMq31EsOfBX3tMDkg== 2024-09-18T14:18:31.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA34epmlBk3DhAALHEPlGMq31EsOfBX3tMDkg=="}' 2024-09-18T14:18:31.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 77e514a7-b7cf-4967-acb8-f65677560a17 -i td/osd-bluefs-volume-ops/0/new.json 2024-09-18T14:18:31.578 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:18:31.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-09-18T14:18:31.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA34epmlBk3DhAALHEPlGMq31EsOfBX3tMDkg== --osd-uuid 77e514a7-b7cf-4967-acb8-f65677560a17 2024-09-18T14:18:31.613 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:18:31.612+0000 7fa10780b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:18:31.615 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:18:31.614+0000 7fa10780b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:18:31.616 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:18:31.615+0000 7fa10780b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:18:31.616 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:18:31.616+0000 7fa10780b440 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-09-18T14:18:31.616 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:18:31.616+0000 7fa10780b440 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-09-18T14:18:31.617 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:18:31.616+0000 7fa10780b440 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-09-18T14:18:31.660 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:18:31.660+0000 7fa10780b440 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-09-18T14:18:40.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-09-18T14:18:40.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:18:40.669 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T14:18:40.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T14:18:40.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:18:41.189 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T14:18:41.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T14:18:41.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:18:41.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:18:41.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:18:41.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:18:41.211 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:18:41.210+0000 7f02f4634440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:18:41.213 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:18:41.212+0000 7f02f4634440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:18:41.214 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:18:41.213+0000 7f02f4634440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:18:41.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T14:18:41.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:18:41.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:18:41.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:18:41.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:18:41.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:18:41.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:18:41.589 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:18:41.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:18:41.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:18:41.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:18:42.228 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:18:42.228+0000 7f02f4634440 -1 Falling back to public interface 2024-09-18T14:18:42.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:18:42.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:18:42.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:18:42.974 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:18:42.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:18:42.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:18:43.390 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:18:44.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:18:44.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:18:44.392 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:18:44.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:18:44.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:18:44.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:18:44.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:18:45.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:18:45.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:18:45.809 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:18:45.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:18:45.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:18:45.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:18:46.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:18:46.812 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:18:46.811+0000 7f02f4634440 -1 osd.0 0 log_to_monitors true 2024-09-18T14:18:47.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:18:47.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:18:47.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:18:47.227 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:18:47.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:18:47.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:18:47.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:18:48.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:18:48.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:18:48.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:18:48.653 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:18:48.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:18:48.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:18:49.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:18:50.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:18:50.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:18:50.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:18:50.064 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:18:50.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:18:50.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:18:50.470 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:18:51.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:18:51.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:18:51.471 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:18:51.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:18:51.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:18:51.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:18:51.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:18:52.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:18:52.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:18:52.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:18:52.884 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:18:52.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:18:52.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:18:53.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:18:54.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:18:54.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:18:54.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:18:54.259 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:18:54.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:18:54.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:18:54.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:18:55.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:18:55.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:18:55.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:18:55.739 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:18:55.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:18:55.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:18:56.135 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:18:57.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:18:57.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:18:57.137 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:18:57.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:18:57.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:18:57.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:18:57.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:18:58.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:18:58.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:18:58.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:18:58.556 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:18:58.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:18:58.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:18:59.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:19:00.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:19:00.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:19:00.339 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:19:00.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:19:00.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:19:00.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:19:01.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:19:02.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:19:02.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:19:02.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:19:02.274 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:19:02.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:19:02.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:19:03.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:19:04.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:19:04.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:19:04.144 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:19:04.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:19:04.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:19:04.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:19:05.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:19:06.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:19:06.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:19:06.062 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:19:06.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:19:06.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:19:06.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:19:07.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:19:08.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:19:08.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:19:08.200 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T14:19:08.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:19:08.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:19:08.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:19:09.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:19:10.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:19:10.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:19:10.104 INFO:tasks.workunit.client.0.smithi047.stdout:18 2024-09-18T14:19:10.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T14:19:10.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:19:10.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:19:10.514 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/103641,v1:127.0.0.1:6803/103641] [v2:127.0.0.1:6804/103641,v1:127.0.0.1:6805/103641] exists,up 77e514a7-b7cf-4967-acb8-f65677560a17 2024-09-18T14:19:10.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:19:10.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:19:10.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:19:10.515 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-18T14:19:10.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: osd_pid0=103641 2024-09-18T14:19:10.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:41: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 1 2024-09-18T14:19:10.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:19:10.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:19:10.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T14:19:10.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:19:10.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-18T14:19:10.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-18T14:19:10.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:19:10.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:19:10.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:19:10.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-18T14:19:10.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-18T14:19:10.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:19:10.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:19:10.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-18T14:19:10.520 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:19:10.520 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:19:10.520 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:19:10.520 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:19:10.520 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:19:10.520 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:19:10.520 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:19:10.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:19:10.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:19:10.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:19:10.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:19:10.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-18T14:19:10.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-18T14:19:10.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:19:10.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:19:10.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:19:10.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:19:10.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:19:10.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:19:10.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-18T14:19:10.521 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:19:10.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8164e048-2490-4c40-b72e-815a1382b89d 2024-09-18T14:19:10.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 8164e048-2490-4c40-b72e-815a1382b89d' 2024-09-18T14:19:10.522 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 8164e048-2490-4c40-b72e-815a1382b89d 2024-09-18T14:19:10.522 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:19:10.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBe4epmNs2/HxAA3AsXBZv7hhDL3Z7GJKQu0A== 2024-09-18T14:19:10.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBe4epmNs2/HxAA3AsXBZv7hhDL3Z7GJKQu0A=="}' 2024-09-18T14:19:10.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8164e048-2490-4c40-b72e-815a1382b89d -i td/osd-bluefs-volume-ops/1/new.json 2024-09-18T14:19:11.030 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:19:11.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/1/new.json 2024-09-18T14:19:11.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBe4epmNs2/HxAA3AsXBZv7hhDL3Z7GJKQu0A== --osd-uuid 8164e048-2490-4c40-b72e-815a1382b89d 2024-09-18T14:19:11.066 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:19:11.065+0000 7f7f19720440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:19:11.089 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:19:11.088+0000 7f7f19720440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:19:11.090 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:19:11.089+0000 7f7f19720440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:19:11.090 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:19:11.089+0000 7f7f19720440 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2024-09-18T14:19:11.090 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:19:11.089+0000 7f7f19720440 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2024-09-18T14:19:11.090 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:19:11.089+0000 7f7f19720440 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2024-09-18T14:19:11.164 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:19:11.163+0000 7f7f19720440 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_fsid unparsable uuid 2024-09-18T14:19:20.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/1/keyring 2024-09-18T14:19:20.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:19:20.557 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T14:19:20.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T14:19:20.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:19:21.097 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T14:19:21.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T14:19:21.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:19:21.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:19:21.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:19:21.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:19:21.119 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:19:21.118+0000 7fef1de09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:19:21.121 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:19:21.120+0000 7fef1de09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:19:21.122 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:19:21.121+0000 7fef1de09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:19:21.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T14:19:21.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:19:21.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:19:21.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:19:21.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:19:21.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:19:21.497 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:19:21.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:19:21.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:19:21.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:19:21.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:19:22.139 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:19:22.139+0000 7fef1de09440 -1 Falling back to public interface 2024-09-18T14:19:22.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:19:22.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:19:22.902 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:19:22.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:19:22.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:19:22.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:19:23.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:19:24.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:19:24.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:19:24.302 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:19:24.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:19:24.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:19:24.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:19:24.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:19:25.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:19:25.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:19:25.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:19:25.721 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:19:25.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:19:25.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:19:26.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:19:26.815 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:19:26.815+0000 7fef1de09440 -1 osd.1 0 log_to_monitors true 2024-09-18T14:19:27.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:19:27.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:19:27.100 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:19:27.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:19:27.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:19:27.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:19:27.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:19:28.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:19:28.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:19:28.545 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:19:28.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:19:28.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:19:28.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:19:28.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:19:29.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:19:29.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:19:29.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:19:29.935 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:19:29.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:19:29.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:19:30.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:19:31.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:19:31.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:19:31.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:19:31.475 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:19:31.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:19:31.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:19:31.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:19:32.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:19:32.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:19:32.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:19:32.883 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:19:32.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:19:32.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:19:33.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:19:34.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:19:34.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:19:34.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:19:34.308 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:19:34.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:19:34.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:19:34.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:19:35.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:19:35.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:19:35.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:19:35.732 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:19:35.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:19:35.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:19:36.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:19:37.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:19:37.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:19:37.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:19:37.142 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:19:37.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:19:37.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:19:37.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:19:38.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:19:38.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:19:38.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:19:38.576 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:19:38.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:19:38.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:19:39.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:19:40.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:19:40.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:19:40.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:19:40.040 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:19:40.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:19:40.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:19:41.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:19:42.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:19:42.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:19:42.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:19:42.103 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:19:42.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:19:42.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:19:42.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:19:43.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:19:43.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:19:43.771 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:19:43.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:19:43.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:19:43.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:19:44.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:19:45.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:19:45.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:19:45.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:19:45.441 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:19:45.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:19:45.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:19:46.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:19:47.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:19:47.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:19:47.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:19:47.251 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T14:19:47.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:19:47.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:19:48.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:19:49.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:19:49.236 INFO:tasks.workunit.client.0.smithi047.stdout:18 2024-09-18T14:19:49.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:19:49.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T14:19:49.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:19:49.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:19:49.649 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/106203,v1:127.0.0.1:6811/106203] [v2:127.0.0.1:6812/106203,v1:127.0.0.1:6813/106203] exists,up 8164e048-2490-4c40-b72e-815a1382b89d 2024-09-18T14:19:49.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:19:49.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:19:49.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:19:49.650 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-18T14:19:49.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: osd_pid1=106203 2024-09-18T14:19:49.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:43: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 2 2024-09-18T14:19:49.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:19:49.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:19:49.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T14:19:49.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:19:49.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-18T14:19:49.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-18T14:19:49.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:19:49.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:19:49.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:19:49.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-18T14:19:49.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-18T14:19:49.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:19:49.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:19:49.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-18T14:19:49.656 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:19:49.656 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:19:49.656 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:19:49.656 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:19:49.657 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:19:49.657 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:19:49.657 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:19:49.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:19:49.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:19:49.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:19:49.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:19:49.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-18T14:19:49.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-18T14:19:49.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:19:49.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:19:49.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:19:49.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:19:49.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:19:49.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:19:49.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-18T14:19:49.658 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:19:49.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c58dcf9b-e2da-4e51-9972-4ac5044eccd6 2024-09-18T14:19:49.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 c58dcf9b-e2da-4e51-9972-4ac5044eccd6' 2024-09-18T14:19:49.658 INFO:tasks.workunit.client.0.smithi047.stdout:add osd2 c58dcf9b-e2da-4e51-9972-4ac5044eccd6 2024-09-18T14:19:49.659 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:19:49.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCF4epm3fzVJxAAW5FDVNkve7+qCu1lHaAK0w== 2024-09-18T14:19:49.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCF4epm3fzVJxAAW5FDVNkve7+qCu1lHaAK0w=="}' 2024-09-18T14:19:49.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c58dcf9b-e2da-4e51-9972-4ac5044eccd6 -i td/osd-bluefs-volume-ops/2/new.json 2024-09-18T14:19:50.224 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:19:50.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/2/new.json 2024-09-18T14:19:50.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCF4epm3fzVJxAAW5FDVNkve7+qCu1lHaAK0w== --osd-uuid c58dcf9b-e2da-4e51-9972-4ac5044eccd6 2024-09-18T14:19:50.259 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:19:50.259+0000 7fc92b808440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:19:50.273 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:19:50.273+0000 7fc92b808440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:19:50.274 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:19:50.274+0000 7fc92b808440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:19:50.275 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:19:50.274+0000 7fc92b808440 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2024-09-18T14:19:50.275 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:19:50.274+0000 7fc92b808440 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2024-09-18T14:19:50.275 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:19:50.274+0000 7fc92b808440 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2024-09-18T14:19:50.340 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:19:50.340+0000 7fc92b808440 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_fsid unparsable uuid 2024-09-18T14:19:59.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/2/keyring 2024-09-18T14:19:59.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:19:59.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T14:19:59.629 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd2 key to auth repository 2024-09-18T14:19:59.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:20:00.135 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T14:20:00.135 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T14:20:00.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:20:00.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:20:00.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:20:00.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:20:00.157 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:20:00.156+0000 7fb1a5671440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:20:00.159 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:20:00.159+0000 7fb1a5671440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:20:00.160 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:20:00.160+0000 7fb1a5671440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:20:00.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T14:20:00.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:20:00.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T14:20:00.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:20:00.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:20:00.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:20:00.499 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:20:00.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:20:00.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:20:00.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:20:00.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:20:01.180 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:20:01.180+0000 7fb1a5671440 -1 Falling back to public interface 2024-09-18T14:20:01.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:20:01.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:20:01.904 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:20:01.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:20:01.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:20:01.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:20:02.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:20:03.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:20:03.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:20:03.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:20:03.366 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:20:03.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:20:03.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:20:03.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:20:04.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:20:04.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:20:04.749 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:20:04.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:20:04.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:20:04.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:20:05.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:20:05.910 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:20:05.910+0000 7fb1a5671440 -1 osd.2 0 log_to_monitors true 2024-09-18T14:20:06.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:20:06.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:20:06.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:20:06.148 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:20:06.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:20:06.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:20:06.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:20:07.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:20:07.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:20:07.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:20:07.697 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:20:07.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:20:07.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:20:08.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:20:09.212 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:20:09.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:20:09.213 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:20:09.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:20:09.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:20:09.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:20:09.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:20:10.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:20:10.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:20:10.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:20:10.712 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:20:10.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:20:10.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:20:11.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:20:12.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:20:12.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:20:12.145 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:20:12.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:20:12.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:20:12.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:20:12.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:20:13.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:20:13.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:20:13.651 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:20:13.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:20:13.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:20:13.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:20:14.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:20:15.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:20:15.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:20:15.123 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:20:15.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:20:15.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:20:15.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:20:15.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:20:16.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:20:16.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:20:16.555 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:20:16.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:20:16.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:20:16.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:20:16.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:20:17.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:20:17.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:20:17.969 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:20:17.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:20:17.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:20:17.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:20:18.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:20:19.612 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:20:19.612 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:20:19.613 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:20:19.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:20:19.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:20:19.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:20:20.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:20:21.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:20:21.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:20:21.149 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:20:21.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:20:21.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:20:21.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:20:22.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:20:23.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:20:23.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:20:23.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:20:23.462 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:20:23.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:20:23.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:20:23.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:20:24.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:20:24.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:20:24.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:20:24.921 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:20:24.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:20:24.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:20:25.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:20:26.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:20:26.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:20:26.401 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T14:20:26.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:20:26.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:20:26.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:20:27.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:20:28.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:20:28.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:20:28.070 INFO:tasks.workunit.client.0.smithi047.stdout:18 2024-09-18T14:20:28.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T14:20:28.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:20:28.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:20:28.519 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/108765,v1:127.0.0.1:6819/108765] [v2:127.0.0.1:6820/108765,v1:127.0.0.1:6821/108765] exists,up c58dcf9b-e2da-4e51-9972-4ac5044eccd6 2024-09-18T14:20:28.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:20:28.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:20:28.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:20:28.520 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-18T14:20:28.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: osd_pid2=108765 2024-09-18T14:20:28.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:45: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 3 2024-09-18T14:20:28.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:20:28.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:20:28.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-18T14:20:28.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:20:28.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-18T14:20:28.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-18T14:20:28.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:20:28.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:20:28.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:20:28.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-18T14:20:28.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-18T14:20:28.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:20:28.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:20:28.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-18T14:20:28.525 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:20:28.525 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:20:28.525 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:20:28.525 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:20:28.525 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:20:28.525 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:20:28.525 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:20:28.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:20:28.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:20:28.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:20:28.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:20:28.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-18T14:20:28.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-18T14:20:28.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:20:28.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:20:28.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:20:28.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:20:28.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:20:28.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:20:28.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-18T14:20:28.527 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:20:28.527 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7dc3b3b6-05fa-4664-93b4-abdebdb06ac0 2024-09-18T14:20:28.528 INFO:tasks.workunit.client.0.smithi047.stdout:add osd3 7dc3b3b6-05fa-4664-93b4-abdebdb06ac0 2024-09-18T14:20:28.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 7dc3b3b6-05fa-4664-93b4-abdebdb06ac0' 2024-09-18T14:20:28.528 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:20:28.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCs4epmNGoeIBAAI2B5eWVfn1t3Gy3FxUU7pg== 2024-09-18T14:20:28.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCs4epmNGoeIBAAI2B5eWVfn1t3Gy3FxUU7pg=="}' 2024-09-18T14:20:28.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7dc3b3b6-05fa-4664-93b4-abdebdb06ac0 -i td/osd-bluefs-volume-ops/3/new.json 2024-09-18T14:20:29.036 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:20:29.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/3/new.json 2024-09-18T14:20:29.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCs4epmNGoeIBAAI2B5eWVfn1t3Gy3FxUU7pg== --osd-uuid 7dc3b3b6-05fa-4664-93b4-abdebdb06ac0 2024-09-18T14:20:29.072 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:20:29.071+0000 7f9ae3010440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:20:29.095 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:20:29.095+0000 7f9ae3010440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:20:29.096 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:20:29.096+0000 7f9ae3010440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:20:29.097 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:20:29.096+0000 7f9ae3010440 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2024-09-18T14:20:29.097 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:20:29.096+0000 7f9ae3010440 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2024-09-18T14:20:29.097 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:20:29.096+0000 7f9ae3010440 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2024-09-18T14:20:29.187 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:20:29.187+0000 7f9ae3010440 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_fsid unparsable uuid 2024-09-18T14:20:39.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/3/keyring 2024-09-18T14:20:39.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:20:39.054 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd3 key to auth repository 2024-09-18T14:20:39.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-18T14:20:39.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:20:39.565 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.3 2024-09-18T14:20:39.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-18T14:20:39.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:20:39.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:20:39.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:20:39.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:20:39.587 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:20:39.586+0000 7ff95b43e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:20:39.588 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:20:39.588+0000 7ff95b43e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:20:39.589 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:20:39.589+0000 7ff95b43e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:20:39.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-18T14:20:39.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:20:39.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T14:20:39.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:20:39.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:20:39.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:20:39.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:20:39.932 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:20:39.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:20:39.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:20:40.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:20:40.604 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:20:40.603+0000 7ff95b43e440 -1 Falling back to public interface 2024-09-18T14:20:41.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:20:41.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:20:41.328 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:20:41.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:20:41.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:20:41.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:20:41.746 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:20:42.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:20:42.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:20:42.747 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:20:42.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:20:42.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:20:42.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:20:43.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:20:44.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:20:44.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:20:44.138 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:20:44.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:20:44.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:20:44.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:20:44.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:20:45.200 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:20:45.199+0000 7ff95b43e440 -1 osd.3 0 log_to_monitors true 2024-09-18T14:20:45.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:20:45.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:20:45.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:20:45.592 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:20:45.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:20:45.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:20:46.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:20:47.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:20:47.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:20:47.019 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:20:47.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:20:47.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:20:47.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:20:47.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:20:48.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:20:48.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:20:48.401 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:20:48.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:20:48.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:20:48.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:20:48.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:20:49.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:20:49.811 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:20:49.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:20:49.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:20:49.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:20:49.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:20:50.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:20:51.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:20:51.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:20:51.243 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:20:51.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:20:51.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:20:51.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:20:51.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:20:52.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:20:52.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:20:52.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:20:52.711 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:20:52.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:20:52.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:20:53.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:20:54.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:20:54.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:20:54.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:20:54.166 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:20:54.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:20:54.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:20:54.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:20:55.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:20:55.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:20:55.587 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:20:55.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:20:55.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:20:55.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:20:56.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:20:57.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:20:57.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:20:57.027 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:20:57.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:20:57.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:20:57.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:20:57.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:20:58.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:20:58.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:20:58.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:20:58.610 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:20:58.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:20:58.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:20:59.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:21:00.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:21:00.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:21:00.149 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:21:00.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:21:00.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:21:00.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:21:00.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:21:01.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:21:01.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:21:01.898 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:21:01.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:21:01.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:21:01.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:21:02.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:21:03.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:21:03.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:21:03.702 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:21:03.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:21:03.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:21:03.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:21:04.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:21:05.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:21:05.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:21:05.857 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T14:21:05.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:21:05.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:21:05.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:21:06.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:21:07.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:21:07.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:21:07.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T14:21:07.333 INFO:tasks.workunit.client.0.smithi047.stdout:18 2024-09-18T14:21:07.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:21:07.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:21:08.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:21:09.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:21:09.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:21:09.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T14:21:09.084 INFO:tasks.workunit.client.0.smithi047.stdout:19 2024-09-18T14:21:09.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:21:09.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:21:09.518 INFO:tasks.workunit.client.0.smithi047.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/111331,v1:127.0.0.1:6827/111331] [v2:127.0.0.1:6828/111331,v1:127.0.0.1:6829/111331] exists,up 7dc3b3b6-05fa-4664-93b4-abdebdb06ac0 2024-09-18T14:21:09.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:21:09.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:21:09.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:21:09.519 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-18T14:21:09.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: osd_pid3=111331 2024-09-18T14:21:09.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:48: TEST_bluestore: sleep 5 2024-09-18T14:21:14.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:50: TEST_bluestore: create_pool foo 16 2024-09-18T14:21:14.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-09-18T14:21:15.340 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'foo' already exists 2024-09-18T14:21:15.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T14:21:16.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:53: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-18T14:21:16.441 INFO:tasks.workunit.client.0.smithi047.stdout:hints = 1 2024-09-18T14:21:16.441 INFO:tasks.workunit.client.0.smithi047.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-18T14:21:16.441 INFO:tasks.workunit.client.0.smithi047.stdout:Object prefix: benchmark_data_smithi047_112709 2024-09-18T14:21:16.441 INFO:tasks.workunit.client.0.smithi047.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T14:21:16.441 INFO:tasks.workunit.client.0.smithi047.stdout: 0 0 0 0 0 0 - 0 2024-09-18T14:21:17.441 INFO:tasks.workunit.client.0.smithi047.stdout: 1 16 54 38 0.148428 0.148438 0.31033 0.33452 2024-09-18T14:21:18.441 INFO:tasks.workunit.client.0.smithi047.stdout: 2 16 113 97 0.189433 0.230469 0.266811 0.308111 2024-09-18T14:21:19.441 INFO:tasks.workunit.client.0.smithi047.stdout: 3 16 172 156 0.203102 0.230469 0.241893 0.295374 2024-09-18T14:21:20.441 INFO:tasks.workunit.client.0.smithi047.stdout: 4 16 227 211 0.206029 0.214844 0.285306 0.293147 2024-09-18T14:21:21.441 INFO:tasks.workunit.client.0.smithi047.stdout: 5 16 284 268 0.20935 0.222656 0.287792 0.289702 2024-09-18T14:21:22.441 INFO:tasks.workunit.client.0.smithi047.stdout: 6 16 347 331 0.21547 0.246094 0.263686 0.28589 2024-09-18T14:21:23.441 INFO:tasks.workunit.client.0.smithi047.stdout: 7 16 396 380 0.212028 0.191406 0.276663 0.286803 2024-09-18T14:21:24.441 INFO:tasks.workunit.client.0.smithi047.stdout: 8 16 451 435 0.212376 0.214844 0.277689 0.287397 2024-09-18T14:21:25.442 INFO:tasks.workunit.client.0.smithi047.stdout: 9 16 515 499 0.216553 0.25 0.25186 0.285234 2024-09-18T14:21:26.442 INFO:tasks.workunit.client.0.smithi047.stdout: 10 16 572 556 0.217161 0.222656 0.268276 0.284243 2024-09-18T14:21:27.442 INFO:tasks.workunit.client.0.smithi047.stdout: 11 16 627 611 0.216947 0.214844 0.300203 0.281949 2024-09-18T14:21:28.442 INFO:tasks.workunit.client.0.smithi047.stdout: 12 16 691 675 0.219699 0.25 0.300653 0.281596 2024-09-18T14:21:29.442 INFO:tasks.workunit.client.0.smithi047.stdout: 13 16 748 732 0.219924 0.222656 0.277812 0.282178 2024-09-18T14:21:30.443 INFO:tasks.workunit.client.0.smithi047.stdout: 14 16 803 787 0.219559 0.214844 0.285853 0.282776 2024-09-18T14:21:31.443 INFO:tasks.workunit.client.0.smithi047.stdout: 15 16 860 844 0.219763 0.222656 0.242704 0.282197 2024-09-18T14:21:32.443 INFO:tasks.workunit.client.0.smithi047.stdout: 16 16 915 899 0.219454 0.214844 0.294888 0.282395 2024-09-18T14:21:33.443 INFO:tasks.workunit.client.0.smithi047.stdout: 17 16 979 963 0.221249 0.25 0.268539 0.281057 2024-09-18T14:21:34.443 INFO:tasks.workunit.client.0.smithi047.stdout: 18 16 1027 1011 0.219373 0.1875 0.270087 0.282517 2024-09-18T14:21:35.443 INFO:tasks.workunit.client.0.smithi047.stdout: 19 16 1084 1068 0.219544 0.222656 0.286326 0.282275 2024-09-18T14:21:36.445 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T14:21:36.443871+0000 min lat: 0.14335 max lat: 0.598077 avg lat: 0.282051 2024-09-18T14:21:36.445 INFO:tasks.workunit.client.0.smithi047.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T14:21:36.445 INFO:tasks.workunit.client.0.smithi047.stdout: 20 16 1139 1123 0.219308 0.214844 0.317815 0.282051 2024-09-18T14:21:37.443 INFO:tasks.workunit.client.0.smithi047.stdout: 21 16 1203 1187 0.220767 0.25 0.283247 0.281623 2024-09-18T14:21:38.443 INFO:tasks.workunit.client.0.smithi047.stdout: 22 16 1260 1244 0.220851 0.222656 0.296866 0.281787 2024-09-18T14:21:39.444 INFO:tasks.workunit.client.0.smithi047.stdout: 23 16 1315 1299 0.220589 0.214844 0.263154 0.281976 2024-09-18T14:21:40.444 INFO:tasks.workunit.client.0.smithi047.stdout: 24 16 1372 1356 0.220674 0.222656 0.30203 0.282262 2024-09-18T14:21:41.444 INFO:tasks.workunit.client.0.smithi047.stdout: 25 16 1427 1411 0.22044 0.214844 0.316602 0.281948 2024-09-18T14:21:42.444 INFO:tasks.workunit.client.0.smithi047.stdout: 26 16 1484 1468 0.220524 0.222656 0.272428 0.282022 2024-09-18T14:21:43.444 INFO:tasks.workunit.client.0.smithi047.stdout: 27 16 1539 1523 0.220312 0.214844 0.300861 0.28198 2024-09-18T14:21:44.444 INFO:tasks.workunit.client.0.smithi047.stdout: 28 16 1571 1555 0.216907 0.125 0.735878 0.285117 2024-09-18T14:21:45.444 INFO:tasks.workunit.client.0.smithi047.stdout: 29 16 1628 1612 0.217104 0.222656 0.285513 0.28625 2024-09-18T14:21:46.445 INFO:tasks.workunit.client.0.smithi047.stdout: 30 16 1692 1676 0.2182 0.25 0.303383 0.285573 2024-09-18T14:21:47.445 INFO:tasks.workunit.client.0.smithi047.stdout:Total time run: 30.223 2024-09-18T14:21:47.445 INFO:tasks.workunit.client.0.smithi047.stdout:Total writes made: 1692 2024-09-18T14:21:47.445 INFO:tasks.workunit.client.0.smithi047.stdout:Write size: 4096 2024-09-18T14:21:47.445 INFO:tasks.workunit.client.0.smithi047.stdout:Object size: 4096 2024-09-18T14:21:47.445 INFO:tasks.workunit.client.0.smithi047.stdout:Bandwidth (MB/sec): 0.218687 2024-09-18T14:21:47.445 INFO:tasks.workunit.client.0.smithi047.stdout:Stddev Bandwidth: 0.0273097 2024-09-18T14:21:47.445 INFO:tasks.workunit.client.0.smithi047.stdout:Max bandwidth (MB/sec): 0.25 2024-09-18T14:21:47.445 INFO:tasks.workunit.client.0.smithi047.stdout:Min bandwidth (MB/sec): 0.125 2024-09-18T14:21:47.445 INFO:tasks.workunit.client.0.smithi047.stdout:Average IOPS: 55 2024-09-18T14:21:47.446 INFO:tasks.workunit.client.0.smithi047.stdout:Stddev IOPS: 6.99129 2024-09-18T14:21:47.446 INFO:tasks.workunit.client.0.smithi047.stdout:Max IOPS: 64 2024-09-18T14:21:47.446 INFO:tasks.workunit.client.0.smithi047.stdout:Min IOPS: 32 2024-09-18T14:21:47.446 INFO:tasks.workunit.client.0.smithi047.stdout:Average Latency(s): 0.285371 2024-09-18T14:21:47.446 INFO:tasks.workunit.client.0.smithi047.stdout:Stddev Latency(s): 0.0499043 2024-09-18T14:21:47.446 INFO:tasks.workunit.client.0.smithi047.stdout:Max latency(s): 0.736199 2024-09-18T14:21:47.446 INFO:tasks.workunit.client.0.smithi047.stdout:Min latency(s): 0.14335 2024-09-18T14:21:47.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:55: TEST_bluestore: echo 'after bench' 2024-09-18T14:21:47.576 INFO:tasks.workunit.client.0.smithi047.stdout:after bench 2024-09-18T14:21:47.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 103641 2024-09-18T14:21:47.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: sleep 1 2024-09-18T14:21:48.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 103641 2024-09-18T14:21:48.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 58: kill: (103641) - No such process 2024-09-18T14:21:48.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:59: TEST_bluestore: ceph osd down 0 2024-09-18T14:21:48.991 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already down. 2024-09-18T14:21:49.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 106203 2024-09-18T14:21:49.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: sleep 1 2024-09-18T14:21:50.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 106203 2024-09-18T14:21:50.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 60: kill: (106203) - No such process 2024-09-18T14:21:50.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:61: TEST_bluestore: ceph osd down 1 2024-09-18T14:21:50.476 INFO:tasks.workunit.client.0.smithi047.stderr:osd.1 is already down. 2024-09-18T14:21:50.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 108765 2024-09-18T14:21:50.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: sleep 1 2024-09-18T14:21:51.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 108765 2024-09-18T14:21:51.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 62: kill: (108765) - No such process 2024-09-18T14:21:51.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:63: TEST_bluestore: ceph osd down 2 2024-09-18T14:21:51.906 INFO:tasks.workunit.client.0.smithi047.stderr:osd.2 is already down. 2024-09-18T14:21:51.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 111331 2024-09-18T14:21:51.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: sleep 1 2024-09-18T14:21:52.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 111331 2024-09-18T14:21:52.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 64: kill: (111331) - No such process 2024-09-18T14:21:52.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:65: TEST_bluestore: ceph osd down 3 2024-09-18T14:21:53.418 INFO:tasks.workunit.client.0.smithi047.stderr:osd.3 is already down. 2024-09-18T14:21:53.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:68: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-18T14:21:55.207 INFO:tasks.workunit.client.0.smithi047.stdout:fsck success 2024-09-18T14:21:55.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:69: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-18T14:21:57.005 INFO:tasks.workunit.client.0.smithi047.stdout:fsck success 2024-09-18T14:21:57.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:70: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-18T14:21:59.049 INFO:tasks.workunit.client.0.smithi047.stdout:fsck success 2024-09-18T14:21:59.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:71: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-18T14:22:00.843 INFO:tasks.workunit.client.0.smithi047.stdout:fsck success 2024-09-18T14:22:00.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:73: TEST_bluestore: truncate td/osd-bluefs-volume-ops/0/block -s 4294967296 2024-09-18T14:22:00.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:74: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-expand 2024-09-18T14:22:00.883 INFO:tasks.workunit.client.0.smithi047.stdout:inferring bluefs devices from bluestore path 2024-09-18T14:22:01.780 INFO:tasks.workunit.client.0.smithi047.stdout:0 : device size 0x1ffff000 : using 0x11e00000(286 MiB) 2024-09-18T14:22:01.781 INFO:tasks.workunit.client.0.smithi047.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-18T14:22:01.781 INFO:tasks.workunit.client.0.smithi047.stdout:2 : device size 0x100000000 : using 0x8053c000(2.0 GiB) 2024-09-18T14:22:01.781 INFO:tasks.workunit.client.0.smithi047.stdout:Expanding DB/WAL... 2024-09-18T14:22:01.781 INFO:tasks.workunit.client.0.smithi047.stdout:0 : expanding to 0x536870912 2024-09-18T14:22:01.809 INFO:tasks.workunit.client.0.smithi047.stdout:0 : size label updated to 536870912 2024-09-18T14:22:01.810 INFO:tasks.workunit.client.0.smithi047.stdout:1 : expanding to 0x1073741824 2024-09-18T14:22:01.834 INFO:tasks.workunit.client.0.smithi047.stdout:1 : size label updated to 1073741824 2024-09-18T14:22:01.834 INFO:tasks.workunit.client.0.smithi047.stdout:2 : expanding from 0x80000000 to 0x100000000 2024-09-18T14:22:02.019 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:22:02.019+0000 7fe96aeea780 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/0: (21) Is a directory 2024-09-18T14:22:02.019 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:22:02.019+0000 7fe96aeea780 -1 bluestore(td/osd-bluefs-volume-ops/0) unable to read label for td/osd-bluefs-volume-ops/0: (21) Is a directory 2024-09-18T14:22:07.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:75: TEST_bluestore: truncate td/osd-bluefs-volume-ops/1/block -s 4311744512 2024-09-18T14:22:07.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:76: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 bluefs-bdev-expand 2024-09-18T14:22:07.087 INFO:tasks.workunit.client.0.smithi047.stdout:inferring bluefs devices from bluestore path 2024-09-18T14:22:08.014 INFO:tasks.workunit.client.0.smithi047.stdout:0 : device size 0x1ffff000 : using 0x11e00000(286 MiB) 2024-09-18T14:22:08.014 INFO:tasks.workunit.client.0.smithi047.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-18T14:22:08.015 INFO:tasks.workunit.client.0.smithi047.stdout:2 : device size 0x101000000 : using 0x81527000(2.0 GiB) 2024-09-18T14:22:08.015 INFO:tasks.workunit.client.0.smithi047.stdout:Expanding DB/WAL... 2024-09-18T14:22:08.015 INFO:tasks.workunit.client.0.smithi047.stdout:0 : expanding to 0x536870912 2024-09-18T14:22:08.033 INFO:tasks.workunit.client.0.smithi047.stdout:0 : size label updated to 536870912 2024-09-18T14:22:08.033 INFO:tasks.workunit.client.0.smithi047.stdout:1 : expanding to 0x1073741824 2024-09-18T14:22:08.050 INFO:tasks.workunit.client.0.smithi047.stdout:1 : size label updated to 1073741824 2024-09-18T14:22:08.050 INFO:tasks.workunit.client.0.smithi047.stdout:2 : expanding from 0x80000000 to 0x101000000 2024-09-18T14:22:08.209 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:22:08.208+0000 7f88ff969780 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/1: (21) Is a directory 2024-09-18T14:22:08.210 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:22:08.208+0000 7f88ff969780 -1 bluestore(td/osd-bluefs-volume-ops/1) unable to read label for td/osd-bluefs-volume-ops/1: (21) Is a directory 2024-09-18T14:22:13.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:77: TEST_bluestore: truncate td/osd-bluefs-volume-ops/2/block -s 4295099392 2024-09-18T14:22:13.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:78: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 bluefs-bdev-expand 2024-09-18T14:22:13.345 INFO:tasks.workunit.client.0.smithi047.stdout:inferring bluefs devices from bluestore path 2024-09-18T14:22:14.258 INFO:tasks.workunit.client.0.smithi047.stdout:0 : device size 0x1ffff000 : using 0x11e00000(286 MiB) 2024-09-18T14:22:14.258 INFO:tasks.workunit.client.0.smithi047.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-18T14:22:14.258 INFO:tasks.workunit.client.0.smithi047.stdout:2 : device size 0x100020000 : using 0x80568000(2.0 GiB) 2024-09-18T14:22:14.258 INFO:tasks.workunit.client.0.smithi047.stdout:Expanding DB/WAL... 2024-09-18T14:22:14.258 INFO:tasks.workunit.client.0.smithi047.stdout:0 : expanding to 0x536870912 2024-09-18T14:22:14.281 INFO:tasks.workunit.client.0.smithi047.stdout:0 : size label updated to 536870912 2024-09-18T14:22:14.281 INFO:tasks.workunit.client.0.smithi047.stdout:1 : expanding to 0x1073741824 2024-09-18T14:22:14.306 INFO:tasks.workunit.client.0.smithi047.stdout:1 : size label updated to 1073741824 2024-09-18T14:22:14.306 INFO:tasks.workunit.client.0.smithi047.stdout:2 : expanding from 0x80000000 to 0x100020000 2024-09-18T14:22:14.490 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:22:14.490+0000 7ff7280d7780 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/2: (21) Is a directory 2024-09-18T14:22:14.491 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:22:14.490+0000 7ff7280d7780 -1 bluestore(td/osd-bluefs-volume-ops/2) unable to read label for td/osd-bluefs-volume-ops/2: (21) Is a directory 2024-09-18T14:22:19.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:79: TEST_bluestore: truncate td/osd-bluefs-volume-ops/3/block -s 4293918720 2024-09-18T14:22:19.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:80: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 bluefs-bdev-expand 2024-09-18T14:22:19.666 INFO:tasks.workunit.client.0.smithi047.stdout:inferring bluefs devices from bluestore path 2024-09-18T14:22:20.580 INFO:tasks.workunit.client.0.smithi047.stdout:0 : device size 0x1ffff000 : using 0x11e00000(286 MiB) 2024-09-18T14:22:20.580 INFO:tasks.workunit.client.0.smithi047.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-18T14:22:20.580 INFO:tasks.workunit.client.0.smithi047.stdout:2 : device size 0xfff00000 : using 0x80428000(2.0 GiB) 2024-09-18T14:22:20.580 INFO:tasks.workunit.client.0.smithi047.stdout:Expanding DB/WAL... 2024-09-18T14:22:20.580 INFO:tasks.workunit.client.0.smithi047.stdout:0 : expanding to 0x536870912 2024-09-18T14:22:20.605 INFO:tasks.workunit.client.0.smithi047.stdout:0 : size label updated to 536870912 2024-09-18T14:22:20.605 INFO:tasks.workunit.client.0.smithi047.stdout:1 : expanding to 0x1073741824 2024-09-18T14:22:20.630 INFO:tasks.workunit.client.0.smithi047.stdout:1 : size label updated to 1073741824 2024-09-18T14:22:20.630 INFO:tasks.workunit.client.0.smithi047.stdout:2 : expanding from 0x80000000 to 0xfff00000 2024-09-18T14:22:20.806 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:22:20.805+0000 7f505000a780 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/3: (21) Is a directory 2024-09-18T14:22:20.806 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:22:20.805+0000 7f505000a780 -1 bluestore(td/osd-bluefs-volume-ops/3) unable to read label for td/osd-bluefs-volume-ops/3: (21) Is a directory 2024-09-18T14:22:26.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:83: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-18T14:22:28.082 INFO:tasks.workunit.client.0.smithi047.stdout:fsck success 2024-09-18T14:22:28.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:84: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-18T14:22:29.873 INFO:tasks.workunit.client.0.smithi047.stdout:fsck success 2024-09-18T14:22:29.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:85: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-18T14:22:31.668 INFO:tasks.workunit.client.0.smithi047.stdout:fsck success 2024-09-18T14:22:31.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:86: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-18T14:22:33.462 INFO:tasks.workunit.client.0.smithi047.stdout:fsck success 2024-09-18T14:22:33.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:88: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-sizes 2024-09-18T14:22:33.491 INFO:tasks.workunit.client.0.smithi047.stdout:inferring bluefs devices from bluestore path 2024-09-18T14:22:35.259 INFO:tasks.workunit.client.0.smithi047.stdout:0 : device size 0x1ffff000 : using 0x11e00000(286 MiB) 2024-09-18T14:22:35.260 INFO:tasks.workunit.client.0.smithi047.stdout:1 : device size 0x3fffe000 : using 0x1500000(21 MiB) 2024-09-18T14:22:35.260 INFO:tasks.workunit.client.0.smithi047.stdout:2 : device size 0x100000000 : using 0x53c000(5.2 MiB) 2024-09-18T14:22:35.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:90: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/block.db --command bluefs-bdev-migrate 2024-09-18T14:22:35.279 INFO:tasks.workunit.client.0.smithi047.stdout:inferring bluefs devices from bluestore path 2024-09-18T14:22:39.044 INFO:tasks.workunit.client.0.smithi047.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2024-09-18T14:22:39.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:95: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-18T14:22:40.352 INFO:tasks.workunit.client.0.smithi047.stdout:fsck success 2024-09-18T14:22:40.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:98: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2024-09-18T14:22:40.380 INFO:tasks.workunit.client.0.smithi047.stdout:inferring bluefs devices from bluestore path 2024-09-18T14:22:42.393 INFO:tasks.workunit.client.0.smithi047.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-09-18T14:22:42.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:103: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-18T14:22:43.678 INFO:tasks.workunit.client.0.smithi047.stdout:fsck success 2024-09-18T14:22:43.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:106: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block.wal --devs-source td/osd-bluefs-volume-ops/2/block.db --dev-target td/osd-bluefs-volume-ops/2/block --command bluefs-bdev-migrate 2024-09-18T14:22:43.703 INFO:tasks.workunit.client.0.smithi047.stdout:inferring bluefs devices from bluestore path 2024-09-18T14:22:45.723 INFO:tasks.workunit.client.0.smithi047.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2024-09-18T14:22:45.723 INFO:tasks.workunit.client.0.smithi047.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2024-09-18T14:22:45.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:112: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-18T14:22:46.509 INFO:tasks.workunit.client.0.smithi047.stdout:fsck success 2024-09-18T14:22:46.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:115: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block.wal --command bluefs-bdev-migrate 2024-09-18T14:22:46.536 INFO:tasks.workunit.client.0.smithi047.stdout:inferring bluefs devices from bluestore path 2024-09-18T14:22:46.536 INFO:tasks.workunit.client.0.smithi047.stderr:Migrate to WAL device isn't supported. 2024-09-18T14:22:46.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:121: TEST_bluestore: '[' 1 -eq 0 ']' 2024-09-18T14:22:46.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:124: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-18T14:22:48.567 INFO:tasks.workunit.client.0.smithi047.stdout:fsck success 2024-09-18T14:22:48.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:127: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2024-09-18T14:22:48.596 INFO:tasks.workunit.client.0.smithi047.stdout:inferring bluefs devices from bluestore path 2024-09-18T14:22:52.116 INFO:tasks.workunit.client.0.smithi047.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2024-09-18T14:22:52.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:132: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-18T14:22:53.425 INFO:tasks.workunit.client.0.smithi047.stdout:fsck success 2024-09-18T14:22:53.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:134: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --dev-target td/osd-bluefs-volume-ops/3/block.db --command bluefs-bdev-migrate 2024-09-18T14:22:53.451 INFO:tasks.workunit.client.0.smithi047.stdout:inferring bluefs devices from bluestore path 2024-09-18T14:22:56.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:139: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-18T14:22:58.026 INFO:tasks.workunit.client.0.smithi047.stdout:fsck success 2024-09-18T14:22:58.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:141: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-09-18T14:22:58.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:22:58.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:22:58.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T14:22:58.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:22:58.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-18T14:22:58.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-18T14:22:58.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:22:58.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:22:58.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:22:58.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-18T14:22:58.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-18T14:22:58.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:22:58.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:22:58.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-18T14:22:58.041 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:22:58.041 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:22:58.041 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:22:58.041 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:22:58.041 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:22:58.041 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:22:58.041 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:22:58.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:22:58.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:22:58.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-18T14:22:58.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-18T14:22:58.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:22:58.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:22:58.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:22:58.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:22:58.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:22:58.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:22:58.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-18T14:22:58.042 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T14:22:58.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T14:22:58.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:22:58.043 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-18T14:22:58.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T14:22:58.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:22:58.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:22:58.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:22:58.064 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:22:58.064+0000 7f5ae3620440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:22:58.067 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:22:58.066+0000 7f5ae3620440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:22:58.069 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:22:58.068+0000 7f5ae3620440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:22:58.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T14:22:58.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:22:58.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:22:58.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:22:58.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:22:58.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:22:58.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:22:58.415 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:22:58.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:22:58.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:22:58.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:22:58.836 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:22:58.836+0000 7f5ae3620440 -1 Falling back to public interface 2024-09-18T14:22:59.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:22:59.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:22:59.816 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:22:59.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:22:59.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:22:59.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:23:00.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:23:01.194 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:23:01.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:23:01.195 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:23:01.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:23:01.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:23:01.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:23:01.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:23:02.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:23:02.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:23:02.567 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:23:02.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:23:02.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:23:02.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:23:02.609 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:23:02.608+0000 7f5ae3620440 -1 osd.0 26 log_to_monitors true 2024-09-18T14:23:03.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:23:04.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:23:04.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:23:04.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:23:04.005 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:23:04.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:23:04.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:23:04.425 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 36 up_thru 36 down_at 27 last_clean_interval [5,26) [v2:127.0.0.1:6802/124971,v1:127.0.0.1:6803/124971] [v2:127.0.0.1:6804/124971,v1:127.0.0.1:6805/124971] exists,up 77e514a7-b7cf-4967-acb8-f65677560a17 2024-09-18T14:23:04.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:23:04.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:23:04.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:23:04.425 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-18T14:23:04.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: osd_pid0=124971 2024-09-18T14:23:04.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:143: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-09-18T14:23:04.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:23:04.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:23:04.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T14:23:04.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:23:04.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-18T14:23:04.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-18T14:23:04.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:23:04.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:23:04.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:23:04.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-18T14:23:04.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-18T14:23:04.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:23:04.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:23:04.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-18T14:23:04.430 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:23:04.430 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:23:04.430 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:23:04.430 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:23:04.430 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:23:04.430 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:23:04.430 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:23:04.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:23:04.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:23:04.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-18T14:23:04.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-18T14:23:04.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:23:04.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:23:04.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:23:04.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:23:04.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:23:04.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:23:04.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-18T14:23:04.431 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T14:23:04.432 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T14:23:04.432 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:23:04.432 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-09-18T14:23:04.432 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T14:23:04.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:23:04.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:23:04.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:23:04.451 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:23:04.450+0000 7f41e1822440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:23:04.453 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:23:04.452+0000 7f41e1822440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:23:04.454 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:23:04.453+0000 7f41e1822440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:23:04.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T14:23:04.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:23:04.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:23:04.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:23:04.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:23:04.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:23:04.844 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:23:04.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:23:04.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:23:04.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:23:05.223 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:23:05.223+0000 7f41e1822440 -1 Falling back to public interface 2024-09-18T14:23:05.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:23:06.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:23:06.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:23:06.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:23:06.238 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:23:06.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:23:06.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:23:06.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:23:07.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:23:07.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:23:07.647 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:23:07.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:23:07.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:23:07.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:23:08.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:23:09.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:23:09.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:23:09.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:23:09.017 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:23:09.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:23:09.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:23:09.244 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:23:09.244+0000 7f41e1822440 -1 osd.1 28 log_to_monitors true 2024-09-18T14:23:09.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:23:10.419 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:23:10.420 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:23:10.420 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:23:10.420 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:23:10.420 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:23:10.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:23:10.840 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 39 up_thru 39 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/125952,v1:127.0.0.1:6811/125952] [v2:127.0.0.1:6812/125952,v1:127.0.0.1:6813/125952] exists,up 8164e048-2490-4c40-b72e-815a1382b89d 2024-09-18T14:23:10.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:23:10.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:23:10.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:23:10.841 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-18T14:23:10.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: osd_pid1=125952 2024-09-18T14:23:10.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:145: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-09-18T14:23:10.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:23:10.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:23:10.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T14:23:10.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:23:10.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-18T14:23:10.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-18T14:23:10.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:23:10.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:23:10.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:23:10.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-18T14:23:10.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-18T14:23:10.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:23:10.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:23:10.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-18T14:23:10.846 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:23:10.846 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:23:10.846 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:23:10.846 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:23:10.846 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:23:10.846 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:23:10.846 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:23:10.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:23:10.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:23:10.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-18T14:23:10.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-18T14:23:10.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:23:10.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:23:10.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:23:10.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:23:10.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:23:10.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:23:10.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-18T14:23:10.847 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T14:23:10.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T14:23:10.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:23:10.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-09-18T14:23:10.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T14:23:10.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:23:10.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:23:10.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:23:10.866 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:23:10.865+0000 7f8b39619440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:23:10.868 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:23:10.868+0000 7f8b39619440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:23:10.869 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:23:10.869+0000 7f8b39619440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:23:11.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T14:23:11.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:23:11.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T14:23:11.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:23:11.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:23:11.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:23:11.244 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:23:11.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:23:11.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:23:11.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:23:11.390 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:23:11.390+0000 7f8b39619440 -1 Falling back to public interface 2024-09-18T14:23:11.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:23:12.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:23:12.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:23:12.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:23:12.654 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:23:12.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:23:12.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:23:13.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:23:14.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:23:14.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:23:14.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:23:14.120 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:23:14.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:23:14.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:23:14.230 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:23:14.229+0000 7f8b39619440 -1 osd.2 31 log_to_monitors true 2024-09-18T14:23:14.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:23:15.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:23:15.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:23:15.598 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:23:15.599 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:23:15.599 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:23:15.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:23:16.004 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 42 up_thru 42 down_at 32 last_clean_interval [15,31) [v2:127.0.0.1:6818/126933,v1:127.0.0.1:6819/126933] [v2:127.0.0.1:6820/126933,v1:127.0.0.1:6821/126933] exists,up c58dcf9b-e2da-4e51-9972-4ac5044eccd6 2024-09-18T14:23:16.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:23:16.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:23:16.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:23:16.005 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-18T14:23:16.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: osd_pid2=126933 2024-09-18T14:23:16.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:147: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-09-18T14:23:16.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:23:16.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:23:16.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-18T14:23:16.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:23:16.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-18T14:23:16.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-18T14:23:16.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:23:16.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:23:16.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:23:16.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-18T14:23:16.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-18T14:23:16.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:23:16.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:23:16.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-18T14:23:16.010 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:23:16.010 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:23:16.010 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:23:16.010 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:23:16.010 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:23:16.010 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:23:16.010 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:23:16.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:23:16.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:23:16.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-18T14:23:16.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-18T14:23:16.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:23:16.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:23:16.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:23:16.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:23:16.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:23:16.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:23:16.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-18T14:23:16.011 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.3 2024-09-18T14:23:16.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-18T14:23:16.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:23:16.012 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-09-18T14:23:16.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-18T14:23:16.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:23:16.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:23:16.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:23:16.030 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:23:16.029+0000 7f0de8a29440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:23:16.032 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:23:16.031+0000 7f0de8a29440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:23:16.033 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:23:16.032+0000 7f0de8a29440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:23:16.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-18T14:23:16.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:23:16.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T14:23:16.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:23:16.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:23:16.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:23:16.373 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:23:16.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:23:16.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:23:16.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:23:16.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:23:16.813 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:23:16.813+0000 7f0de8a29440 -1 Falling back to public interface 2024-09-18T14:23:17.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:23:17.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:23:17.784 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:23:17.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:23:17.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:23:17.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:23:18.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:23:19.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:23:19.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:23:19.244 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:23:19.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:23:19.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:23:19.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:23:19.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:23:20.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:23:20.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:23:20.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:23:20.672 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:23:20.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:23:20.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:23:20.797 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:23:20.796+0000 7f0de8a29440 -1 osd.3 33 log_to_monitors true 2024-09-18T14:23:21.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:23:22.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:23:22.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:23:22.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:23:22.084 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:23:22.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:23:22.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:23:22.582 INFO:tasks.workunit.client.0.smithi047.stdout:osd.3 up in weight 1 up_from 46 up_thru 46 down_at 34 last_clean_interval [20,33) [v2:127.0.0.1:6826/127879,v1:127.0.0.1:6827/127879] [v2:127.0.0.1:6828/127879,v1:127.0.0.1:6829/127879] exists,up 7dc3b3b6-05fa-4664-93b4-abdebdb06ac0 2024-09-18T14:23:22.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:23:22.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:23:22.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:23:22.583 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-18T14:23:22.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: osd_pid3=127879 2024-09-18T14:23:22.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:150: TEST_bluestore: wait_for_clean 2024-09-18T14:23:22.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:23:22.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:23:22.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:23:22.585 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:23:22.585 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:23:22.585 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:23:22.585 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:23:22.585 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:23:22.585 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:23:22.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T14:23:22.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:23:22.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:23:22.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:23:22.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:23:22.674 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:23:23.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:23:23.060 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T14:23:23.060 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T14:23:23.060 INFO:tasks.workunit.client.0.smithi047.stderr:3' 2024-09-18T14:23:23.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:23:23.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:23:23.060 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:23:23.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=154618822663 2024-09-18T14:23:23.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 154618822663 2024-09-18T14:23:23.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-154618822663' 2024-09-18T14:23:23.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:23:23.175 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:23:23.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=167503724549 2024-09-18T14:23:23.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 167503724549 2024-09-18T14:23:23.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-154618822663 1-167503724549' 2024-09-18T14:23:23.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:23:23.284 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:23:23.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=180388626436 2024-09-18T14:23:23.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 180388626436 2024-09-18T14:23:23.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-154618822663 1-167503724549 2-180388626436' 2024-09-18T14:23:23.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:23:23.418 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T14:23:23.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=197568495619 2024-09-18T14:23:23.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 197568495619 2024-09-18T14:23:23.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-154618822663 1-167503724549 2-180388626436 3-197568495619' 2024-09-18T14:23:23.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:23:23.527 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-154618822663 2024-09-18T14:23:23.527 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:23:23.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:23:23.529 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-154618822663 2024-09-18T14:23:23.529 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:23:23.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=154618822663 2024-09-18T14:23:23.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 154618822663' 2024-09-18T14:23:23.530 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 154618822663 2024-09-18T14:23:23.530 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:23:23.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 154618822661 -lt 154618822663 2024-09-18T14:23:23.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:23:24.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:23:24.930 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:23:25.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 154618822663 -lt 154618822663 2024-09-18T14:23:25.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:23:25.307 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-167503724549 2024-09-18T14:23:25.308 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:23:25.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:23:25.309 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-167503724549 2024-09-18T14:23:25.309 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:23:25.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=167503724549 2024-09-18T14:23:25.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 167503724549' 2024-09-18T14:23:25.310 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 167503724549 2024-09-18T14:23:25.311 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:23:25.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 167503724550 -lt 167503724549 2024-09-18T14:23:25.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:23:25.680 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-180388626436 2024-09-18T14:23:25.680 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:23:25.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:23:25.682 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-180388626436 2024-09-18T14:23:25.682 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:23:25.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=180388626436 2024-09-18T14:23:25.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 180388626436' 2024-09-18T14:23:25.683 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 180388626436 2024-09-18T14:23:25.683 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:23:26.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 180388626437 -lt 180388626436 2024-09-18T14:23:26.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:23:26.047 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-197568495619 2024-09-18T14:23:26.047 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:23:26.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T14:23:26.049 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-197568495619 2024-09-18T14:23:26.049 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:23:26.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=197568495619 2024-09-18T14:23:26.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 197568495619' 2024-09-18T14:23:26.050 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 197568495619 2024-09-18T14:23:26.050 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T14:23:26.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 197568495619 -lt 197568495619 2024-09-18T14:23:26.412 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:23:26.412 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:23:26.412 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:23:26.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-09-18T14:23:26.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:23:26.905 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:23:26.905 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:23:26.905 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:23:26.905 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:23:26.905 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:23:26.905 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:23:27.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-09-18T14:23:27.307 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:23:27.307 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:23:27.307 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:23:27.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-09-18T14:23:27.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:23:27.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:23:27.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:153: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-18T14:23:27.857 INFO:tasks.workunit.client.0.smithi047.stdout:hints = 1 2024-09-18T14:23:27.857 INFO:tasks.workunit.client.0.smithi047.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-18T14:23:27.857 INFO:tasks.workunit.client.0.smithi047.stdout:Object prefix: benchmark_data_smithi047_129228 2024-09-18T14:23:27.857 INFO:tasks.workunit.client.0.smithi047.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T14:23:27.858 INFO:tasks.workunit.client.0.smithi047.stdout: 0 0 0 0 0 0 - 0 2024-09-18T14:23:28.858 INFO:tasks.workunit.client.0.smithi047.stdout: 1 16 81 65 0.253871 0.253906 0.194685 0.218647 2024-09-18T14:23:29.857 INFO:tasks.workunit.client.0.smithi047.stdout: 2 16 153 137 0.267547 0.28125 0.168014 0.223468 2024-09-18T14:23:30.857 INFO:tasks.workunit.client.0.smithi047.stdout: 3 16 229 213 0.277313 0.296875 0.175147 0.217573 2024-09-18T14:23:31.858 INFO:tasks.workunit.client.0.smithi047.stdout: 4 16 302 286 0.279266 0.285156 0.159455 0.216594 2024-09-18T14:23:32.858 INFO:tasks.workunit.client.0.smithi047.stdout: 5 16 391 375 0.292937 0.347656 0.17129 0.208845 2024-09-18T14:23:33.858 INFO:tasks.workunit.client.0.smithi047.stdout: 6 16 447 431 0.280567 0.21875 0.251191 0.215469 2024-09-18T14:23:34.858 INFO:tasks.workunit.client.0.smithi047.stdout: 7 16 505 489 0.272847 0.226562 0.251837 0.221855 2024-09-18T14:23:35.858 INFO:tasks.workunit.client.0.smithi047.stdout: 8 16 565 549 0.268033 0.234375 0.327524 0.229593 2024-09-18T14:23:36.858 INFO:tasks.workunit.client.0.smithi047.stdout: 9 16 627 611 0.265158 0.242188 0.279433 0.232051 2024-09-18T14:23:37.858 INFO:tasks.workunit.client.0.smithi047.stdout: 10 16 690 674 0.263248 0.246094 0.268672 0.235075 2024-09-18T14:23:38.858 INFO:tasks.workunit.client.0.smithi047.stdout: 11 16 740 724 0.257071 0.195312 0.310242 0.240383 2024-09-18T14:23:39.859 INFO:tasks.workunit.client.0.smithi047.stdout: 12 16 802 786 0.255827 0.242188 0.302024 0.241676 2024-09-18T14:23:40.859 INFO:tasks.workunit.client.0.smithi047.stdout: 13 16 863 847 0.254475 0.238281 0.19558 0.24295 2024-09-18T14:23:41.859 INFO:tasks.workunit.client.0.smithi047.stdout: 14 16 919 903 0.251921 0.21875 0.168191 0.245184 2024-09-18T14:23:42.859 INFO:tasks.workunit.client.0.smithi047.stdout: 15 16 981 965 0.251269 0.242188 0.243176 0.246217 2024-09-18T14:23:43.859 INFO:tasks.workunit.client.0.smithi047.stdout: 16 16 1043 1027 0.250699 0.242188 0.361936 0.246213 2024-09-18T14:23:44.859 INFO:tasks.workunit.client.0.smithi047.stdout: 17 16 1120 1104 0.253643 0.300781 0.176281 0.245381 2024-09-18T14:23:45.860 INFO:tasks.workunit.client.0.smithi047.stdout: 18 16 1201 1185 0.257128 0.316406 0.182914 0.242262 2024-09-18T14:23:46.860 INFO:tasks.workunit.client.0.smithi047.stdout: 19 16 1276 1260 0.259012 0.292969 0.209264 0.239521 2024-09-18T14:23:47.860 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T14:23:47.860543+0000 min lat: 0.100696 max lat: 0.477093 avg lat: 0.238029 2024-09-18T14:23:47.860 INFO:tasks.workunit.client.0.smithi047.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T14:23:47.860 INFO:tasks.workunit.client.0.smithi047.stdout: 20 16 1353 1337 0.261099 0.300781 0.210039 0.238029 2024-09-18T14:23:48.861 INFO:tasks.workunit.client.0.smithi047.stdout: 21 16 1436 1420 0.264103 0.324219 0.21125 0.235307 2024-09-18T14:23:49.860 INFO:tasks.workunit.client.0.smithi047.stdout: 22 16 1491 1475 0.261862 0.214844 0.254579 0.23679 2024-09-18T14:23:50.860 INFO:tasks.workunit.client.0.smithi047.stdout: 23 16 1571 1555 0.264062 0.3125 0.186445 0.235479 2024-09-18T14:23:51.860 INFO:tasks.workunit.client.0.smithi047.stdout: 24 16 1657 1641 0.267056 0.335938 0.268099 0.233419 2024-09-18T14:23:52.860 INFO:tasks.workunit.client.0.smithi047.stdout: 25 16 1728 1712 0.267466 0.277344 0.127145 0.233072 2024-09-18T14:23:53.860 INFO:tasks.workunit.client.0.smithi047.stdout: 26 16 1800 1784 0.267995 0.28125 0.210416 0.23176 2024-09-18T14:23:54.861 INFO:tasks.workunit.client.0.smithi047.stdout: 27 16 1888 1872 0.270799 0.34375 0.227847 0.229931 2024-09-18T14:23:55.861 INFO:tasks.workunit.client.0.smithi047.stdout: 28 16 1958 1942 0.270892 0.273438 0.225628 0.229766 2024-09-18T14:23:56.861 INFO:tasks.workunit.client.0.smithi047.stdout: 29 16 2035 2019 0.271921 0.300781 0.225709 0.229054 2024-09-18T14:23:57.861 INFO:tasks.workunit.client.0.smithi047.stdout: 30 16 2108 2092 0.272361 0.285156 0.183795 0.228745 2024-09-18T14:23:58.861 INFO:tasks.workunit.client.0.smithi047.stdout:Total time run: 30.1121 2024-09-18T14:23:58.861 INFO:tasks.workunit.client.0.smithi047.stdout:Total writes made: 2108 2024-09-18T14:23:58.862 INFO:tasks.workunit.client.0.smithi047.stdout:Write size: 4096 2024-09-18T14:23:58.862 INFO:tasks.workunit.client.0.smithi047.stdout:Object size: 4096 2024-09-18T14:23:58.862 INFO:tasks.workunit.client.0.smithi047.stdout:Bandwidth (MB/sec): 0.273458 2024-09-18T14:23:58.862 INFO:tasks.workunit.client.0.smithi047.stdout:Stddev Bandwidth: 0.0412626 2024-09-18T14:23:58.862 INFO:tasks.workunit.client.0.smithi047.stdout:Max bandwidth (MB/sec): 0.347656 2024-09-18T14:23:58.862 INFO:tasks.workunit.client.0.smithi047.stdout:Min bandwidth (MB/sec): 0.195312 2024-09-18T14:23:58.862 INFO:tasks.workunit.client.0.smithi047.stdout:Average IOPS: 70 2024-09-18T14:23:58.862 INFO:tasks.workunit.client.0.smithi047.stdout:Stddev IOPS: 10.5632 2024-09-18T14:23:58.862 INFO:tasks.workunit.client.0.smithi047.stdout:Max IOPS: 89 2024-09-18T14:23:58.862 INFO:tasks.workunit.client.0.smithi047.stdout:Min IOPS: 50 2024-09-18T14:23:58.862 INFO:tasks.workunit.client.0.smithi047.stdout:Average Latency(s): 0.22829 2024-09-18T14:23:58.862 INFO:tasks.workunit.client.0.smithi047.stdout:Stddev Latency(s): 0.0553955 2024-09-18T14:23:58.862 INFO:tasks.workunit.client.0.smithi047.stdout:Max latency(s): 0.477093 2024-09-18T14:23:58.862 INFO:tasks.workunit.client.0.smithi047.stdout:Min latency(s): 0.100696 2024-09-18T14:23:58.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 124971 2024-09-18T14:23:58.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: sleep 1 2024-09-18T14:23:59.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 124971 2024-09-18T14:23:59.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 156: kill: (124971) - No such process 2024-09-18T14:23:59.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:157: TEST_bluestore: ceph osd down 0 2024-09-18T14:24:00.479 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already down. 2024-09-18T14:24:00.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 125952 2024-09-18T14:24:00.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: sleep 1 2024-09-18T14:24:01.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 125952 2024-09-18T14:24:01.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 158: kill: (125952) - No such process 2024-09-18T14:24:01.491 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:159: TEST_bluestore: ceph osd down 1 2024-09-18T14:24:01.917 INFO:tasks.workunit.client.0.smithi047.stderr:osd.1 is already down. 2024-09-18T14:24:01.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 126933 2024-09-18T14:24:01.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: sleep 1 2024-09-18T14:24:02.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 126933 2024-09-18T14:24:02.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 160: kill: (126933) - No such process 2024-09-18T14:24:02.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:161: TEST_bluestore: ceph osd down 2 2024-09-18T14:24:03.337 INFO:tasks.workunit.client.0.smithi047.stderr:osd.2 is already down. 2024-09-18T14:24:03.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 127879 2024-09-18T14:24:03.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: sleep 1 2024-09-18T14:24:04.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 127879 2024-09-18T14:24:04.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 162: kill: (127879) - No such process 2024-09-18T14:24:04.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:163: TEST_bluestore: ceph osd down 3 2024-09-18T14:24:04.848 INFO:tasks.workunit.client.0.smithi047.stderr:osd.3 is already down. 2024-09-18T14:24:04.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:166: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-18T14:24:06.389 INFO:tasks.workunit.client.0.smithi047.stdout:fsck success 2024-09-18T14:24:06.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:168: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal count=512 bs=1M 2024-09-18T14:24:06.661 INFO:tasks.workunit.client.0.smithi047.stderr:512+0 records in 2024-09-18T14:24:06.661 INFO:tasks.workunit.client.0.smithi047.stderr:512+0 records out 2024-09-18T14:24:06.661 INFO:tasks.workunit.client.0.smithi047.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.259573 s, 2.1 GB/s 2024-09-18T14:24:06.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:169: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --dev-target td/osd-bluefs-volume-ops/0/wal --command bluefs-bdev-new-wal 2024-09-18T14:24:06.676 INFO:tasks.workunit.client.0.smithi047.stdout:inferring bluefs devices from bluestore path 2024-09-18T14:24:11.441 INFO:tasks.workunit.client.0.smithi047.stdout:WAL device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal 2024-09-18T14:24:11.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:173: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-18T14:24:13.226 INFO:tasks.workunit.client.0.smithi047.stdout:fsck success 2024-09-18T14:24:13.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:176: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-18T14:24:14.526 INFO:tasks.workunit.client.0.smithi047.stdout:fsck success 2024-09-18T14:24:14.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:178: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db count=1024 bs=1M 2024-09-18T14:24:15.032 INFO:tasks.workunit.client.0.smithi047.stderr:1024+0 records in 2024-09-18T14:24:15.032 INFO:tasks.workunit.client.0.smithi047.stderr:1024+0 records out 2024-09-18T14:24:15.032 INFO:tasks.workunit.client.0.smithi047.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.498827 s, 2.2 GB/s 2024-09-18T14:24:15.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:179: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --dev-target td/osd-bluefs-volume-ops/1/db --command bluefs-bdev-new-db 2024-09-18T14:24:15.048 INFO:tasks.workunit.client.0.smithi047.stdout:inferring bluefs devices from bluestore path 2024-09-18T14:24:22.564 INFO:tasks.workunit.client.0.smithi047.stdout:DB device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db 2024-09-18T14:24:22.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:183: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block --dev-target td/osd-bluefs-volume-ops/1/block.db --command bluefs-bdev-migrate 2024-09-18T14:24:22.585 INFO:tasks.workunit.client.0.smithi047.stdout:inferring bluefs devices from bluestore path 2024-09-18T14:24:24.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:188: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-18T14:24:26.389 INFO:tasks.workunit.client.0.smithi047.stdout:fsck success 2024-09-18T14:24:26.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:191: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-18T14:24:27.431 INFO:tasks.workunit.client.0.smithi047.stdout:fsck success 2024-09-18T14:24:27.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:193: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-db 2024-09-18T14:24:27.455 INFO:tasks.workunit.client.0.smithi047.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-09-18T14:24:27.455 INFO:tasks.workunit.client.0.smithi047.stdout:inferring bluefs devices from bluestore path 2024-09-18T14:24:28.468 INFO:tasks.workunit.client.0.smithi047.stdout:DB device added 2024-09-18T14:24:28.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:196: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-wal 2024-09-18T14:24:28.487 INFO:tasks.workunit.client.0.smithi047.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-09-18T14:24:28.487 INFO:tasks.workunit.client.0.smithi047.stdout:inferring bluefs devices from bluestore path 2024-09-18T14:24:29.750 INFO:tasks.workunit.client.0.smithi047.stdout:WAL device added 2024-09-18T14:24:29.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:199: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/block.db --command bluefs-bdev-migrate 2024-09-18T14:24:29.773 INFO:tasks.workunit.client.0.smithi047.stdout:inferring bluefs devices from bluestore path 2024-09-18T14:24:35.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:204: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-18T14:24:37.352 INFO:tasks.workunit.client.0.smithi047.stdout:fsck success 2024-09-18T14:24:37.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:207: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-18T14:24:38.645 INFO:tasks.workunit.client.0.smithi047.stdout:fsck success 2024-09-18T14:24:38.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:209: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --command bluefs-bdev-new-wal 2024-09-18T14:24:38.671 INFO:tasks.workunit.client.0.smithi047.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-09-18T14:24:38.671 INFO:tasks.workunit.client.0.smithi047.stdout:inferring bluefs devices from bluestore path 2024-09-18T14:24:40.183 INFO:tasks.workunit.client.0.smithi047.stdout:WAL device added 2024-09-18T14:24:40.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:212: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2024-09-18T14:24:40.205 INFO:tasks.workunit.client.0.smithi047.stdout:inferring bluefs devices from bluestore path 2024-09-18T14:24:43.972 INFO:tasks.workunit.client.0.smithi047.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block.db 2024-09-18T14:24:43.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:217: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-18T14:24:45.280 INFO:tasks.workunit.client.0.smithi047.stdout:fsck success 2024-09-18T14:24:45.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:219: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-09-18T14:24:45.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:24:45.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:24:45.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T14:24:45.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:24:45.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-18T14:24:45.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-18T14:24:45.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:24:45.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:24:45.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:24:45.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-18T14:24:45.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-18T14:24:45.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:24:45.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:24:45.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-18T14:24:45.293 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:24:45.294 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:24:45.294 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:24:45.294 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:24:45.294 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:24:45.294 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:24:45.294 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:24:45.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:24:45.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:24:45.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-18T14:24:45.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-18T14:24:45.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:24:45.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:24:45.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:24:45.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:24:45.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:24:45.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:24:45.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-18T14:24:45.295 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T14:24:45.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T14:24:45.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:24:45.296 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-18T14:24:45.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T14:24:45.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:24:45.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:24:45.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:24:45.318 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:24:45.317+0000 7f1b9891d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:24:45.320 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:24:45.320+0000 7f1b9891d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:24:45.322 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:24:45.321+0000 7f1b9891d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:24:45.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T14:24:45.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:24:45.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:24:45.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:24:45.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:24:45.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:45.686 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:24:45.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:24:45.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:45.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:24:46.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:46.338 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:24:46.338+0000 7f1b9891d440 -1 Falling back to public interface 2024-09-18T14:24:47.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:47.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:47.063 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:24:47.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:24:47.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:47.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:24:47.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:48.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:48.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:48.445 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:24:48.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:24:48.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:48.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:24:48.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:49.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:49.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:49.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:24:49.874 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:24:49.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:49.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:24:50.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:50.910 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:24:50.909+0000 7f1b9891d440 -1 osd.0 48 log_to_monitors true 2024-09-18T14:24:51.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:51.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:51.292 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:24:51.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:24:51.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:51.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:24:51.689 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 58 up_thru 58 down_at 49 last_clean_interval [36,48) [v2:127.0.0.1:6802/135414,v1:127.0.0.1:6803/135414] [v2:127.0.0.1:6804/135414,v1:127.0.0.1:6805/135414] exists,up 77e514a7-b7cf-4967-acb8-f65677560a17 2024-09-18T14:24:51.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:24:51.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:24:51.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:24:51.693 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-18T14:24:51.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: osd_pid0=135414 2024-09-18T14:24:51.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:221: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-09-18T14:24:51.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:24:51.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:24:51.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T14:24:51.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:24:51.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-18T14:24:51.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-18T14:24:51.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:24:51.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:24:51.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:24:51.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-18T14:24:51.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-18T14:24:51.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:24:51.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:24:51.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-18T14:24:51.694 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:24:51.694 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:24:51.694 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:24:51.694 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:24:51.694 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:24:51.695 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:24:51.695 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:24:51.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:24:51.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:24:51.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-18T14:24:51.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-18T14:24:51.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:24:51.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:24:51.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:24:51.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:24:51.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:24:51.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:24:51.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-18T14:24:51.696 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T14:24:51.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T14:24:51.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:24:51.697 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-09-18T14:24:51.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T14:24:51.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:24:51.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:24:51.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:24:51.712 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:24:51.712+0000 7eff7b353440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:24:51.734 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:24:51.733+0000 7eff7b353440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:24:51.735 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:24:51.734+0000 7eff7b353440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:24:52.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T14:24:52.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:24:52.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:24:52.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:24:52.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:24:52.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:52.131 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:24:52.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:24:52.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:52.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:24:52.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:52.753 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:24:52.753+0000 7eff7b353440 -1 Falling back to public interface 2024-09-18T14:24:53.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:53.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:53.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:24:53.550 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:24:53.551 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:53.551 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:24:53.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:54.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:54.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:54.928 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:24:54.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:24:54.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:54.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:24:55.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:56.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:56.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:56.361 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:24:56.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:24:56.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:56.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:24:56.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:57.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:57.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:57.734 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:24:57.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:24:57.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:57.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:24:57.854 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:24:57.854+0000 7eff7b353440 -1 osd.1 51 log_to_monitors true 2024-09-18T14:24:58.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:24:59.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:24:59.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:59.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:24:59.178 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:24:59.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:59.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:24:59.590 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 61 up_thru 61 down_at 52 last_clean_interval [39,51) [v2:127.0.0.1:6810/136405,v1:127.0.0.1:6811/136405] [v2:127.0.0.1:6812/136405,v1:127.0.0.1:6813/136405] exists,up 8164e048-2490-4c40-b72e-815a1382b89d 2024-09-18T14:24:59.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:24:59.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:24:59.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:24:59.591 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-18T14:24:59.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: osd_pid1=136405 2024-09-18T14:24:59.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:223: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-09-18T14:24:59.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:24:59.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:24:59.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T14:24:59.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:24:59.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-18T14:24:59.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-18T14:24:59.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:24:59.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:24:59.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:24:59.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-18T14:24:59.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-18T14:24:59.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:24:59.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:24:59.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-18T14:24:59.596 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:24:59.596 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:24:59.596 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:24:59.596 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:24:59.596 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:24:59.596 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:24:59.596 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:24:59.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:24:59.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:24:59.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-18T14:24:59.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-18T14:24:59.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:24:59.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:24:59.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:24:59.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:24:59.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:24:59.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:24:59.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-18T14:24:59.597 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T14:24:59.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T14:24:59.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:24:59.598 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-09-18T14:24:59.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T14:24:59.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:24:59.599 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:24:59.599 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:24:59.617 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:24:59.616+0000 7f275d388440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:24:59.618 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:24:59.618+0000 7f275d388440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:24:59.619 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:24:59.619+0000 7f275d388440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:24:59.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T14:24:59.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:24:59.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T14:24:59.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:24:59.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:24:59.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:24:59.964 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:24:59.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:24:59.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:24:59.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:25:00.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:00.635 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:25:00.635+0000 7f275d388440 -1 Falling back to public interface 2024-09-18T14:25:01.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:01.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:01.370 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:25:01.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:25:01.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:01.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:25:01.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:02.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:02.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:02.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:25:02.789 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:25:02.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:02.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:25:03.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:04.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:04.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:04.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:25:04.225 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:25:04.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:04.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:25:04.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:05.642 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:05.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:05.643 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:25:05.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:25:05.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:05.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:25:05.966 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:25:05.966+0000 7f275d388440 -1 osd.2 53 log_to_monitors true 2024-09-18T14:25:06.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:07.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:07.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:07.026 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:25:07.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:25:07.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:07.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:25:07.682 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 64 up_thru 64 down_at 54 last_clean_interval [42,53) [v2:127.0.0.1:6818/137422,v1:127.0.0.1:6819/137422] [v2:127.0.0.1:6820/137422,v1:127.0.0.1:6821/137422] exists,up c58dcf9b-e2da-4e51-9972-4ac5044eccd6 2024-09-18T14:25:07.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:25:07.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:25:07.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:25:07.690 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-18T14:25:07.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: osd_pid2=137422 2024-09-18T14:25:07.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:225: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-09-18T14:25:07.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:25:07.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:25:07.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-18T14:25:07.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:25:07.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-18T14:25:07.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-18T14:25:07.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:25:07.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:25:07.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:25:07.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-18T14:25:07.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-18T14:25:07.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:25:07.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:25:07.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-18T14:25:07.692 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:25:07.693 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:25:07.693 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:25:07.693 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:25:07.693 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:25:07.693 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:25:07.693 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:25:07.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:25:07.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:25:07.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-18T14:25:07.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-18T14:25:07.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:25:07.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:25:07.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:25:07.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:25:07.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:25:07.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:25:07.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-18T14:25:07.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-18T14:25:07.696 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.3 2024-09-18T14:25:07.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:25:07.698 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-09-18T14:25:07.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-18T14:25:07.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:25:07.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:25:07.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:25:07.708 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:25:07.707+0000 7fc6b4e69440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:25:07.710 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:25:07.709+0000 7fc6b4e69440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:25:07.711 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:25:07.710+0000 7fc6b4e69440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:25:08.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-18T14:25:08.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:25:08.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T14:25:08.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:25:08.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:25:08.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:08.124 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:25:08.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:25:08.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:08.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:25:08.485 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:25:08.484+0000 7fc6b4e69440 -1 Falling back to public interface 2024-09-18T14:25:08.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:09.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:09.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:09.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:25:09.498 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:25:09.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:09.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:25:09.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:10.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:10.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:10.916 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:25:10.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:25:10.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:10.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:25:11.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:12.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:12.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:12.281 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:25:12.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:25:12.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:12.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:25:12.666 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:25:12.784 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:25:12.783+0000 7fc6b4e69440 -1 osd.3 55 log_to_monitors true 2024-09-18T14:25:13.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:25:13.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:25:13.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:25:13.669 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:25:13.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:25:13.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:25:14.193 INFO:tasks.workunit.client.0.smithi047.stdout:osd.3 up in weight 1 up_from 68 up_thru 54 down_at 56 last_clean_interval [46,55) [v2:127.0.0.1:6826/138438,v1:127.0.0.1:6827/138438] [v2:127.0.0.1:6828/138438,v1:127.0.0.1:6829/138438] exists,up 7dc3b3b6-05fa-4664-93b4-abdebdb06ac0 2024-09-18T14:25:14.194 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:25:14.194 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:25:14.194 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:25:14.194 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-18T14:25:14.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: osd_pid3=138438 2024-09-18T14:25:14.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:229: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-18T14:25:14.223 INFO:tasks.workunit.client.0.smithi047.stdout:hints = 1 2024-09-18T14:25:14.223 INFO:tasks.workunit.client.0.smithi047.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-18T14:25:14.223 INFO:tasks.workunit.client.0.smithi047.stdout:Object prefix: benchmark_data_smithi047_139404 2024-09-18T14:25:14.224 INFO:tasks.workunit.client.0.smithi047.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T14:25:14.224 INFO:tasks.workunit.client.0.smithi047.stdout: 0 0 0 0 0 0 - 0 2024-09-18T14:25:15.223 INFO:tasks.workunit.client.0.smithi047.stdout: 1 16 49 33 0.128892 0.128906 0.982485 0.461873 2024-09-18T14:25:16.224 INFO:tasks.workunit.client.0.smithi047.stdout: 2 16 99 83 0.162092 0.195312 0.276995 0.366171 2024-09-18T14:25:17.224 INFO:tasks.workunit.client.0.smithi047.stdout: 3 16 161 145 0.188783 0.242188 0.18328 0.319112 2024-09-18T14:25:18.224 INFO:tasks.workunit.client.0.smithi047.stdout: 4 16 201 185 0.180646 0.15625 0.2018 0.32589 2024-09-18T14:25:19.224 INFO:tasks.workunit.client.0.smithi047.stdout: 5 16 263 247 0.192949 0.242188 0.325047 0.314001 2024-09-18T14:25:20.224 INFO:tasks.workunit.client.0.smithi047.stdout: 6 16 321 305 0.198547 0.226562 0.284776 0.309112 2024-09-18T14:25:21.224 INFO:tasks.workunit.client.0.smithi047.stdout: 7 16 379 363 0.202545 0.226562 0.176301 0.304185 2024-09-18T14:25:22.224 INFO:tasks.workunit.client.0.smithi047.stdout: 8 16 431 415 0.202615 0.203125 0.205488 0.301495 2024-09-18T14:25:23.224 INFO:tasks.workunit.client.0.smithi047.stdout: 9 16 484 468 0.203103 0.207031 0.192307 0.303439 2024-09-18T14:25:24.224 INFO:tasks.workunit.client.0.smithi047.stdout: 10 16 537 521 0.203493 0.207031 0.220502 0.301324 2024-09-18T14:25:25.224 INFO:tasks.workunit.client.0.smithi047.stdout: 11 16 597 581 0.206298 0.234375 0.308832 0.299643 2024-09-18T14:25:26.225 INFO:tasks.workunit.client.0.smithi047.stdout: 12 16 650 634 0.206357 0.207031 0.303328 0.299391 2024-09-18T14:25:27.225 INFO:tasks.workunit.client.0.smithi047.stdout: 13 16 704 688 0.206708 0.210938 0.264596 0.298692 2024-09-18T14:25:28.225 INFO:tasks.workunit.client.0.smithi047.stdout: 14 16 758 742 0.207008 0.210938 0.276799 0.29845 2024-09-18T14:25:29.225 INFO:tasks.workunit.client.0.smithi047.stdout: 15 16 819 803 0.20909 0.238281 0.251564 0.296731 2024-09-18T14:25:30.225 INFO:tasks.workunit.client.0.smithi047.stdout: 16 16 875 859 0.209692 0.21875 0.180567 0.2953 2024-09-18T14:25:31.225 INFO:tasks.workunit.client.0.smithi047.stdout: 17 16 929 913 0.209763 0.210938 0.301863 0.29521 2024-09-18T14:25:32.225 INFO:tasks.workunit.client.0.smithi047.stdout: 18 16 985 969 0.210261 0.21875 0.362611 0.294923 2024-09-18T14:25:33.226 INFO:tasks.workunit.client.0.smithi047.stdout: 19 16 1035 1019 0.209473 0.195312 0.389404 0.295181 2024-09-18T14:25:34.226 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T14:25:34.226434+0000 min lat: 0.150889 max lat: 0.982486 avg lat: 0.297433 2024-09-18T14:25:34.226 INFO:tasks.workunit.client.0.smithi047.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T14:25:34.226 INFO:tasks.workunit.client.0.smithi047.stdout: 20 16 1082 1066 0.208178 0.183594 0.326635 0.297433 2024-09-18T14:25:35.226 INFO:tasks.workunit.client.0.smithi047.stdout: 21 16 1139 1123 0.208866 0.222656 0.209337 0.297354 2024-09-18T14:25:36.226 INFO:tasks.workunit.client.0.smithi047.stdout: 22 16 1186 1170 0.207716 0.183594 0.325508 0.298329 2024-09-18T14:25:37.226 INFO:tasks.workunit.client.0.smithi047.stdout: 23 16 1241 1225 0.208024 0.214844 0.350511 0.297957 2024-09-18T14:25:38.226 INFO:tasks.workunit.client.0.smithi047.stdout: 24 16 1294 1278 0.207982 0.207031 0.190144 0.29919 2024-09-18T14:25:39.226 INFO:tasks.workunit.client.0.smithi047.stdout: 25 16 1336 1320 0.206224 0.164062 0.331901 0.300995 2024-09-18T14:25:40.227 INFO:tasks.workunit.client.0.smithi047.stdout: 26 16 1388 1372 0.206104 0.203125 0.181081 0.301461 2024-09-18T14:25:41.227 INFO:tasks.workunit.client.0.smithi047.stdout: 27 16 1441 1425 0.206138 0.207031 0.254924 0.301207 2024-09-18T14:25:42.227 INFO:tasks.workunit.client.0.smithi047.stdout: 28 16 1496 1480 0.206447 0.214844 0.274319 0.300824 2024-09-18T14:25:43.227 INFO:tasks.workunit.client.0.smithi047.stdout: 29 16 1553 1537 0.207006 0.222656 0.208047 0.300882 2024-09-18T14:25:44.227 INFO:tasks.workunit.client.0.smithi047.stdout: 30 16 1603 1587 0.206615 0.195312 0.376361 0.301039 2024-09-18T14:25:45.227 INFO:tasks.workunit.client.0.smithi047.stdout:Total time run: 30.2328 2024-09-18T14:25:45.228 INFO:tasks.workunit.client.0.smithi047.stdout:Total writes made: 1603 2024-09-18T14:25:45.228 INFO:tasks.workunit.client.0.smithi047.stdout:Write size: 4096 2024-09-18T14:25:45.228 INFO:tasks.workunit.client.0.smithi047.stdout:Object size: 4096 2024-09-18T14:25:45.228 INFO:tasks.workunit.client.0.smithi047.stdout:Bandwidth (MB/sec): 0.207117 2024-09-18T14:25:45.228 INFO:tasks.workunit.client.0.smithi047.stdout:Stddev Bandwidth: 0.0248825 2024-09-18T14:25:45.228 INFO:tasks.workunit.client.0.smithi047.stdout:Max bandwidth (MB/sec): 0.242188 2024-09-18T14:25:45.228 INFO:tasks.workunit.client.0.smithi047.stdout:Min bandwidth (MB/sec): 0.128906 2024-09-18T14:25:45.228 INFO:tasks.workunit.client.0.smithi047.stdout:Average IOPS: 53 2024-09-18T14:25:45.228 INFO:tasks.workunit.client.0.smithi047.stdout:Stddev IOPS: 6.36992 2024-09-18T14:25:45.228 INFO:tasks.workunit.client.0.smithi047.stdout:Max IOPS: 62 2024-09-18T14:25:45.228 INFO:tasks.workunit.client.0.smithi047.stdout:Min IOPS: 33 2024-09-18T14:25:45.228 INFO:tasks.workunit.client.0.smithi047.stdout:Average Latency(s): 0.301052 2024-09-18T14:25:45.228 INFO:tasks.workunit.client.0.smithi047.stdout:Stddev Latency(s): 0.06894 2024-09-18T14:25:45.228 INFO:tasks.workunit.client.0.smithi047.stdout:Max latency(s): 0.982486 2024-09-18T14:25:45.229 INFO:tasks.workunit.client.0.smithi047.stdout:Min latency(s): 0.150889 2024-09-18T14:25:45.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 135414 2024-09-18T14:25:45.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: sleep 1 2024-09-18T14:25:46.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 135414 2024-09-18T14:25:46.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 232: kill: (135414) - No such process 2024-09-18T14:25:46.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:233: TEST_bluestore: ceph osd down 0 2024-09-18T14:25:46.727 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already down. 2024-09-18T14:25:46.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 136405 2024-09-18T14:25:46.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: sleep 1 2024-09-18T14:25:47.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 136405 2024-09-18T14:25:47.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 234: kill: (136405) - No such process 2024-09-18T14:25:47.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:235: TEST_bluestore: ceph osd down 1 2024-09-18T14:25:48.153 INFO:tasks.workunit.client.0.smithi047.stderr:osd.1 is already down. 2024-09-18T14:25:48.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 137422 2024-09-18T14:25:48.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: sleep 1 2024-09-18T14:25:49.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 137422 2024-09-18T14:25:49.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 236: kill: (137422) - No such process 2024-09-18T14:25:49.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:237: TEST_bluestore: ceph osd down 2 2024-09-18T14:25:49.636 INFO:tasks.workunit.client.0.smithi047.stderr:osd.2 is already down. 2024-09-18T14:25:49.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 138438 2024-09-18T14:25:49.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: sleep 1 2024-09-18T14:25:50.648 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 138438 2024-09-18T14:25:50.648 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 238: kill: (138438) - No such process 2024-09-18T14:25:50.648 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:239: TEST_bluestore: ceph osd down 3 2024-09-18T14:25:51.138 INFO:tasks.workunit.client.0.smithi047.stderr:osd.3 is already down. 2024-09-18T14:25:51.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:242: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-18T14:25:53.181 INFO:tasks.workunit.client.0.smithi047.stdout:fsck success 2024-09-18T14:25:53.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:244: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/db2 count=1024 bs=1M 2024-09-18T14:25:53.703 INFO:tasks.workunit.client.0.smithi047.stderr:1024+0 records in 2024-09-18T14:25:53.703 INFO:tasks.workunit.client.0.smithi047.stderr:1024+0 records out 2024-09-18T14:25:53.703 INFO:tasks.workunit.client.0.smithi047.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.510848 s, 2.1 GB/s 2024-09-18T14:25:53.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:245: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/db2 --command bluefs-bdev-migrate 2024-09-18T14:25:53.719 INFO:tasks.workunit.client.0.smithi047.stdout:inferring bluefs devices from bluestore path 2024-09-18T14:26:02.235 INFO:tasks.workunit.client.0.smithi047.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2024-09-18T14:26:02.235 INFO:tasks.workunit.client.0.smithi047.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/db2 2024-09-18T14:26:02.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:250: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-18T14:26:04.022 INFO:tasks.workunit.client.0.smithi047.stdout:fsck success 2024-09-18T14:26:04.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:254: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal2 count=512 bs=1M 2024-09-18T14:26:04.294 INFO:tasks.workunit.client.0.smithi047.stderr:512+0 records in 2024-09-18T14:26:04.295 INFO:tasks.workunit.client.0.smithi047.stderr:512+0 records out 2024-09-18T14:26:04.295 INFO:tasks.workunit.client.0.smithi047.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.261229 s, 2.1 GB/s 2024-09-18T14:26:04.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:255: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/wal2 --command bluefs-bdev-migrate 2024-09-18T14:26:04.311 INFO:tasks.workunit.client.0.smithi047.stdout:inferring bluefs devices from bluestore path 2024-09-18T14:26:11.581 INFO:tasks.workunit.client.0.smithi047.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2024-09-18T14:26:11.581 INFO:tasks.workunit.client.0.smithi047.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal2 2024-09-18T14:26:11.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:259: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/0/wal 2024-09-18T14:26:11.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:261: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-18T14:26:13.439 INFO:tasks.workunit.client.0.smithi047.stdout:fsck success 2024-09-18T14:26:13.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:264: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-18T14:26:15.230 INFO:tasks.workunit.client.0.smithi047.stdout:fsck success 2024-09-18T14:26:15.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:266: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db2 count=1024 bs=1M 2024-09-18T14:26:15.747 INFO:tasks.workunit.client.0.smithi047.stderr:1024+0 records in 2024-09-18T14:26:15.747 INFO:tasks.workunit.client.0.smithi047.stderr:1024+0 records out 2024-09-18T14:26:15.747 INFO:tasks.workunit.client.0.smithi047.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.508558 s, 2.1 GB/s 2024-09-18T14:26:15.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:267: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --devs-source td/osd-bluefs-volume-ops/1/block.wal --dev-target td/osd-bluefs-volume-ops/1/db2 --command bluefs-bdev-migrate 2024-09-18T14:26:15.762 INFO:tasks.workunit.client.0.smithi047.stdout:inferring bluefs devices from bluestore path 2024-09-18T14:26:26.534 INFO:tasks.workunit.client.0.smithi047.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-09-18T14:26:26.534 INFO:tasks.workunit.client.0.smithi047.stdout: device removed:0 td/osd-bluefs-volume-ops/1/block.wal 2024-09-18T14:26:26.534 INFO:tasks.workunit.client.0.smithi047.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db2 2024-09-18T14:26:26.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:273: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db 2024-09-18T14:26:26.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:275: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-18T14:26:27.944 INFO:tasks.workunit.client.0.smithi047.stdout:fsck success 2024-09-18T14:26:27.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:277: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2024-09-18T14:26:27.969 INFO:tasks.workunit.client.0.smithi047.stdout:inferring bluefs devices from bluestore path 2024-09-18T14:26:31.239 INFO:tasks.workunit.client.0.smithi047.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-09-18T14:26:31.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:282: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db2 2024-09-18T14:26:31.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:284: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-18T14:26:32.155 INFO:tasks.workunit.client.0.smithi047.stdout:fsck success 2024-09-18T14:26:32.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:287: TEST_bluestore: ceph-objectstore-tool --type bluestore --data-path td/osd-bluefs-volume-ops/2 --op fsck --no-mon-config 2024-09-18T14:26:33.944 INFO:tasks.workunit.client.0.smithi047.stdout:fsck success 2024-09-18T14:26:33.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:290: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2024-09-18T14:26:34.455 INFO:tasks.workunit.client.0.smithi047.stderr:1024+0 records in 2024-09-18T14:26:34.455 INFO:tasks.workunit.client.0.smithi047.stderr:1024+0 records out 2024-09-18T14:26:34.455 INFO:tasks.workunit.client.0.smithi047.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.500892 s, 2.1 GB/s 2024-09-18T14:26:34.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:291: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2024-09-18T14:26:34.471 INFO:tasks.workunit.client.0.smithi047.stdout:inferring bluefs devices from bluestore path 2024-09-18T14:26:34.471 INFO:tasks.workunit.client.0.smithi047.stderr:Unable to migrate Slow volume to new location, please allocate new DB or WAL with --bluefs-bdev-new-db(wal) command 2024-09-18T14:26:34.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:297: TEST_bluestore: '[' 1 -eq 0 ']' 2024-09-18T14:26:34.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:300: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-18T14:26:36.251 INFO:tasks.workunit.client.0.smithi047.stdout:fsck success 2024-09-18T14:26:36.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:303: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2024-09-18T14:26:43.176 INFO:tasks.workunit.client.0.smithi047.stderr:1024+0 records in 2024-09-18T14:26:43.231 INFO:tasks.workunit.client.0.smithi047.stderr:1024+0 records out 2024-09-18T14:26:43.231 INFO:tasks.workunit.client.0.smithi047.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 6.7916 s, 158 MB/s 2024-09-18T14:26:43.231 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:305: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --devs-source td/osd-bluefs-volume-ops/2/block.db --devs-source td/osd-bluefs-volume-ops/2/block.wal --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2024-09-18T14:26:43.232 INFO:tasks.workunit.client.0.smithi047.stdout:inferring bluefs devices from bluestore path 2024-09-18T14:26:47.467 INFO:tasks.workunit.client.0.smithi047.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2024-09-18T14:26:47.467 INFO:tasks.workunit.client.0.smithi047.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2024-09-18T14:26:47.467 INFO:tasks.workunit.client.0.smithi047.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/2/db2 2024-09-18T14:26:47.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:312: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-18T14:26:48.771 INFO:tasks.workunit.client.0.smithi047.stdout:fsck success 2024-09-18T14:26:48.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:315: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/3/wal2 count=1024 bs=1M 2024-09-18T14:26:49.288 INFO:tasks.workunit.client.0.smithi047.stderr:1024+0 records in 2024-09-18T14:26:49.288 INFO:tasks.workunit.client.0.smithi047.stderr:1024+0 records out 2024-09-18T14:26:49.289 INFO:tasks.workunit.client.0.smithi047.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.509027 s, 2.1 GB/s 2024-09-18T14:26:49.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:317: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/wal2 --command bluefs-bdev-migrate 2024-09-18T14:26:49.304 INFO:tasks.workunit.client.0.smithi047.stdout:inferring bluefs devices from bluestore path 2024-09-18T14:26:59.072 INFO:tasks.workunit.client.0.smithi047.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block 2024-09-18T14:26:59.072 INFO:tasks.workunit.client.0.smithi047.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2024-09-18T14:26:59.072 INFO:tasks.workunit.client.0.smithi047.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/3/wal2 2024-09-18T14:26:59.095 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:323: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-18T14:27:00.377 INFO:tasks.workunit.client.0.smithi047.stdout:fsck success 2024-09-18T14:27:00.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:325: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-09-18T14:27:00.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:27:00.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:27:00.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T14:27:00.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:27:00.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-18T14:27:00.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-18T14:27:00.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:27:00.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:27:00.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:27:00.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-18T14:27:00.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-18T14:27:00.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:27:00.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:27:00.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-18T14:27:00.387 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:27:00.387 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:27:00.387 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:27:00.388 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:27:00.388 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:27:00.388 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:27:00.388 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:27:00.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:27:00.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:27:00.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-18T14:27:00.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-18T14:27:00.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:27:00.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:27:00.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:27:00.389 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:27:00.389 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:27:00.389 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:27:00.389 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-18T14:27:00.389 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T14:27:00.390 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T14:27:00.390 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:27:00.390 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-18T14:27:00.390 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T14:27:00.390 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:27:00.390 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:27:00.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:27:00.411 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:27:00.410+0000 7fa716c11440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:27:00.414 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:27:00.413+0000 7fa716c11440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:27:00.415 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:27:00.414+0000 7fa716c11440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:27:00.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T14:27:00.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:27:00.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:27:00.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:27:00.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:27:00.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:27:00.764 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:27:00.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:27:00.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:27:00.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:27:01.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:27:01.433 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:27:01.433+0000 7fa716c11440 -1 Falling back to public interface 2024-09-18T14:27:02.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:27:02.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:27:02.168 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:27:02.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:27:02.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:27:02.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:27:02.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:27:03.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:27:03.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:27:03.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:27:03.543 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:27:03.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:27:03.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:27:03.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:27:04.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:27:04.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:27:04.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:27:04.964 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:27:04.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:27:04.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:27:05.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:27:05.987 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:27:05.986+0000 7fa716c11440 -1 osd.0 70 log_to_monitors true 2024-09-18T14:27:06.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:27:06.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:27:06.353 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:27:06.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:27:06.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:27:06.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:27:06.759 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 80 up_thru 80 down_at 71 last_clean_interval [58,70) [v2:127.0.0.1:6802/145622,v1:127.0.0.1:6803/145622] [v2:127.0.0.1:6804/145622,v1:127.0.0.1:6805/145622] exists,up 77e514a7-b7cf-4967-acb8-f65677560a17 2024-09-18T14:27:06.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:27:06.760 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:27:06.760 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:27:06.760 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-18T14:27:06.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: osd_pid0=145622 2024-09-18T14:27:06.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:327: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-09-18T14:27:06.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:27:06.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:27:06.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T14:27:06.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:27:06.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-18T14:27:06.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-18T14:27:06.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:27:06.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:27:06.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:27:06.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-18T14:27:06.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-18T14:27:06.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:27:06.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:27:06.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-18T14:27:06.765 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:27:06.766 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:27:06.766 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:27:06.766 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:27:06.766 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:27:06.766 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:27:06.766 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:27:06.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:27:06.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:27:06.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-18T14:27:06.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-18T14:27:06.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:27:06.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:27:06.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:27:06.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:27:06.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:27:06.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:27:06.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-18T14:27:06.768 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T14:27:06.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T14:27:06.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:27:06.770 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-09-18T14:27:06.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T14:27:06.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:27:06.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:27:06.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:27:06.785 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:27:06.784+0000 7f8f4bf20440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:27:06.787 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:27:06.786+0000 7f8f4bf20440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:27:06.788 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:27:06.787+0000 7f8f4bf20440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:27:07.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T14:27:07.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:27:07.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:27:07.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:27:07.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:27:07.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:27:07.134 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:27:07.135 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:27:07.135 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:27:07.135 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:27:07.301 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:27:07.301+0000 7f8f4bf20440 -1 Falling back to public interface 2024-09-18T14:27:07.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:27:08.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:27:08.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:27:08.525 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:27:08.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:27:08.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:27:08.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:27:08.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:27:09.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:27:09.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:27:09.901 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:27:09.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:27:09.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:27:09.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:27:10.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:27:10.353 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:27:10.353+0000 7f8f4bf20440 -1 osd.1 72 log_to_monitors true 2024-09-18T14:27:11.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:27:11.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:27:11.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:27:11.283 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:27:11.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:27:11.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:27:11.688 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 83 up_thru 83 down_at 73 last_clean_interval [61,72) [v2:127.0.0.1:6810/146613,v1:127.0.0.1:6811/146613] [v2:127.0.0.1:6812/146613,v1:127.0.0.1:6813/146613] exists,up 8164e048-2490-4c40-b72e-815a1382b89d 2024-09-18T14:27:11.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:27:11.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:27:11.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:27:11.689 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-18T14:27:11.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: osd_pid1=146613 2024-09-18T14:27:11.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:329: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-09-18T14:27:11.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:27:11.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:27:11.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T14:27:11.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:27:11.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-18T14:27:11.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-18T14:27:11.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:27:11.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:27:11.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:27:11.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-18T14:27:11.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-18T14:27:11.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:27:11.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:27:11.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-18T14:27:11.694 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:27:11.694 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:27:11.694 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:27:11.694 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:27:11.694 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:27:11.694 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:27:11.694 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:27:11.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:27:11.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:27:11.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-18T14:27:11.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-18T14:27:11.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:27:11.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:27:11.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:27:11.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:27:11.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:27:11.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:27:11.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-18T14:27:11.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T14:27:11.695 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T14:27:11.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:27:11.696 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-09-18T14:27:11.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T14:27:11.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:27:11.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:27:11.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:27:11.714 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:27:11.713+0000 7f86b8008440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:27:11.716 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:27:11.715+0000 7f86b8008440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:27:11.717 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:27:11.716+0000 7f86b8008440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:27:12.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T14:27:12.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:27:12.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T14:27:12.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:27:12.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:27:12.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:27:12.083 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:27:12.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:27:12.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:27:12.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:27:12.480 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:27:12.480+0000 7f86b8008440 -1 Falling back to public interface 2024-09-18T14:27:12.491 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:27:13.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:27:13.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:27:13.493 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:27:13.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:27:13.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:27:13.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:27:13.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:27:14.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:27:14.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:27:14.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:27:14.890 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:27:14.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:27:14.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:27:15.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:27:16.018 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:27:16.017+0000 7f86b8008440 -1 osd.2 75 log_to_monitors true 2024-09-18T14:27:16.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:27:16.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:27:16.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:27:16.288 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:27:16.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:27:16.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:27:16.771 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 86 up_thru 73 down_at 76 last_clean_interval [64,75) [v2:127.0.0.1:6818/147559,v1:127.0.0.1:6819/147559] [v2:127.0.0.1:6820/147559,v1:127.0.0.1:6821/147559] exists,up c58dcf9b-e2da-4e51-9972-4ac5044eccd6 2024-09-18T14:27:16.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:27:16.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:27:16.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:27:16.772 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-18T14:27:16.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: osd_pid2=147559 2024-09-18T14:27:16.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:331: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-09-18T14:27:16.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:27:16.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:27:16.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-18T14:27:16.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:27:16.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-18T14:27:16.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-18T14:27:16.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:27:16.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:27:16.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:27:16.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-18T14:27:16.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-18T14:27:16.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:27:16.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:27:16.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-18T14:27:16.806 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:27:16.806 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:27:16.806 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:27:16.806 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:27:16.806 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:27:16.806 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:27:16.806 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:27:16.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:27:16.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:27:16.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-18T14:27:16.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-18T14:27:16.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:27:16.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:27:16.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:27:16.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:27:16.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:27:16.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:27:16.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-18T14:27:16.807 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.3 2024-09-18T14:27:16.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-18T14:27:16.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:27:16.809 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-09-18T14:27:16.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-18T14:27:16.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:27:16.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:27:16.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:27:16.826 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:27:16.825+0000 7f1f83c3a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:27:16.905 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:27:16.905+0000 7f1f83c3a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:27:16.907 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:27:16.906+0000 7f1f83c3a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:27:17.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-18T14:27:17.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:27:17.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T14:27:17.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:27:17.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:27:17.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:27:17.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:27:17.380 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:27:17.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:27:17.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:27:17.671 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:27:17.671+0000 7f1f83c3a440 -1 Falling back to public interface 2024-09-18T14:27:17.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:27:18.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:27:18.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:27:18.824 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:27:18.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:27:18.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:27:18.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:27:19.231 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:27:20.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:27:20.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:27:20.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:27:20.233 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:27:20.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:27:20.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:27:20.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:27:21.532 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:27:21.531+0000 7f1f83c3a440 -1 osd.3 77 log_to_monitors true 2024-09-18T14:27:21.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:27:21.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:27:21.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:27:21.626 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:27:21.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:27:21.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:27:22.294 INFO:tasks.workunit.client.0.smithi047.stdout:osd.3 up in weight 1 up_from 90 up_thru 76 down_at 78 last_clean_interval [68,77) [v2:127.0.0.1:6826/148532,v1:127.0.0.1:6827/148532] [v2:127.0.0.1:6828/148532,v1:127.0.0.1:6829/148532] exists,up 7dc3b3b6-05fa-4664-93b4-abdebdb06ac0 2024-09-18T14:27:22.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:27:22.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:27:22.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:27:22.295 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-18T14:27:22.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: osd_pid3=148532 2024-09-18T14:27:22.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:335: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-18T14:27:22.412 INFO:tasks.workunit.client.0.smithi047.stdout:hints = 1 2024-09-18T14:27:22.413 INFO:tasks.workunit.client.0.smithi047.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-18T14:27:22.413 INFO:tasks.workunit.client.0.smithi047.stdout:Object prefix: benchmark_data_smithi047_149456 2024-09-18T14:27:22.413 INFO:tasks.workunit.client.0.smithi047.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T14:27:22.413 INFO:tasks.workunit.client.0.smithi047.stdout: 0 0 0 0 0 0 - 0 2024-09-18T14:27:23.413 INFO:tasks.workunit.client.0.smithi047.stdout: 1 16 49 33 0.128899 0.128906 0.0917405 0.232164 2024-09-18T14:27:24.413 INFO:tasks.workunit.client.0.smithi047.stdout: 2 16 147 131 0.255835 0.382812 0.150443 0.239369 2024-09-18T14:27:25.413 INFO:tasks.workunit.client.0.smithi047.stdout: 3 16 250 234 0.304658 0.402344 0.107998 0.198916 2024-09-18T14:27:26.413 INFO:tasks.workunit.client.0.smithi047.stdout: 4 16 362 346 0.337854 0.4375 0.0828418 0.180252 2024-09-18T14:27:27.413 INFO:tasks.workunit.client.0.smithi047.stdout: 5 16 489 473 0.369493 0.496094 0.144391 0.168694 2024-09-18T14:27:28.413 INFO:tasks.workunit.client.0.smithi047.stdout: 6 16 593 577 0.37561 0.40625 0.0886138 0.164343 2024-09-18T14:27:29.413 INFO:tasks.workunit.client.0.smithi047.stdout: 7 16 717 701 0.391141 0.484375 0.125898 0.159337 2024-09-18T14:27:30.413 INFO:tasks.workunit.client.0.smithi047.stdout: 8 16 829 813 0.39693 0.4375 0.152482 0.157091 2024-09-18T14:27:31.413 INFO:tasks.workunit.client.0.smithi047.stdout: 9 16 950 934 0.40534 0.472656 0.108651 0.153429 2024-09-18T14:27:32.413 INFO:tasks.workunit.client.0.smithi047.stdout: 10 16 1070 1054 0.411674 0.46875 0.125239 0.150605 2024-09-18T14:27:33.414 INFO:tasks.workunit.client.0.smithi047.stdout: 11 16 1188 1172 0.416146 0.460938 0.166858 0.14855 2024-09-18T14:27:34.414 INFO:tasks.workunit.client.0.smithi047.stdout: 12 16 1287 1271 0.413689 0.386719 0.107739 0.149919 2024-09-18T14:27:35.414 INFO:tasks.workunit.client.0.smithi047.stdout: 13 16 1408 1392 0.41822 0.472656 0.0917759 0.148527 2024-09-18T14:27:36.414 INFO:tasks.workunit.client.0.smithi047.stdout: 14 16 1525 1509 0.420987 0.457031 0.10847 0.147499 2024-09-18T14:27:37.414 INFO:tasks.workunit.client.0.smithi047.stdout: 15 16 1647 1631 0.424689 0.476562 0.120667 0.146361 2024-09-18T14:27:38.414 INFO:tasks.workunit.client.0.smithi047.stdout: 16 16 1750 1734 0.423289 0.402344 0.109037 0.146814 2024-09-18T14:27:39.414 INFO:tasks.workunit.client.0.smithi047.stdout: 17 16 1843 1827 0.419757 0.363281 0.340067 0.147505 2024-09-18T14:27:40.415 INFO:tasks.workunit.client.0.smithi047.stdout: 18 16 1950 1934 0.419655 0.417969 0.0956071 0.147938 2024-09-18T14:27:41.415 INFO:tasks.workunit.client.0.smithi047.stdout: 19 16 2076 2060 0.423469 0.492188 0.126088 0.147014 2024-09-18T14:27:42.415 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T14:27:42.415452+0000 min lat: 0.0609438 max lat: 1.09282 avg lat: 0.146732 2024-09-18T14:27:42.415 INFO:tasks.workunit.client.0.smithi047.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T14:27:42.415 INFO:tasks.workunit.client.0.smithi047.stdout: 20 16 2189 2173 0.424364 0.441406 0.10911 0.146732 2024-09-18T14:27:43.415 INFO:tasks.workunit.client.0.smithi047.stdout: 21 16 2301 2285 0.424988 0.4375 0.161853 0.14643 2024-09-18T14:27:44.415 INFO:tasks.workunit.client.0.smithi047.stdout: 22 16 2415 2399 0.425909 0.445312 0.106848 0.146431 2024-09-18T14:27:45.415 INFO:tasks.workunit.client.0.smithi047.stdout: 23 16 2512 2496 0.423863 0.378906 0.107637 0.146781 2024-09-18T14:27:46.415 INFO:tasks.workunit.client.0.smithi047.stdout: 24 16 2626 2610 0.424755 0.445312 0.114163 0.146449 2024-09-18T14:27:47.415 INFO:tasks.workunit.client.0.smithi047.stdout: 25 16 2743 2727 0.426043 0.457031 0.134522 0.146287 2024-09-18T14:27:48.416 INFO:tasks.workunit.client.0.smithi047.stdout: 26 16 2854 2838 0.426331 0.433594 0.167128 0.146173 2024-09-18T14:27:49.416 INFO:tasks.workunit.client.0.smithi047.stdout: 27 16 2984 2968 0.429347 0.507812 0.150737 0.145356 2024-09-18T14:27:50.416 INFO:tasks.workunit.client.0.smithi047.stdout: 28 16 3083 3067 0.427823 0.386719 0.172236 0.145443 2024-09-18T14:27:51.416 INFO:tasks.workunit.client.0.smithi047.stdout: 29 16 3163 3147 0.423845 0.3125 0.186378 0.147114 2024-09-18T14:27:52.416 INFO:tasks.workunit.client.0.smithi047.stdout: 30 16 3284 3268 0.42547 0.472656 0.105232 0.146686 2024-09-18T14:27:53.417 INFO:tasks.workunit.client.0.smithi047.stdout:Total time run: 30.1095 2024-09-18T14:27:53.417 INFO:tasks.workunit.client.0.smithi047.stdout:Total writes made: 3284 2024-09-18T14:27:53.417 INFO:tasks.workunit.client.0.smithi047.stdout:Write size: 4096 2024-09-18T14:27:53.417 INFO:tasks.workunit.client.0.smithi047.stdout:Object size: 4096 2024-09-18T14:27:53.417 INFO:tasks.workunit.client.0.smithi047.stdout:Bandwidth (MB/sec): 0.426049 2024-09-18T14:27:53.417 INFO:tasks.workunit.client.0.smithi047.stdout:Stddev Bandwidth: 0.0716096 2024-09-18T14:27:53.417 INFO:tasks.workunit.client.0.smithi047.stdout:Max bandwidth (MB/sec): 0.507812 2024-09-18T14:27:53.417 INFO:tasks.workunit.client.0.smithi047.stdout:Min bandwidth (MB/sec): 0.128906 2024-09-18T14:27:53.417 INFO:tasks.workunit.client.0.smithi047.stdout:Average IOPS: 109 2024-09-18T14:27:53.417 INFO:tasks.workunit.client.0.smithi047.stdout:Stddev IOPS: 18.3321 2024-09-18T14:27:53.418 INFO:tasks.workunit.client.0.smithi047.stdout:Max IOPS: 130 2024-09-18T14:27:53.418 INFO:tasks.workunit.client.0.smithi047.stdout:Min IOPS: 33 2024-09-18T14:27:53.418 INFO:tasks.workunit.client.0.smithi047.stdout:Average Latency(s): 0.146563 2024-09-18T14:27:53.418 INFO:tasks.workunit.client.0.smithi047.stdout:Stddev Latency(s): 0.0593155 2024-09-18T14:27:53.418 INFO:tasks.workunit.client.0.smithi047.stdout:Max latency(s): 1.09282 2024-09-18T14:27:53.418 INFO:tasks.workunit.client.0.smithi047.stdout:Min latency(s): 0.0609438 2024-09-18T14:27:53.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:337: TEST_bluestore: wait_for_clean 2024-09-18T14:27:53.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:27:53.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:27:53.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:27:53.486 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:27:53.486 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:27:53.487 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:27:53.487 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:27:53.487 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:27:53.487 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:27:53.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T14:27:53.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:27:53.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:27:53.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:27:53.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:27:53.588 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:27:53.989 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:27:53.989 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T14:27:53.989 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T14:27:53.989 INFO:tasks.workunit.client.0.smithi047.stderr:3' 2024-09-18T14:27:53.989 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:27:53.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:27:53.990 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:27:54.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=343597383692 2024-09-18T14:27:54.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 343597383692 2024-09-18T14:27:54.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-343597383692' 2024-09-18T14:27:54.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:27:54.099 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:27:54.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=356482285579 2024-09-18T14:27:54.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 356482285579 2024-09-18T14:27:54.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-343597383692 1-356482285579' 2024-09-18T14:27:54.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:27:54.206 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:27:54.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=369367187466 2024-09-18T14:27:54.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 369367187466 2024-09-18T14:27:54.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-343597383692 1-356482285579 2-369367187466' 2024-09-18T14:27:54.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:27:54.314 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T14:27:54.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=386547056649 2024-09-18T14:27:54.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 386547056649 2024-09-18T14:27:54.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-343597383692 1-356482285579 2-369367187466 3-386547056649' 2024-09-18T14:27:54.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:27:54.422 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-343597383692 2024-09-18T14:27:54.422 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:27:54.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:27:54.423 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-343597383692 2024-09-18T14:27:54.423 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:27:54.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=343597383692 2024-09-18T14:27:54.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 343597383692' 2024-09-18T14:27:54.424 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 343597383692 2024-09-18T14:27:54.425 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:27:54.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 343597383690 -lt 343597383692 2024-09-18T14:27:54.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:27:55.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:27:55.788 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:27:56.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 343597383692 -lt 343597383692 2024-09-18T14:27:56.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:27:56.168 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-356482285579 2024-09-18T14:27:56.168 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:27:56.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:27:56.170 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-356482285579 2024-09-18T14:27:56.170 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:27:56.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=356482285579 2024-09-18T14:27:56.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 356482285579' 2024-09-18T14:27:56.171 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 356482285579 2024-09-18T14:27:56.171 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:27:56.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 356482285579 -lt 356482285579 2024-09-18T14:27:56.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:27:56.532 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-369367187466 2024-09-18T14:27:56.532 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:27:56.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:27:56.534 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-369367187466 2024-09-18T14:27:56.534 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:27:56.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=369367187466 2024-09-18T14:27:56.535 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 369367187466 2024-09-18T14:27:56.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 369367187466' 2024-09-18T14:27:56.535 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:27:56.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 369367187466 -lt 369367187466 2024-09-18T14:27:56.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:27:56.900 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-386547056649 2024-09-18T14:27:56.900 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:27:56.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T14:27:56.901 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-386547056649 2024-09-18T14:27:56.902 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:27:56.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=386547056649 2024-09-18T14:27:56.903 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 386547056649 2024-09-18T14:27:56.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 386547056649' 2024-09-18T14:27:56.903 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T14:27:57.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 386547056650 -lt 386547056649 2024-09-18T14:27:57.267 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:27:57.268 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:27:57.268 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:27:57.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-09-18T14:27:57.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:27:57.759 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:27:57.759 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:27:57.759 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:27:57.759 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:27:57.759 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:27:57.760 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:27:58.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-09-18T14:27:58.117 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:27:58.117 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:27:58.117 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:27:58.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-09-18T14:27:58.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:27:58.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:27:58.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-09-18T14:27:58.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:27:58.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:27:58.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-18T14:27:58.635 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:27:58.635 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:27:58.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:27:58.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:27:58.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:27:58.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:27:58.743 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:27:58.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:27:58.744 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:27:58.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:27:58.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:27:58.746 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:27:58.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:27:58.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:27:58.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:27:58.747 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:27:58.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:27:58.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:27:58.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-18T14:27:59.251 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:27:59.252 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:27:59.252 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:27:59.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.102152 2024-09-18T14:27:59.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:27:59.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:27:59.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-09-18T14:27:59.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-09-18T14:27:59.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:27:59.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2024-09-18T14:27:59.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:27:59.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:27:59.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-18T14:27:59.254 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:27:59.254 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:27:59.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:27:59.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:27:59.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:27:59.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:27:59.255 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:27:59.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:27:59.256 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:27:59.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:27:59.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:27:59.258 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:27:59.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:27:59.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:27:59.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:27:59.259 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:27:59.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:27:59.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:27:59.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-18T14:27:59.262 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:27:59.262 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:27:59.262 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:27:59.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.102152 2024-09-18T14:27:59.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:27:59.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:27:59.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2024-09-18T14:27:59.264 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T14:27:59.264 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:27:59.264 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:27:59.264 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.102152 2024-09-18T14:27:59.267 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T14:27:59.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T14:27:59.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T14:27:59.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-09-18T14:27:59.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore2 td/osd-bluefs-volume-ops 2024-09-18T14:27:59.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:341: TEST_bluestore2: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:27:59.267 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:343: TEST_bluestore2: ulimit -n 2024-09-18T14:27:59.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:343: TEST_bluestore2: local flimit=4096 2024-09-18T14:27:59.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:344: TEST_bluestore2: '[' 4096 -lt 1536 ']' 2024-09-18T14:27:59.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:347: TEST_bluestore2: export CEPH_MON=127.0.0.1:7146 2024-09-18T14:27:59.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:347: TEST_bluestore2: CEPH_MON=127.0.0.1:7146 2024-09-18T14:27:59.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:348: TEST_bluestore2: export CEPH_ARGS 2024-09-18T14:27:59.268 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:349: TEST_bluestore2: uuidgen 2024-09-18T14:27:59.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:349: TEST_bluestore2: CEPH_ARGS+='--fsid=a4c8b576-3caa-45c9-9ed5-fcd93777ab86 --auth-supported=none ' 2024-09-18T14:27:59.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:350: TEST_bluestore2: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-18T14:27:59.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:351: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_size=4294967296 ' 2024-09-18T14:27:59.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:352: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-09-18T14:27:59.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:353: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-09-18T14:27:59.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:354: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_wal_create=false ' 2024-09-18T14:27:59.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:355: TEST_bluestore2: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-09-18T14:27:59.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:356: TEST_bluestore2: CEPH_ARGS+='--osd_pool_default_size=1 ' 2024-09-18T14:27:59.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:357: TEST_bluestore2: CEPH_ARGS+='--osd_pool_default_min_size=1 ' 2024-09-18T14:27:59.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:358: TEST_bluestore2: CEPH_ARGS+='--bluestore_debug_enforce_settings=ssd ' 2024-09-18T14:27:59.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:360: TEST_bluestore2: run_mon td/osd-bluefs-volume-ops a 2024-09-18T14:27:59.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:27:59.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T14:27:59.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T14:27:59.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T14:27:59.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-09-18T14:27:59.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-09-18T14:27:59.614 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T14:27:59.614 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:27:59.614 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:27:59.615 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:27:59.615 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:27:59.615 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:27:59.615 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:27:59.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T14:27:59.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T14:27:59.867 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T14:27:59.868 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:27:59.868 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:27:59.868 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T14:27:59.868 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T14:27:59.868 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:27:59.869 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:27:59.869 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:27:59.869 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:27:59.869 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:27:59.869 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:27:59.870 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.102152/ceph-mon.a.asok 2024-09-18T14:27:59.870 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:27:59.870 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.102152/ceph-mon.a.asok config get fsid 2024-09-18T14:27:59.951 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T14:27:59.951 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:27:59.951 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:27:59.952 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T14:27:59.952 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T14:27:59.952 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:27:59.952 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:27:59.952 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:27:59.952 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:27:59.952 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:27:59.952 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:27:59.952 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.102152/ceph-mon.a.asok 2024-09-18T14:27:59.953 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:27:59.953 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.102152/ceph-mon.a.asok config get mon_host 2024-09-18T14:28:00.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:361: TEST_bluestore2: run_mgr td/osd-bluefs-volume-ops x 2024-09-18T14:28:00.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:28:00.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T14:28:00.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T14:28:00.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T14:28:00.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-09-18T14:28:00.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T14:28:00.886 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T14:28:00.886 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:28:00.886 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:28:00.886 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:28:00.887 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:28:00.887 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:28:00.887 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:28:00.887 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:28:00.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:28:00.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:362: TEST_bluestore2: run_osd td/osd-bluefs-volume-ops 0 2024-09-18T14:28:00.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:28:00.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:28:00.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T14:28:00.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:28:00.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-18T14:28:00.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=a4c8b576-3caa-45c9-9ed5-fcd93777ab86 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-09-18T14:28:00.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:28:00.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:28:00.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:28:00.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-18T14:28:00.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-18T14:28:00.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:28:00.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:28:00.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-18T14:28:00.922 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:28:00.922 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:28:00.922 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:28:00.922 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:28:00.922 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:28:00.922 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:28:00.923 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:28:00.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:28:00.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:28:00.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:28:00.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:28:00.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-18T14:28:00.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-18T14:28:00.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:28:00.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:28:00.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:28:00.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:28:00.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:28:00.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:28:00.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-18T14:28:00.924 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:28:00.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=257388fb-a771-413d-8457-f0d3d1b980a9 2024-09-18T14:28:00.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 257388fb-a771-413d-8457-f0d3d1b980a9' 2024-09-18T14:28:00.924 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 257388fb-a771-413d-8457-f0d3d1b980a9 2024-09-18T14:28:00.925 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:28:00.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBw4+pmGuLXNxAA3DAwlMPKuEjXQQjXVV5gkw== 2024-09-18T14:28:00.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBw4+pmGuLXNxAA3DAwlMPKuEjXQQjXVV5gkw=="}' 2024-09-18T14:28:00.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 257388fb-a771-413d-8457-f0d3d1b980a9 -i td/osd-bluefs-volume-ops/0/new.json 2024-09-18T14:28:01.228 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:28:01.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-09-18T14:28:01.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=a4c8b576-3caa-45c9-9ed5-fcd93777ab86 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBw4+pmGuLXNxAA3DAwlMPKuEjXQQjXVV5gkw== --osd-uuid 257388fb-a771-413d-8457-f0d3d1b980a9 2024-09-18T14:28:01.268 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:28:01.267+0000 7f131cc09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:28:01.272 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:28:01.272+0000 7f131cc09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:28:01.273 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:28:01.273+0000 7f131cc09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:28:01.274 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:28:01.273+0000 7f131cc09440 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-09-18T14:28:01.274 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:28:01.273+0000 7f131cc09440 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-09-18T14:28:01.274 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:28:01.273+0000 7f131cc09440 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-09-18T14:28:01.313 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:28:01.312+0000 7f131cc09440 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-09-18T14:28:08.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-09-18T14:28:08.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:28:08.652 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T14:28:08.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T14:28:08.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:28:09.165 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T14:28:09.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T14:28:09.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=a4c8b576-3caa-45c9-9ed5-fcd93777ab86 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:28:09.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:28:09.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:28:09.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:28:09.193 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:28:09.192+0000 7f6296214440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:28:09.194 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:28:09.194+0000 7f6296214440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:28:09.196 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:28:09.195+0000 7f6296214440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:28:09.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T14:28:09.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:28:09.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:28:09.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:28:09.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:28:09.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:28:09.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:28:09.602 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:28:09.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:28:09.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:28:09.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:28:09.983 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:28:09.983+0000 7f6296214440 -1 Falling back to public interface 2024-09-18T14:28:10.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:28:10.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:28:10.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:28:10.973 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:28:10.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:28:10.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:28:11.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:28:12.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:28:12.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:28:12.396 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:28:12.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:28:12.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:28:12.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:28:12.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:28:13.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:28:13.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:28:13.813 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:28:13.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:28:13.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:28:13.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:28:13.872 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:28:13.871+0000 7f6296214440 -1 osd.0 0 log_to_monitors true 2024-09-18T14:28:14.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:28:15.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:28:15.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:28:15.444 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:28:15.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:28:15.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:28:15.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:28:16.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:28:17.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:28:17.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:28:17.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:28:17.525 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:28:17.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:28:17.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:28:17.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:28:18.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:28:18.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:28:18.961 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:28:18.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:28:18.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:28:18.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:28:19.406 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/151367,v1:127.0.0.1:6803/151367] [v2:127.0.0.1:6804/151367,v1:127.0.0.1:6805/151367] exists,up 257388fb-a771-413d-8457-f0d3d1b980a9 2024-09-18T14:28:19.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:28:19.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:28:19.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:28:19.407 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:363: TEST_bluestore2: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-18T14:28:19.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:363: TEST_bluestore2: osd_pid0=151367 2024-09-18T14:28:19.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:365: TEST_bluestore2: sleep 5 2024-09-18T14:28:24.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:366: TEST_bluestore2: create_pool foo 16 2024-09-18T14:28:24.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-09-18T14:28:25.064 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'foo' already exists 2024-09-18T14:28:25.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T14:28:26.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:368: TEST_bluestore2: retry = 0 2024-09-18T14:28:26.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 368: retry: command not found 2024-09-18T14:28:26.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:369: TEST_bluestore2: [[ '' -le 5 ]] 2024-09-18T14:28:26.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:371: TEST_bluestore2: timeout 60 rados bench -p foo 10 write --write-omap --no-cleanup 2024-09-18T14:28:26.140 INFO:tasks.workunit.client.0.smithi047.stdout:hints = 1 2024-09-18T14:28:26.144 INFO:tasks.workunit.client.0.smithi047.stdout:Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects 2024-09-18T14:28:26.144 INFO:tasks.workunit.client.0.smithi047.stdout:Object prefix: benchmark_data_smithi047_152666 2024-09-18T14:28:26.178 INFO:tasks.workunit.client.0.smithi047.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T14:28:26.178 INFO:tasks.workunit.client.0.smithi047.stdout: 0 0 0 0 0 0 - 0 2024-09-18T14:28:27.178 INFO:tasks.workunit.client.0.smithi047.stdout: 1 16 33 17 67.9936 68 0.885118 0.749313 2024-09-18T14:28:28.178 INFO:tasks.workunit.client.0.smithi047.stdout: 2 16 64 48 95.9901 124 0.515375 0.655729 2024-09-18T14:28:29.179 INFO:tasks.workunit.client.0.smithi047.stdout: 3 16 81 65 86.6559 68 0.542164 0.627707 2024-09-18T14:28:30.179 INFO:tasks.workunit.client.0.smithi047.stdout: 4 16 113 97 96.987 128 0.492139 0.609346 2024-09-18T14:28:31.179 INFO:tasks.workunit.client.0.smithi047.stdout: 5 16 144 128 102.386 124 0.659913 0.604864 2024-09-18T14:28:32.179 INFO:tasks.workunit.client.0.smithi047.stdout: 6 16 161 145 96.6527 68 0.508941 0.604916 2024-09-18T14:28:33.179 INFO:tasks.workunit.client.0.smithi047.stdout: 7 16 177 161 91.9863 64 0.984231 0.631481 2024-09-18T14:28:34.179 INFO:tasks.workunit.client.0.smithi047.stdout: 8 16 177 161 80.4877 0 - 0.631481 2024-09-18T14:28:35.180 INFO:tasks.workunit.client.0.smithi047.stdout: 9 16 193 177 78.6546 32 1.65964 0.729643 2024-09-18T14:28:36.180 INFO:tasks.workunit.client.0.smithi047.stdout: 10 16 208 192 76.7882 60 1.75892 0.809962 2024-09-18T14:28:37.180 INFO:tasks.workunit.client.0.smithi047.stdout: 11 15 208 193 70.1711 4 1.84453 0.815322 2024-09-18T14:28:38.180 INFO:tasks.workunit.client.0.smithi047.stdout:Total time run: 11.6896 2024-09-18T14:28:38.180 INFO:tasks.workunit.client.0.smithi047.stdout:Total writes made: 208 2024-09-18T14:28:38.181 INFO:tasks.workunit.client.0.smithi047.stdout:Write size: 4194304 2024-09-18T14:28:38.181 INFO:tasks.workunit.client.0.smithi047.stdout:Object size: 4194304 2024-09-18T14:28:38.181 INFO:tasks.workunit.client.0.smithi047.stdout:Bandwidth (MB/sec): 71.1743 2024-09-18T14:28:38.181 INFO:tasks.workunit.client.0.smithi047.stdout:Stddev Bandwidth: 44.6791 2024-09-18T14:28:38.181 INFO:tasks.workunit.client.0.smithi047.stdout:Max bandwidth (MB/sec): 128 2024-09-18T14:28:38.181 INFO:tasks.workunit.client.0.smithi047.stdout:Min bandwidth (MB/sec): 0 2024-09-18T14:28:38.181 INFO:tasks.workunit.client.0.smithi047.stdout:Average IOPS: 17 2024-09-18T14:28:38.181 INFO:tasks.workunit.client.0.smithi047.stdout:Stddev IOPS: 11.1698 2024-09-18T14:28:38.181 INFO:tasks.workunit.client.0.smithi047.stdout:Max IOPS: 32 2024-09-18T14:28:38.181 INFO:tasks.workunit.client.0.smithi047.stdout:Min IOPS: 0 2024-09-18T14:28:38.181 INFO:tasks.workunit.client.0.smithi047.stdout:Average Latency(s): 0.891268 2024-09-18T14:28:38.181 INFO:tasks.workunit.client.0.smithi047.stdout:Stddev Latency(s): 0.495916 2024-09-18T14:28:38.181 INFO:tasks.workunit.client.0.smithi047.stdout:Max latency(s): 1.86902 2024-09-18T14:28:38.181 INFO:tasks.workunit.client.0.smithi047.stdout:Min latency(s): 0.106924 2024-09-18T14:28:38.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:374: TEST_bluestore2: sleep 10 2024-09-18T14:28:48.401 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-09-18T14:28:48.402 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: jq .bluefs.db_used_bytes 2024-09-18T14:28:48.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: db_used=1072693248 2024-09-18T14:28:48.533 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-09-18T14:28:48.533 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: jq .bluefs.slow_used_bytes 2024-09-18T14:28:48.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: spilled_over=1409351680 2024-09-18T14:28:48.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:378: TEST_bluestore2: (( retry+=1 )) 2024-09-18T14:28:48.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: test 1409351680 -eq 0 2024-09-18T14:28:48.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: break 2024-09-18T14:28:48.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:381: TEST_bluestore2: test 1409351680 -gt 0 2024-09-18T14:28:48.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: kill 151367 2024-09-18T14:28:48.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: sleep 1 2024-09-18T14:28:49.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: kill 151367 2024-09-18T14:28:49.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 383: kill: (151367) - No such process 2024-09-18T14:28:49.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:384: TEST_bluestore2: ceph osd down 0 2024-09-18T14:28:50.075 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already down. 2024-09-18T14:28:50.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:386: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/block --command bluefs-bdev-migrate 2024-09-18T14:28:50.105 INFO:tasks.workunit.client.0.smithi047.stdout:inferring bluefs devices from bluestore path 2024-09-18T14:28:59.872 INFO:tasks.workunit.client.0.smithi047.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2024-09-18T14:28:59.949 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:391: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --command bluefs-bdev-sizes 2024-09-18T14:28:59.969 INFO:tasks.workunit.client.0.smithi047.stdout:inferring bluefs devices from bluestore path 2024-09-18T14:29:01.236 INFO:tasks.workunit.client.0.smithi047.stdout:1 : device size 0x100000000 : using 0x94129000(2.3 GiB) 2024-09-18T14:29:01.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:394: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --command fsck 2024-09-18T14:29:02.782 INFO:tasks.workunit.client.0.smithi047.stdout:fsck success 2024-09-18T14:29:02.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:397: TEST_bluestore2: activate_osd td/osd-bluefs-volume-ops 0 2024-09-18T14:29:02.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:29:02.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:29:02.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T14:29:02.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:29:02.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-18T14:29:02.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=a4c8b576-3caa-45c9-9ed5-fcd93777ab86 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-09-18T14:29:02.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:29:02.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:29:02.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:29:02.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-18T14:29:02.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-18T14:29:02.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:29:02.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:29:02.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-18T14:29:02.824 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:29:02.824 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:29:02.824 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:29:02.824 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:29:02.825 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:29:02.825 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:29:02.825 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:29:02.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:29:02.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:29:02.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-18T14:29:02.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-18T14:29:02.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:29:02.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:29:02.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:29:02.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:29:02.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:29:02.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:29:02.826 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-18T14:29:02.826 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T14:29:02.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T14:29:02.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=a4c8b576-3caa-45c9-9ed5-fcd93777ab86 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:29:02.827 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-18T14:29:02.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T14:29:02.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:29:02.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:29:02.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:29:02.851 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:29:02.850+0000 7faf84f23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:29:02.853 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:29:02.853+0000 7faf84f23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:29:02.854 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:29:02.854+0000 7faf84f23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:29:03.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T14:29:03.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:29:03.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:29:03.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:29:03.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:29:03.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:29:03.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:29:03.232 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:29:03.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:29:03.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:29:03.369 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:29:03.369+0000 7faf84f23440 -1 Falling back to public interface 2024-09-18T14:29:03.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:29:04.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:29:04.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:29:04.604 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:29:04.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:29:04.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:29:04.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:29:05.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:29:06.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:29:06.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:29:06.033 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:29:06.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:29:06.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:29:06.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:29:06.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:29:07.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:29:07.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:29:07.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:29:07.464 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:29:07.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:29:07.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:29:07.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:29:08.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:29:08.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:29:08.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:29:08.873 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:29:08.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:29:08.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:29:09.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:29:10.297 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:29:10.296+0000 7faf84f23440 -1 osd.0 10 log_to_monitors true 2024-09-18T14:29:10.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:29:10.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:29:10.394 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:29:10.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:29:10.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:29:10.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:29:10.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:29:11.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:29:11.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:29:11.901 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:29:11.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:29:11.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:29:11.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:29:13.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:29:14.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:29:14.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:29:14.276 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:29:14.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:29:14.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:29:14.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:29:15.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:29:15.176 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:29:15.175+0000 7faf84f23440 -1 bdev(0x559c04444800 td/osd-bluefs-volume-ops/0/block) aio_submit retries 8 2024-09-18T14:29:16.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:29:16.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:29:16.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:29:16.171 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:29:16.172 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:29:16.172 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:29:16.639 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 13 up_thru 7 down_at 11 last_clean_interval [5,10) [v2:127.0.0.1:6802/153899,v1:127.0.0.1:6803/153899] [v2:127.0.0.1:6804/153899,v1:127.0.0.1:6805/153899] exists,up 257388fb-a771-413d-8457-f0d3d1b980a9 2024-09-18T14:29:16.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:29:16.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:29:16.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:29:16.639 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:398: TEST_bluestore2: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-18T14:29:16.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:398: TEST_bluestore2: osd_pid0=153899 2024-09-18T14:29:16.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:400: TEST_bluestore2: wait_for_clean 2024-09-18T14:29:16.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:29:16.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:29:16.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:29:16.641 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:29:16.641 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:29:16.641 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:29:16.642 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:29:16.642 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:29:16.642 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:29:16.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T14:29:16.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:29:16.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:29:16.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:29:16.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:29:16.734 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:29:17.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T14:29:17.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:29:17.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:29:17.409 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:29:17.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574852 2024-09-18T14:29:17.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574852 2024-09-18T14:29:17.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-55834574852' 2024-09-18T14:29:17.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:29:17.524 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-55834574852 2024-09-18T14:29:17.524 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:29:17.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:29:17.525 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-55834574852 2024-09-18T14:29:17.525 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:29:17.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574852 2024-09-18T14:29:17.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 55834574852' 2024-09-18T14:29:17.527 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 55834574852 2024-09-18T14:29:17.527 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:29:18.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 2 -lt 55834574852 2024-09-18T14:29:18.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:29:19.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:29:19.115 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:29:19.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574852 -lt 55834574852 2024-09-18T14:29:19.689 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:29:19.689 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:29:19.689 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:29:20.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-09-18T14:29:20.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:29:20.360 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:29:20.360 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:29:20.360 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:29:20.360 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:29:20.360 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:29:20.360 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:29:20.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T14:29:20.801 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:29:20.801 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:29:20.802 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:29:21.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 16 2024-09-18T14:29:21.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' -1 2024-09-18T14:29:21.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T14:29:21.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=1 2024-09-18T14:29:21.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:29:21.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:29:21.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:29:21.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:29:21.589 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:29:21.589 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:29:21.589 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:29:21.589 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:29:21.589 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:29:21.590 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:29:21.988 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T14:29:21.989 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:29:21.989 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:29:21.989 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:29:22.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 16 2024-09-18T14:29:22.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T14:29:22.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:29:22.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:29:22.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:29:22.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:29:22.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:29:22.720 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:29:22.721 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:29:23.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:29:23.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:29:23.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-09-18T14:29:23.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:29:23.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-18T14:29:23.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:29:23.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:29:23.708 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:29:23.708 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:29:23.708 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:29:23.709 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:29:23.709 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:29:23.709 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:29:24.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-09-18T14:29:24.235 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:29:24.236 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:29:24.236 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:29:24.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-09-18T14:29:24.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:29:24.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:29:24.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-09-18T14:29:24.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:29:24.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:29:24.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-18T14:29:24.744 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:29:24.744 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:29:24.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:29:24.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:29:24.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:29:24.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:29:24.850 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:29:24.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:29:24.851 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:29:24.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:29:24.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:29:24.852 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:29:24.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:29:24.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:29:24.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:29:24.854 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:29:24.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:29:24.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:29:24.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-18T14:29:25.135 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:29:25.136 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:29:25.136 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:29:25.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.102152 2024-09-18T14:29:25.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:29:25.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:29:25.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-09-18T14:29:25.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-09-18T14:29:25.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:29:25.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2024-09-18T14:29:25.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:29:25.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:29:25.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-18T14:29:25.138 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:29:25.138 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:29:25.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:29:25.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:29:25.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:29:25.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:29:25.139 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:29:25.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:29:25.140 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:29:25.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:29:25.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:29:25.142 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:29:25.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:29:25.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:29:25.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:29:25.143 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:29:25.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:29:25.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:29:25.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-18T14:29:25.146 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:29:25.146 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:29:25.146 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:29:25.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.102152 2024-09-18T14:29:25.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:29:25.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:29:25.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2024-09-18T14:29:25.149 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T14:29:25.149 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:29:25.149 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:29:25.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.102152 2024-09-18T14:29:25.150 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T14:29:25.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T14:29:25.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T14:29:25.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-09-18T14:29:25.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore_expand td/osd-bluefs-volume-ops 2024-09-18T14:29:25.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:404: TEST_bluestore_expand: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:29:25.151 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:406: TEST_bluestore_expand: ulimit -n 2024-09-18T14:29:25.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:406: TEST_bluestore_expand: local flimit=4096 2024-09-18T14:29:25.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:407: TEST_bluestore_expand: '[' 4096 -lt 1536 ']' 2024-09-18T14:29:25.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:410: TEST_bluestore_expand: export CEPH_MON=127.0.0.1:7146 2024-09-18T14:29:25.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:410: TEST_bluestore_expand: CEPH_MON=127.0.0.1:7146 2024-09-18T14:29:25.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:411: TEST_bluestore_expand: export CEPH_ARGS 2024-09-18T14:29:25.152 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:412: TEST_bluestore_expand: uuidgen 2024-09-18T14:29:25.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:412: TEST_bluestore_expand: CEPH_ARGS+='--fsid=ba5077a7-e810-4624-83cd-4e82291db892 --auth-supported=none ' 2024-09-18T14:29:25.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:413: TEST_bluestore_expand: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-18T14:29:25.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:414: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_size=4294967296 ' 2024-09-18T14:29:25.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:415: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-09-18T14:29:25.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:416: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-09-18T14:29:25.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:417: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_wal_create=false ' 2024-09-18T14:29:25.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:418: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-09-18T14:29:25.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:419: TEST_bluestore_expand: CEPH_ARGS+='--osd_pool_default_size=1 ' 2024-09-18T14:29:25.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:420: TEST_bluestore_expand: CEPH_ARGS+='--osd_pool_default_min_size=1 ' 2024-09-18T14:29:25.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:421: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_debug_enforce_settings=ssd ' 2024-09-18T14:29:25.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:423: TEST_bluestore_expand: run_mon td/osd-bluefs-volume-ops a 2024-09-18T14:29:25.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:29:25.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T14:29:25.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T14:29:25.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T14:29:25.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-09-18T14:29:25.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-09-18T14:29:25.506 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T14:29:25.506 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:29:25.506 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:29:25.507 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:29:25.507 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:29:25.507 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:29:25.507 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:29:25.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T14:29:25.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T14:29:25.834 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T14:29:25.834 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:29:25.834 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:29:25.834 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T14:29:25.834 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T14:29:25.834 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:29:25.834 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:29:25.835 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:29:25.835 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:29:25.835 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:29:25.835 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:29:25.835 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.102152/ceph-mon.a.asok 2024-09-18T14:29:25.835 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:29:25.835 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.102152/ceph-mon.a.asok config get fsid 2024-09-18T14:29:25.914 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T14:29:25.914 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:29:25.914 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:29:25.914 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T14:29:25.914 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T14:29:25.914 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:29:25.914 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:29:25.914 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:29:25.915 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:29:25.915 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:29:25.915 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:29:25.915 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.102152/ceph-mon.a.asok 2024-09-18T14:29:25.915 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:29:25.915 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.102152/ceph-mon.a.asok config get mon_host 2024-09-18T14:29:25.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:424: TEST_bluestore_expand: run_mgr td/osd-bluefs-volume-ops x 2024-09-18T14:29:25.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:29:25.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T14:29:25.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T14:29:25.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T14:29:25.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-09-18T14:29:25.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T14:29:26.847 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T14:29:26.847 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:29:26.847 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:29:26.847 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:29:26.847 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:29:26.848 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:29:26.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:29:26.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:29:26.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:29:26.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:425: TEST_bluestore_expand: run_osd td/osd-bluefs-volume-ops 0 2024-09-18T14:29:26.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:29:26.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:29:26.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T14:29:26.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:29:26.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-18T14:29:26.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=a4c8b576-3caa-45c9-9ed5-fcd93777ab86 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=ba5077a7-e810-4624-83cd-4e82291db892 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-09-18T14:29:26.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:29:26.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:29:26.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:29:26.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-18T14:29:26.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-18T14:29:26.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:29:26.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:29:26.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-18T14:29:26.885 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:29:26.885 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:29:26.885 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:29:26.885 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:29:26.885 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:29:26.885 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:29:26.885 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:29:26.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:29:26.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:29:26.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:29:26.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:29:26.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-18T14:29:26.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-18T14:29:26.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:29:26.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:29:26.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:29:26.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:29:26.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:29:26.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:29:26.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-18T14:29:26.886 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:29:26.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7e39cdf1-b7de-4918-813f-ed6145153b2c 2024-09-18T14:29:26.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 7e39cdf1-b7de-4918-813f-ed6145153b2c' 2024-09-18T14:29:26.887 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 7e39cdf1-b7de-4918-813f-ed6145153b2c 2024-09-18T14:29:26.887 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:29:26.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDG4+pmS2r9NRAAhWIvuRKVeb4dlZmAh4sRcQ== 2024-09-18T14:29:26.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDG4+pmS2r9NRAAhWIvuRKVeb4dlZmAh4sRcQ=="}' 2024-09-18T14:29:26.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7e39cdf1-b7de-4918-813f-ed6145153b2c -i td/osd-bluefs-volume-ops/0/new.json 2024-09-18T14:29:27.260 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:29:27.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-09-18T14:29:27.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=a4c8b576-3caa-45c9-9ed5-fcd93777ab86 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=ba5077a7-e810-4624-83cd-4e82291db892 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDG4+pmS2r9NRAAhWIvuRKVeb4dlZmAh4sRcQ== --osd-uuid 7e39cdf1-b7de-4918-813f-ed6145153b2c 2024-09-18T14:29:27.304 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:29:27.303+0000 7efc26782440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:29:27.306 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:29:27.305+0000 7efc26782440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:29:27.307 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:29:27.306+0000 7efc26782440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:29:27.307 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:29:27.307+0000 7efc26782440 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-09-18T14:29:27.307 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:29:27.307+0000 7efc26782440 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-09-18T14:29:27.307 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:29:27.307+0000 7efc26782440 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-09-18T14:29:27.391 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:29:27.390+0000 7efc26782440 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-09-18T14:29:34.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-09-18T14:29:34.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:29:34.740 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T14:29:34.740 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T14:29:34.740 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:29:35.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T14:29:35.267 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T14:29:35.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:29:35.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=a4c8b576-3caa-45c9-9ed5-fcd93777ab86 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=ba5077a7-e810-4624-83cd-4e82291db892 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:29:35.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:29:35.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:29:35.300 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:29:35.299+0000 7f910c635440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:29:35.313 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:29:35.312+0000 7f910c635440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:29:35.314 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:29:35.314+0000 7f910c635440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:29:35.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T14:29:35.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:29:35.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:29:35.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:29:35.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:29:35.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:29:35.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:29:35.672 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:29:35.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:29:35.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:29:36.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:29:36.101 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:29:36.100+0000 7f910c635440 -1 Falling back to public interface 2024-09-18T14:29:37.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:29:37.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:29:37.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:29:37.049 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:29:37.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:29:37.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:29:37.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:29:38.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:29:38.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:29:38.480 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:29:38.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:29:38.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:29:38.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:29:38.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:29:39.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:29:39.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:29:39.889 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:29:39.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:29:39.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:29:39.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:29:40.014 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:29:40.013+0000 7f910c635440 -1 osd.0 0 log_to_monitors true 2024-09-18T14:29:40.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:29:41.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:29:41.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:29:41.304 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:29:41.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:29:41.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:29:41.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:29:42.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:29:43.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:29:43.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:29:43.342 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:29:43.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:29:43.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:29:43.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:29:44.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:29:45.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:29:45.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:29:45.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:29:45.049 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:29:45.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:29:45.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:29:45.441 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/156801,v1:127.0.0.1:6803/156801] [v2:127.0.0.1:6804/156801,v1:127.0.0.1:6805/156801] exists,up 7e39cdf1-b7de-4918-813f-ed6145153b2c 2024-09-18T14:29:45.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:29:45.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:29:45.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:29:45.442 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:426: TEST_bluestore_expand: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-18T14:29:45.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:426: TEST_bluestore_expand: osd_pid0=156801 2024-09-18T14:29:45.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:428: TEST_bluestore_expand: sleep 5 2024-09-18T14:29:50.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:429: TEST_bluestore_expand: create_pool foo 16 2024-09-18T14:29:50.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-09-18T14:29:51.022 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'foo' already exists 2024-09-18T14:29:51.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T14:29:52.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:432: TEST_bluestore_expand: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-18T14:29:52.074 INFO:tasks.workunit.client.0.smithi047.stdout:hints = 1 2024-09-18T14:29:52.074 INFO:tasks.workunit.client.0.smithi047.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-18T14:29:52.074 INFO:tasks.workunit.client.0.smithi047.stdout:Object prefix: benchmark_data_smithi047_157844 2024-09-18T14:29:52.074 INFO:tasks.workunit.client.0.smithi047.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T14:29:52.074 INFO:tasks.workunit.client.0.smithi047.stdout: 0 0 0 0 0 0 - 0 2024-09-18T14:29:53.074 INFO:tasks.workunit.client.0.smithi047.stdout: 1 16 131 115 0.449163 0.449219 0.134036 0.12135 2024-09-18T14:29:54.075 INFO:tasks.workunit.client.0.smithi047.stdout: 2 16 259 243 0.474541 0.5 0.125608 0.126707 2024-09-18T14:29:55.074 INFO:tasks.workunit.client.0.smithi047.stdout: 3 16 400 384 0.499936 0.550781 0.161745 0.123933 2024-09-18T14:29:56.074 INFO:tasks.workunit.client.0.smithi047.stdout: 4 16 544 528 0.515556 0.5625 0.117608 0.119553 2024-09-18T14:29:57.075 INFO:tasks.workunit.client.0.smithi047.stdout: 5 16 672 656 0.512426 0.5 0.150643 0.120382 2024-09-18T14:29:58.075 INFO:tasks.workunit.client.0.smithi047.stdout: 6 16 816 800 0.520757 0.5625 0.126052 0.118917 2024-09-18T14:29:59.075 INFO:tasks.workunit.client.0.smithi047.stdout: 7 16 944 928 0.517782 0.5 0.151096 0.119356 2024-09-18T14:30:00.075 INFO:tasks.workunit.client.0.smithi047.stdout: 8 16 1040 1024 0.499926 0.375 0.125677 0.124553 2024-09-18T14:30:01.075 INFO:tasks.workunit.client.0.smithi047.stdout: 9 16 1168 1152 0.499926 0.5 0.192086 0.124332 2024-09-18T14:30:02.075 INFO:tasks.workunit.client.0.smithi047.stdout: 10 16 1312 1296 0.506175 0.5625 0.125952 0.122827 2024-09-18T14:30:03.076 INFO:tasks.workunit.client.0.smithi047.stdout: 11 16 1427 1411 0.500992 0.449219 0.127976 0.122886 2024-09-18T14:30:04.076 INFO:tasks.workunit.client.0.smithi047.stdout: 12 16 1568 1552 0.505134 0.550781 0.0918121 0.123234 2024-09-18T14:30:05.076 INFO:tasks.workunit.client.0.smithi047.stdout: 13 16 1696 1680 0.504735 0.5 0.15075 0.123101 2024-09-18T14:30:06.076 INFO:tasks.workunit.client.0.smithi047.stdout: 14 16 1840 1824 0.508856 0.5625 0.100533 0.122482 2024-09-18T14:30:07.076 INFO:tasks.workunit.client.0.smithi047.stdout: 15 16 1955 1939 0.504876 0.449219 0.159399 0.122788 2024-09-18T14:30:08.076 INFO:tasks.workunit.client.0.smithi047.stdout: 16 16 2083 2067 0.504568 0.5 0.133749 0.123392 2024-09-18T14:30:09.076 INFO:tasks.workunit.client.0.smithi047.stdout: 17 16 2195 2179 0.500619 0.4375 0.142599 0.123836 2024-09-18T14:30:10.076 INFO:tasks.workunit.client.0.smithi047.stdout: 18 16 2339 2323 0.504052 0.5625 0.108961 0.123501 2024-09-18T14:30:11.077 INFO:tasks.workunit.client.0.smithi047.stdout: 19 16 2467 2451 0.503834 0.5 0.184519 0.123304 2024-09-18T14:30:12.077 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T14:30:12.077346+0000 min lat: 0.0517216 max lat: 0.425583 avg lat: 0.123611 2024-09-18T14:30:12.077 INFO:tasks.workunit.client.0.smithi047.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T14:30:12.077 INFO:tasks.workunit.client.0.smithi047.stdout: 20 16 2595 2579 0.503638 0.5 0.117241 0.123611 2024-09-18T14:30:13.078 INFO:tasks.workunit.client.0.smithi047.stdout: 21 16 2723 2707 0.503461 0.5 0.167452 0.123554 2024-09-18T14:30:14.077 INFO:tasks.workunit.client.0.smithi047.stdout: 22 16 2867 2851 0.50614 0.5625 0.0917694 0.123075 2024-09-18T14:30:15.077 INFO:tasks.workunit.client.0.smithi047.stdout: 23 16 2995 2979 0.505869 0.5 0.177795 0.12302 2024-09-18T14:30:16.077 INFO:tasks.workunit.client.0.smithi047.stdout: 24 16 3123 3107 0.505621 0.5 0.109225 0.123181 2024-09-18T14:30:17.078 INFO:tasks.workunit.client.0.smithi047.stdout: 25 16 3251 3235 0.505393 0.5 0.159159 0.12311 2024-09-18T14:30:18.078 INFO:tasks.workunit.client.0.smithi047.stdout: 26 16 3392 3376 0.507137 0.550781 0.12586 0.122869 2024-09-18T14:30:19.078 INFO:tasks.workunit.client.0.smithi047.stdout: 27 16 3520 3504 0.50687 0.5 0.142343 0.122981 2024-09-18T14:30:20.078 INFO:tasks.workunit.client.0.smithi047.stdout: 28 16 3648 3632 0.506622 0.5 0.117724 0.123293 2024-09-18T14:30:21.078 INFO:tasks.workunit.client.0.smithi047.stdout: 29 16 3763 3747 0.504641 0.449219 0.142617 0.123299 2024-09-18T14:30:22.078 INFO:tasks.workunit.client.0.smithi047.stdout: 30 16 3920 3904 0.508259 0.613281 0.100085 0.122765 2024-09-18T14:30:23.079 INFO:tasks.workunit.client.0.smithi047.stdout:Total time run: 30.0668 2024-09-18T14:30:23.079 INFO:tasks.workunit.client.0.smithi047.stdout:Total writes made: 3920 2024-09-18T14:30:23.079 INFO:tasks.workunit.client.0.smithi047.stdout:Write size: 4096 2024-09-18T14:30:23.079 INFO:tasks.workunit.client.0.smithi047.stdout:Object size: 4096 2024-09-18T14:30:23.079 INFO:tasks.workunit.client.0.smithi047.stdout:Bandwidth (MB/sec): 0.509282 2024-09-18T14:30:23.080 INFO:tasks.workunit.client.0.smithi047.stdout:Stddev Bandwidth: 0.0497474 2024-09-18T14:30:23.080 INFO:tasks.workunit.client.0.smithi047.stdout:Max bandwidth (MB/sec): 0.613281 2024-09-18T14:30:23.080 INFO:tasks.workunit.client.0.smithi047.stdout:Min bandwidth (MB/sec): 0.375 2024-09-18T14:30:23.080 INFO:tasks.workunit.client.0.smithi047.stdout:Average IOPS: 130 2024-09-18T14:30:23.080 INFO:tasks.workunit.client.0.smithi047.stdout:Stddev IOPS: 12.7353 2024-09-18T14:30:23.080 INFO:tasks.workunit.client.0.smithi047.stdout:Max IOPS: 157 2024-09-18T14:30:23.080 INFO:tasks.workunit.client.0.smithi047.stdout:Min IOPS: 96 2024-09-18T14:30:23.080 INFO:tasks.workunit.client.0.smithi047.stdout:Average Latency(s): 0.122686 2024-09-18T14:30:23.080 INFO:tasks.workunit.client.0.smithi047.stdout:Stddev Latency(s): 0.0288167 2024-09-18T14:30:23.080 INFO:tasks.workunit.client.0.smithi047.stdout:Max latency(s): 0.425583 2024-09-18T14:30:23.081 INFO:tasks.workunit.client.0.smithi047.stdout:Min latency(s): 0.0517216 2024-09-18T14:30:23.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:433: TEST_bluestore_expand: sleep 5 2024-09-18T14:30:28.129 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: ceph tell osd.0 perf dump bluefs 2024-09-18T14:30:28.130 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: jq .bluefs.slow_total_bytes 2024-09-18T14:30:28.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: total_space_before=4294967296 2024-09-18T14:30:28.267 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: ceph tell osd.0 bluestore bluefs device info 2024-09-18T14:30:28.267 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: grep BDEV_SLOW -A 2 2024-09-18T14:30:28.267 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: grep free 2024-09-18T14:30:28.267 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut -d: -f 2 2024-09-18T14:30:28.268 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut -d, -f 1 2024-09-18T14:30:28.268 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut '-d ' -f 2 2024-09-18T14:30:28.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: free_space_before=4278808576 2024-09-18T14:30:28.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: kill 156801 2024-09-18T14:30:28.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: sleep 1 2024-09-18T14:30:29.389 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: kill 156801 2024-09-18T14:30:29.389 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 439: kill: (156801) - No such process 2024-09-18T14:30:29.389 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:440: TEST_bluestore_expand: ceph osd down 0 2024-09-18T14:30:29.821 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already down. 2024-09-18T14:30:29.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:443: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 allocmap 2024-09-18T14:30:29.857 INFO:tasks.workunit.client.0.smithi047.stdout:allocmap bluestore.allocmap 2024-09-18T14:30:32.620 INFO:tasks.workunit.client.0.smithi047.stdout:allocmap success 2024-09-18T14:30:32.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:446: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 fsck 2024-09-18T14:30:35.669 INFO:tasks.workunit.client.0.smithi047.stdout:fsck success 2024-09-18T14:30:35.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:448: TEST_bluestore_expand: requested_space=4294967296 2024-09-18T14:30:35.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:449: TEST_bluestore_expand: truncate td/osd-bluefs-volume-ops/0/block -s 4294967296 2024-09-18T14:30:35.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:450: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 bluefs-bdev-expand 2024-09-18T14:30:35.710 INFO:tasks.workunit.client.0.smithi047.stdout:inferring bluefs devices from bluestore path 2024-09-18T14:30:37.886 INFO:tasks.workunit.client.0.smithi047.stdout:1 : device size 0x3fffe000 : using 0x12b00000(299 MiB) 2024-09-18T14:30:37.886 INFO:tasks.workunit.client.0.smithi047.stdout:2 : device size 0x100000000 : using 0xf69000(15 MiB) 2024-09-18T14:30:37.886 INFO:tasks.workunit.client.0.smithi047.stdout:Expanding DB/WAL... 2024-09-18T14:30:37.886 INFO:tasks.workunit.client.0.smithi047.stdout:1 : expanding to 0x1073741824 2024-09-18T14:30:37.914 INFO:tasks.workunit.client.0.smithi047.stdout:1 : size label updated to 1073741824 2024-09-18T14:30:38.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:453: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 fsck 2024-09-18T14:30:41.532 INFO:tasks.workunit.client.0.smithi047.stdout:fsck success 2024-09-18T14:30:41.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:456: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 qfsck 2024-09-18T14:30:41.572 INFO:tasks.workunit.client.0.smithi047.stdout:qfsck bluestore.quick-fsck 2024-09-18T14:30:44.335 INFO:tasks.workunit.client.0.smithi047.stdout:qfsck success 2024-09-18T14:30:44.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:458: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 bluefs-bdev-sizes 2024-09-18T14:30:44.368 INFO:tasks.workunit.client.0.smithi047.stdout:inferring bluefs devices from bluestore path 2024-09-18T14:30:47.133 INFO:tasks.workunit.client.0.smithi047.stdout:1 : device size 0x3fffe000 : using 0x12b00000(299 MiB) 2024-09-18T14:30:47.133 INFO:tasks.workunit.client.0.smithi047.stdout:2 : device size 0x100000000 : using 0xf69000(15 MiB) 2024-09-18T14:30:47.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:460: TEST_bluestore_expand: activate_osd td/osd-bluefs-volume-ops 0 2024-09-18T14:30:47.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:30:47.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:30:47.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T14:30:47.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:30:47.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-18T14:30:47.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=a4c8b576-3caa-45c9-9ed5-fcd93777ab86 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=ba5077a7-e810-4624-83cd-4e82291db892 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-09-18T14:30:47.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:30:47.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:30:47.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:30:47.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-18T14:30:47.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-18T14:30:47.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:30:47.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:30:47.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-18T14:30:47.141 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:30:47.142 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:30:47.142 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:30:47.142 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:30:47.142 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:30:47.142 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:30:47.142 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:30:47.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' 2024-09-18T14:30:47.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:30:47.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-18T14:30:47.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-18T14:30:47.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:30:47.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:30:47.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:30:47.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:30:47.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:30:47.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:30:47.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-18T14:30:47.143 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T14:30:47.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T14:30:47.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=9de9d400-c365-4042-8668-8a151201e201 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=a4c8b576-3caa-45c9-9ed5-fcd93777ab86 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=ba5077a7-e810-4624-83cd-4e82291db892 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102152/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:30:47.144 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-18T14:30:47.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T14:30:47.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:30:47.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:30:47.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:30:47.176 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:30:47.175+0000 7fbe0651e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:30:47.178 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:30:47.178+0000 7fbe0651e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:30:47.179 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:30:47.179+0000 7fbe0651e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:30:47.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T14:30:47.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:30:47.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:30:47.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:30:47.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:30:47.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:47.532 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:30:47.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:30:47.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:47.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:30:47.907 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:47.943 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:30:47.943+0000 7fbe0651e440 -1 Falling back to public interface 2024-09-18T14:30:48.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:48.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:48.909 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:30:48.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:30:48.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:48.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:30:49.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:50.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:50.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:50.332 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:30:50.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:30:50.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:50.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:30:50.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:51.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:51.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:51.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:30:51.753 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:30:51.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:51.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:30:52.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:52.836 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:30:52.835+0000 7fbe0651e440 -1 osd.0 10 log_to_monitors true 2024-09-18T14:30:53.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:53.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:53.146 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:30:53.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:30:53.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:53.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:30:54.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:55.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:55.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:55.122 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:30:55.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:30:55.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:55.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:30:55.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:56.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:56.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:56.507 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:30:56.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:30:56.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:56.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:30:57.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:30:58.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:30:58.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:30:58.140 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:30:58.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:30:58.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:30:58.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:30:58.560 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 13 up_thru 13 down_at 11 last_clean_interval [5,10) [v2:127.0.0.1:6802/160220,v1:127.0.0.1:6803/160220] [v2:127.0.0.1:6804/160220,v1:127.0.0.1:6805/160220] exists,up 7e39cdf1-b7de-4918-813f-ed6145153b2c 2024-09-18T14:30:58.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:30:58.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:30:58.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:30:58.561 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:461: TEST_bluestore_expand: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-18T14:30:58.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:461: TEST_bluestore_expand: osd_pid0=160220 2024-09-18T14:30:58.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:463: TEST_bluestore_expand: wait_for_clean 2024-09-18T14:30:58.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:30:58.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:30:58.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:30:58.563 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:30:58.563 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:30:58.563 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:30:58.564 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:30:58.564 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:30:58.564 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:30:58.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T14:30:58.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:30:58.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:30:58.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:30:58.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:30:58.655 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:30:59.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T14:30:59.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:30:59.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:30:59.039 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:30:59.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574852 2024-09-18T14:30:59.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574852 2024-09-18T14:30:59.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-55834574852' 2024-09-18T14:30:59.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:30:59.159 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-55834574852 2024-09-18T14:30:59.159 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:30:59.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:30:59.160 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-55834574852 2024-09-18T14:30:59.160 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:30:59.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574852 2024-09-18T14:30:59.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 55834574852' 2024-09-18T14:30:59.162 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 55834574852 2024-09-18T14:30:59.162 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:30:59.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574850 -lt 55834574852 2024-09-18T14:30:59.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:31:00.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:31:00.538 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:31:00.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574852 -lt 55834574852 2024-09-18T14:31:00.925 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:31:00.925 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:31:00.925 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:31:01.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-09-18T14:31:01.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:31:01.431 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:31:01.431 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:31:01.431 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:31:01.431 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:31:01.432 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:31:01.432 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:31:01.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-09-18T14:31:01.833 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:31:01.833 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:31:01.833 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:31:02.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-09-18T14:31:02.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:31:02.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:31:02.453 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:465: TEST_bluestore_expand: ceph tell osd.0 perf dump bluefs 2024-09-18T14:31:02.453 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:465: TEST_bluestore_expand: jq .bluefs.slow_total_bytes 2024-09-18T14:31:02.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:465: TEST_bluestore_expand: total_space_after=4294967296 2024-09-18T14:31:02.977 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: ceph tell osd.0 bluestore bluefs device info 2024-09-18T14:31:02.977 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: grep BDEV_SLOW -A 2 2024-09-18T14:31:02.977 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: grep free 2024-09-18T14:31:02.977 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: cut -d: -f 2 2024-09-18T14:31:02.978 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: cut -d, -f 1 2024-09-18T14:31:02.978 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: cut '-d ' -f 2 2024-09-18T14:31:03.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: free_space_after=4278763520 2024-09-18T14:31:03.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:468: TEST_bluestore_expand: '[4294967296' '!=' '4294967296]' 2024-09-18T14:31:03.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 468: [4294967296: command not found 2024-09-18T14:31:03.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:474: TEST_bluestore_expand: total_space_added=0 2024-09-18T14:31:03.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:475: TEST_bluestore_expand: free_space_added=-45056 2024-09-18T14:31:03.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:477: TEST_bluestore_expand: let 'new_used_space=(0 - -45056)' 2024-09-18T14:31:03.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:478: TEST_bluestore_expand: echo 45056 2024-09-18T14:31:03.164 INFO:tasks.workunit.client.0.smithi047.stdout:45056 2024-09-18T14:31:03.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:480: TEST_bluestore_expand: '[' 45056 -gt 131072 ']' 2024-09-18T14:31:03.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:487: TEST_bluestore_expand: kill 160220 2024-09-18T14:31:03.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:487: TEST_bluestore_expand: sleep 1 2024-09-18T14:31:04.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:487: TEST_bluestore_expand: kill 160220 2024-09-18T14:31:04.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:487: TEST_bluestore_expand: sleep 1 2024-09-18T14:31:05.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:487: TEST_bluestore_expand: kill 160220 2024-09-18T14:31:05.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 487: kill: (160220) - No such process 2024-09-18T14:31:05.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:488: TEST_bluestore_expand: ceph osd down 0 2024-09-18T14:31:05.592 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already down. 2024-09-18T14:31:05.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:490: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 qfsck 2024-09-18T14:31:05.627 INFO:tasks.workunit.client.0.smithi047.stdout:qfsck bluestore.quick-fsck 2024-09-18T14:31:08.141 INFO:tasks.workunit.client.0.smithi047.stdout:qfsck success 2024-09-18T14:31:08.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-09-18T14:31:08.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:31:08.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:31:08.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-18T14:31:08.148 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:31:08.148 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:31:08.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:31:08.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:31:08.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:31:08.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:31:08.255 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:31:08.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:31:08.256 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:31:08.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:31:08.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:31:08.258 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:31:08.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:31:08.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:31:08.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:31:08.259 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:31:08.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:31:08.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:31:08.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-18T14:31:08.336 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:31:08.336 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:31:08.336 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:31:08.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.102152 2024-09-18T14:31:08.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:31:08.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:31:08.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T14:31:08.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/osd-bluefs-volume-ops 0 2024-09-18T14:31:08.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-18T14:31:08.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T14:31:08.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-18T14:31:08.338 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:31:08.338 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:31:08.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:31:08.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:31:08.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:31:08.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:31:08.339 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:31:08.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:31:08.341 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:31:08.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:31:08.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:31:08.342 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:31:08.343 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:31:08.343 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:31:08.343 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:31:08.343 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:31:08.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:31:08.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T14:31:08.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-18T14:31:08.346 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:31:08.346 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:31:08.347 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102152 2024-09-18T14:31:08.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.102152 2024-09-18T14:31:08.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:31:08.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:31:08.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T14:31:08.348 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T14:31:08.349 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T14:31:08.375 INFO:tasks.workunit:Running workunit osd/osd-config.sh... 2024-09-18T14:31:08.375 DEBUG:teuthology.orchestra.run.smithi047:workunit test osd/osd-config.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh 2024-09-18T14:31:08.438 INFO:tasks.workunit.client.0.smithi047.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T14:31:08.443 INFO:tasks.workunit.client.0.smithi047.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T14:31:08.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T14:31:08.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T14:31:08.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T14:31:08.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T14:31:08.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T14:31:08.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T14:31:08.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T14:31:08.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T14:31:08.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/osd-config 2024-09-18T14:31:08.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:22: run: local dir=td/osd-config 2024-09-18T14:31:08.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:23: run: shift 2024-09-18T14:31:08.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:25: run: export CEPH_MON=127.0.0.1:7100 2024-09-18T14:31:08.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:25: run: CEPH_MON=127.0.0.1:7100 2024-09-18T14:31:08.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:26: run: export CEPH_ARGS 2024-09-18T14:31:08.444 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:27: run: uuidgen 2024-09-18T14:31:08.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:27: run: CEPH_ARGS+='--fsid=a5a84035-7fec-4a7a-9b75-4292b62674ff --auth-supported=none ' 2024-09-18T14:31:08.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7100 ' 2024-09-18T14:31:08.446 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: set 2024-09-18T14:31:08.446 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-18T14:31:08.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: local 'funcs=TEST_config_init 2024-09-18T14:31:08.447 INFO:tasks.workunit.client.0.smithi047.stderr:TEST_config_track 2024-09-18T14:31:08.447 INFO:tasks.workunit.client.0.smithi047.stderr:TEST_default_adjustment' 2024-09-18T14:31:08.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2024-09-18T14:31:08.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2024-09-18T14:31:08.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2024-09-18T14:31:08.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2024-09-18T14:31:08.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-09-18T14:31:08.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:31:08.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-09-18T14:31:08.447 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:31:08.447 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:31:08.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:31:08.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:31:08.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:31:08.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:31:08.448 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:31:08.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:31:08.449 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:31:08.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:31:08.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:31:08.450 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:31:08.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:31:08.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:31:08.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:31:08.451 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:31:08.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:31:08.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:31:08.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-09-18T14:31:08.454 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:31:08.454 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:31:08.454 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162020 2024-09-18T14:31:08.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.162020 2024-09-18T14:31:08.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:31:08.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:31:08.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2024-09-18T14:31:08.456 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T14:31:08.456 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:31:08.456 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162020 2024-09-18T14:31:08.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.162020 2024-09-18T14:31:08.458 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T14:31:08.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T14:31:08.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T14:31:08.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T14:31:08.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2024-09-18T14:31:08.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_config_init td/osd-config 2024-09-18T14:31:08.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:39: TEST_config_init: local dir=td/osd-config 2024-09-18T14:31:08.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:41: TEST_config_init: run_mon td/osd-config a 2024-09-18T14:31:08.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-config 2024-09-18T14:31:08.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T14:31:08.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T14:31:08.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T14:31:08.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-config/a 2024-09-18T14:31:08.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-config/a --run-dir=td/osd-config 2024-09-18T14:31:08.817 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T14:31:08.817 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:31:08.817 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:31:08.817 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:31:08.817 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:31:08.817 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162020 2024-09-18T14:31:08.817 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.162020/$cluster-$name.asok' 2024-09-18T14:31:08.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-config/a '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.162020/$cluster-$name.asok' --mon-cluster-log-file=td/osd-config/log --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T14:31:09.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T14:31:09.061 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T14:31:09.061 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:31:09.062 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:31:09.062 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T14:31:09.062 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T14:31:09.062 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:31:09.062 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:31:09.062 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:31:09.062 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:31:09.062 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:31:09.062 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162020 2024-09-18T14:31:09.062 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.162020/ceph-mon.a.asok 2024-09-18T14:31:09.063 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:31:09.063 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.162020/ceph-mon.a.asok config get fsid 2024-09-18T14:31:09.141 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T14:31:09.141 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:31:09.141 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:31:09.141 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T14:31:09.142 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T14:31:09.142 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:31:09.142 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:31:09.142 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:31:09.142 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:31:09.142 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:31:09.142 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162020 2024-09-18T14:31:09.142 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.162020/ceph-mon.a.asok 2024-09-18T14:31:09.142 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:31:09.142 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.162020/ceph-mon.a.asok config get mon_host 2024-09-18T14:31:09.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:42: TEST_config_init: run_mgr td/osd-config x 2024-09-18T14:31:09.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-config 2024-09-18T14:31:09.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T14:31:09.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T14:31:09.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T14:31:09.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-config/x 2024-09-18T14:31:09.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T14:31:10.058 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T14:31:10.058 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:31:10.058 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:31:10.058 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:31:10.058 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:31:10.058 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162020 2024-09-18T14:31:10.058 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.162020/$cluster-$name.asok' 2024-09-18T14:31:10.058 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:31:10.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-config/x '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.162020/$cluster-$name.asok' --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:31:10.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:43: TEST_config_init: local stale=1000 2024-09-18T14:31:10.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:44: TEST_config_init: local cache=500 2024-09-18T14:31:10.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:45: TEST_config_init: run_osd td/osd-config 0 --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 2024-09-18T14:31:10.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-config 2024-09-18T14:31:10.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:31:10.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T14:31:10.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:31:10.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-config/0 2024-09-18T14:31:10.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a5a84035-7fec-4a7a-9b75-4292b62674ff --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-18T14:31:10.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:31:10.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:31:10.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:31:10.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-config/0' 2024-09-18T14:31:10.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-config/0/journal' 2024-09-18T14:31:10.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:31:10.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:31:10.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-config' 2024-09-18T14:31:10.085 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:31:10.085 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:31:10.085 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:31:10.085 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:31:10.085 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:31:10.085 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162020 2024-09-18T14:31:10.085 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.162020/$cluster-$name.asok' 2024-09-18T14:31:10.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.162020/$cluster-$name.asok' 2024-09-18T14:31:10.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:31:10.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:31:10.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:31:10.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-config/$name.log' 2024-09-18T14:31:10.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-config/$name.pid' 2024-09-18T14:31:10.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:31:10.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:31:10.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:31:10.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:31:10.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:31:10.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000' 2024-09-18T14:31:10.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-config/0 2024-09-18T14:31:10.086 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:31:10.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3259d9d9-6039-4388-b639-824d1928e1f2 2024-09-18T14:31:10.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3259d9d9-6039-4388-b639-824d1928e1f2' 2024-09-18T14:31:10.087 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 3259d9d9-6039-4388-b639-824d1928e1f2 2024-09-18T14:31:10.088 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:31:10.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAu5OpmvwPSBRAAxwLo0bf10UdDMG4Kwt8LMg== 2024-09-18T14:31:10.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAu5OpmvwPSBRAAxwLo0bf10UdDMG4Kwt8LMg=="}' 2024-09-18T14:31:10.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3259d9d9-6039-4388-b639-824d1928e1f2 -i td/osd-config/0/new.json 2024-09-18T14:31:10.401 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:31:10.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-config/0/new.json 2024-09-18T14:31:10.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=a5a84035-7fec-4a7a-9b75-4292b62674ff --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.162020/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 --mkfs --key AQAu5OpmvwPSBRAAxwLo0bf10UdDMG4Kwt8LMg== --osd-uuid 3259d9d9-6039-4388-b639-824d1928e1f2 2024-09-18T14:31:10.435 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:31:10.434+0000 7f1ed482f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:31:10.452 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:31:10.452+0000 7f1ed482f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:31:10.454 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:31:10.453+0000 7f1ed482f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:31:10.454 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:31:10.453+0000 7f1ed482f440 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-09-18T14:31:10.454 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:31:10.453+0000 7f1ed482f440 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-09-18T14:31:10.454 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:31:10.453+0000 7f1ed482f440 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-09-18T14:31:10.493 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:31:10.492+0000 7f1ed482f440 -1 bluestore(td/osd-config/0) _read_fsid unparsable uuid 2024-09-18T14:31:15.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-config/0/keyring 2024-09-18T14:31:15.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:31:15.790 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T14:31:15.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T14:31:15.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:31:16.303 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T14:31:16.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T14:31:16.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=a5a84035-7fec-4a7a-9b75-4292b62674ff --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.162020/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 2024-09-18T14:31:16.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:31:16.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:31:16.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:31:16.326 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:31:16.325+0000 7f965cf1e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:31:16.328 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:31:16.327+0000 7f965cf1e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:31:16.329 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:31:16.328+0000 7f965cf1e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:31:16.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T14:31:16.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:31:16.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:31:16.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:31:16.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:31:16.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:16.697 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:31:16.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:31:16.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:16.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:31:16.854 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:31:16.853+0000 7f965cf1e440 -1 Falling back to public interface 2024-09-18T14:31:17.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:18.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:18.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:18.115 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:31:18.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:31:18.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:18.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:31:18.491 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:19.337 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:31:19.336+0000 7f965cf1e440 -1 osd.0 0 log_to_monitors true 2024-09-18T14:31:19.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:19.493 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:31:19.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:19.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:31:19.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:19.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:31:19.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:20.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:20.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:20.925 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:31:20.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:31:20.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:20.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:31:21.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:22.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:22.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:22.325 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:31:22.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:31:22.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:22.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:31:22.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:23.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:23.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:23.784 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:31:23.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:31:23.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:23.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:31:24.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:25.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:25.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:25.158 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:31:25.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:31:25.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:25.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:31:25.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:26.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:26.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:26.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:31:26.543 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:31:26.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:26.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:31:27.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:28.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:28.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:28.027 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:31:28.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:31:28.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:28.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:31:28.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:29.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:29.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:29.442 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:31:29.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:31:29.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:29.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:31:29.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:30.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:30.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:30.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:31:30.819 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:31:30.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:30.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:31:31.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:32.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:32.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:32.531 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:31:32.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:31:32.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:32.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:31:33.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:34.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:34.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:34.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:31:34.061 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:31:34.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:34.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:31:35.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:36.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:36.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:36.090 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:31:36.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:31:36.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:36.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:31:36.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:37.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:37.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:37.832 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:31:37.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:31:37.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:37.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:31:38.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:39.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:39.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:39.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:31:39.768 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:31:39.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:39.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:31:40.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:41.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:41.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:41.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:31:41.276 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:31:41.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:41.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:31:41.654 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/163111,v1:127.0.0.1:6803/163111] [v2:127.0.0.1:6804/163111,v1:127.0.0.1:6805/163111] exists,up 3259d9d9-6039-4388-b639-824d1928e1f2 2024-09-18T14:31:41.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:31:41.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:31:41.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:31:41.656 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: get_asok_path osd.0 2024-09-18T14:31:41.656 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:31:41.656 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:31:41.656 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:31:41.656 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:31:41.656 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162020 2024-09-18T14:31:41.656 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.162020/ceph-osd.0.asok 2024-09-18T14:31:41.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: CEPH_ARGS= 2024-09-18T14:31:41.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: ceph --admin-daemon /tmp/ceph-asok.162020/ceph-osd.0.asok log flush 2024-09-18T14:31:41.733 INFO:tasks.workunit.client.0.smithi047.stdout:{} 2024-09-18T14:31:41.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:50: TEST_config_init: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2024-09-18T14:31:41.741 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T14:31:19.649+0000 7f965cf1e440 0 log_channel(cluster) log [WRN] : osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-09-18T14:31:41.741 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T14:31:22.336+0000 7f964c5b5640 10 log_client will send 2024-09-18T14:31:19.650270+0000 osd.0 (osd.0) 1 : cluster [WRN] osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-09-18T14:31:41.741 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T14:31:22.636+0000 7f964d5b7640 10 log_client logged 2024-09-18T14:31:19.650270+0000 osd.0 (osd.0) 1 : cluster [WRN] osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-09-18T14:31:41.741 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T14:31:40.934+0000 7f96589f8640 0 log_channel(cluster) log [WRN] : osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-09-18T14:31:41.741 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T14:31:41.338+0000 7f964c5b5640 10 log_client will send 2024-09-18T14:31:40.935582+0000 osd.0 (osd.0) 4 : cluster [WRN] osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-09-18T14:31:41.741 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T14:31:41.604+0000 7f964d5b7640 10 log_client logged 2024-09-18T14:31:40.935582+0000 osd.0 (osd.0) 4 : cluster [WRN] osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-09-18T14:31:41.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2024-09-18T14:31:41.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-09-18T14:31:41.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:31:41.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-09-18T14:31:41.742 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:31:41.743 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:31:41.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:31:41.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:31:41.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:31:41.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:31:41.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:31:41.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:31:41.849 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:31:41.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:31:41.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:31:41.850 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:31:41.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:31:41.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:31:41.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:31:41.852 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:31:41.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:31:41.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:31:41.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-09-18T14:31:41.858 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:31:41.858 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:31:41.859 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162020 2024-09-18T14:31:41.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.162020 2024-09-18T14:31:41.860 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:31:41.860 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:31:41.861 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2024-09-18T14:31:41.861 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2024-09-18T14:31:41.861 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2024-09-18T14:31:41.861 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2024-09-18T14:31:41.861 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-09-18T14:31:41.861 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:31:41.861 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-09-18T14:31:41.861 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:31:41.861 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:31:41.861 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:31:41.861 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:31:41.861 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:31:41.862 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:31:41.862 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:31:41.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:31:41.863 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:31:41.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:31:41.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:31:41.864 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:31:41.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:31:41.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:31:41.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:31:41.866 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:31:41.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:31:41.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:31:41.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-09-18T14:31:41.868 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:31:41.868 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:31:41.869 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162020 2024-09-18T14:31:41.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.162020 2024-09-18T14:31:41.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:31:41.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:31:41.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2024-09-18T14:31:41.871 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T14:31:41.871 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:31:41.871 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162020 2024-09-18T14:31:41.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.162020 2024-09-18T14:31:41.873 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T14:31:41.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T14:31:41.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T14:31:41.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2024-09-18T14:31:41.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_config_track td/osd-config 2024-09-18T14:31:41.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:54: TEST_config_track: local dir=td/osd-config 2024-09-18T14:31:41.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:56: TEST_config_track: run_mon td/osd-config a 2024-09-18T14:31:41.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-config 2024-09-18T14:31:41.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T14:31:41.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T14:31:41.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T14:31:41.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-config/a 2024-09-18T14:31:41.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-config/a --run-dir=td/osd-config 2024-09-18T14:31:42.237 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T14:31:42.237 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:31:42.237 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:31:42.237 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:31:42.237 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:31:42.237 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162020 2024-09-18T14:31:42.237 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.162020/$cluster-$name.asok' 2024-09-18T14:31:42.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-config/a '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.162020/$cluster-$name.asok' --mon-cluster-log-file=td/osd-config/log --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T14:31:42.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T14:31:42.481 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T14:31:42.481 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:31:42.481 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:31:42.481 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T14:31:42.481 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T14:31:42.482 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:31:42.482 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:31:42.482 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:31:42.482 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:31:42.482 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:31:42.482 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162020 2024-09-18T14:31:42.482 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.162020/ceph-mon.a.asok 2024-09-18T14:31:42.482 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:31:42.483 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.162020/ceph-mon.a.asok config get fsid 2024-09-18T14:31:42.562 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T14:31:42.563 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:31:42.563 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:31:42.563 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T14:31:42.563 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T14:31:42.563 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:31:42.563 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:31:42.563 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:31:42.563 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:31:42.563 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:31:42.563 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162020 2024-09-18T14:31:42.563 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.162020/ceph-mon.a.asok 2024-09-18T14:31:42.563 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:31:42.564 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.162020/ceph-mon.a.asok config get mon_host 2024-09-18T14:31:42.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:57: TEST_config_track: run_mgr td/osd-config x 2024-09-18T14:31:42.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-config 2024-09-18T14:31:42.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T14:31:42.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T14:31:42.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T14:31:42.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-config/x 2024-09-18T14:31:42.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T14:31:43.484 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T14:31:43.485 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:31:43.485 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:31:43.485 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:31:43.485 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:31:43.485 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162020 2024-09-18T14:31:43.485 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.162020/$cluster-$name.asok' 2024-09-18T14:31:43.485 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:31:43.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-config/x '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.162020/$cluster-$name.asok' --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:31:43.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:58: TEST_config_track: run_osd td/osd-config 0 2024-09-18T14:31:43.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-config 2024-09-18T14:31:43.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:31:43.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T14:31:43.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:31:43.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-config/0 2024-09-18T14:31:43.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a5a84035-7fec-4a7a-9b75-4292b62674ff --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-18T14:31:43.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:31:43.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:31:43.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:31:43.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-config/0' 2024-09-18T14:31:43.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-config/0/journal' 2024-09-18T14:31:43.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:31:43.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:31:43.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-config' 2024-09-18T14:31:43.511 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:31:43.511 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:31:43.511 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:31:43.511 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:31:43.512 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:31:43.512 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162020 2024-09-18T14:31:43.512 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.162020/$cluster-$name.asok' 2024-09-18T14:31:43.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.162020/$cluster-$name.asok' 2024-09-18T14:31:43.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:31:43.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:31:43.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:31:43.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-config/$name.log' 2024-09-18T14:31:43.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-config/$name.pid' 2024-09-18T14:31:43.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:31:43.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:31:43.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:31:43.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:31:43.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:31:43.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:31:43.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-config/0 2024-09-18T14:31:43.513 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:31:43.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8c492a53-3a81-4592-8e96-dbe235acf690 2024-09-18T14:31:43.513 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 8c492a53-3a81-4592-8e96-dbe235acf690 2024-09-18T14:31:43.514 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8c492a53-3a81-4592-8e96-dbe235acf690' 2024-09-18T14:31:43.514 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:31:43.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBP5Opmq+k4HxAA9rDCAlcKrEAhFJxUcEoPrw== 2024-09-18T14:31:43.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBP5Opmq+k4HxAA9rDCAlcKrEAhFJxUcEoPrw=="}' 2024-09-18T14:31:43.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8c492a53-3a81-4592-8e96-dbe235acf690 -i td/osd-config/0/new.json 2024-09-18T14:31:43.819 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:31:43.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-config/0/new.json 2024-09-18T14:31:43.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=a5a84035-7fec-4a7a-9b75-4292b62674ff --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.162020/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBP5Opmq+k4HxAA9rDCAlcKrEAhFJxUcEoPrw== --osd-uuid 8c492a53-3a81-4592-8e96-dbe235acf690 2024-09-18T14:31:43.852 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:31:43.852+0000 7f615ea13440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:31:43.864 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:31:43.863+0000 7f615ea13440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:31:43.865 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:31:43.864+0000 7f615ea13440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:31:43.865 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:31:43.865+0000 7f615ea13440 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-09-18T14:31:43.866 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:31:43.865+0000 7f615ea13440 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-09-18T14:31:43.866 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:31:43.865+0000 7f615ea13440 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-09-18T14:31:43.904 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:31:43.903+0000 7f615ea13440 -1 bluestore(td/osd-config/0) _read_fsid unparsable uuid 2024-09-18T14:31:49.248 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-config/0/keyring 2024-09-18T14:31:49.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:31:49.249 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T14:31:49.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T14:31:49.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:31:49.773 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T14:31:49.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T14:31:49.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=a5a84035-7fec-4a7a-9b75-4292b62674ff --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.162020/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:31:49.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:31:49.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:31:49.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:31:49.793 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:31:49.793+0000 7f8fb2e0a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:31:49.818 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:31:49.817+0000 7f8fb2e0a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:31:49.819 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:31:49.819+0000 7f8fb2e0a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:31:50.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T14:31:50.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:31:50.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:31:50.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:31:50.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:31:50.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:50.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:31:50.182 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:31:50.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:50.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:31:50.356 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:31:50.355+0000 7f8fb2e0a440 -1 Falling back to public interface 2024-09-18T14:31:50.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:51.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:51.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:51.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:31:51.593 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:31:51.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:51.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:31:52.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:52.830 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:31:52.830+0000 7f8fb2e0a440 -1 osd.0 0 log_to_monitors true 2024-09-18T14:31:53.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:53.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:53.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:31:53.018 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:31:53.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:53.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:31:53.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:54.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:54.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:54.436 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:31:54.437 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:31:54.437 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:54.437 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:31:54.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:55.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:55.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:55.849 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:31:55.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:31:55.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:55.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:31:56.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:57.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:57.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:57.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:31:57.234 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:31:57.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:57.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:31:57.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:31:58.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:31:58.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:31:58.645 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:31:58.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:31:58.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:31:58.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:31:59.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:00.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:00.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:00.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:32:00.045 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:32:00.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:00.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:32:00.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:01.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:01.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:01.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:32:01.521 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:32:01.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:01.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:32:01.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:02.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:02.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:02.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:32:02.906 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:32:02.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:02.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:32:03.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:04.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:04.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:04.311 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:32:04.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:32:04.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:04.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:32:05.022 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:06.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:06.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:06.024 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:32:06.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:32:06.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:06.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:32:06.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:07.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:07.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:07.513 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:32:07.514 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:32:07.514 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:07.514 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:32:08.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:09.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:09.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:09.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:32:09.321 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:32:09.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:09.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:32:10.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:11.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:11.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:11.056 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:32:11.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:32:11.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:11.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:32:11.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:12.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:12.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:12.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:32:12.561 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:32:12.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:12.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:32:13.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:14.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:14.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:14.235 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:32:14.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:32:14.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:14.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:32:14.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:15.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:15.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:15.972 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T14:32:15.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:32:15.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:15.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:32:16.369 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/165111,v1:127.0.0.1:6803/165111] [v2:127.0.0.1:6804/165111,v1:127.0.0.1:6805/165111] exists,up 8c492a53-3a81-4592-8e96-dbe235acf690 2024-09-18T14:32:16.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:32:16.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:32:16.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:32:16.370 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: CEPH_ARGS= 2024-09-18T14:32:16.370 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: ceph-conf --show-config-value osd_map_cache_size 2024-09-18T14:32:16.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: local osd_map_cache_size=50 2024-09-18T14:32:16.380 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: CEPH_ARGS= 2024-09-18T14:32:16.380 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: ceph-conf --show-config-value osd_pg_epoch_persisted_max_stale 2024-09-18T14:32:16.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: local osd_pg_epoch_persisted_max_stale=40 2024-09-18T14:32:16.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:68: TEST_config_track: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2024-09-18T14:32:16.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:69: TEST_config_track: local stale=100 2024-09-18T14:32:16.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:70: TEST_config_track: ceph tell osd.0 injectargs '--osd-pg-epoch-persisted-max-stale 100' 2024-09-18T14:32:16.536 INFO:tasks.workunit.client.0.smithi047.stdout:{} 2024-09-18T14:32:16.546 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: get_asok_path osd.0 2024-09-18T14:32:16.546 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:32:16.546 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:32:16.546 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:32:16.546 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:32:16.546 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162020 2024-09-18T14:32:16.546 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.162020/ceph-osd.0.asok 2024-09-18T14:32:16.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: CEPH_ARGS= 2024-09-18T14:32:16.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: ceph --admin-daemon /tmp/ceph-asok.162020/ceph-osd.0.asok log flush 2024-09-18T14:32:16.622 INFO:tasks.workunit.client.0.smithi047.stdout:{} 2024-09-18T14:32:16.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:72: TEST_config_track: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2024-09-18T14:32:16.630 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T14:32:16.526+0000 7f8fb05fc640 0 log_channel(cluster) log [WRN] : osd_map_cache_size (50) is not > osd_pg_epoch_persisted_max_stale (100) 2024-09-18T14:32:16.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:73: TEST_config_track: rm td/osd-config/osd.0.log 2024-09-18T14:32:16.632 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: get_asok_path osd.0 2024-09-18T14:32:16.632 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:32:16.632 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:32:16.632 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:32:16.632 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:32:16.632 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162020 2024-09-18T14:32:16.632 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.162020/ceph-osd.0.asok 2024-09-18T14:32:16.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: CEPH_ARGS= 2024-09-18T14:32:16.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: ceph --admin-daemon /tmp/ceph-asok.162020/ceph-osd.0.asok log reopen 2024-09-18T14:32:16.709 INFO:tasks.workunit.client.0.smithi047.stdout:{} 2024-09-18T14:32:16.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2024-09-18T14:32:16.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-09-18T14:32:16.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:32:16.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-09-18T14:32:16.716 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:32:16.717 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:32:16.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:32:16.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:32:16.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:32:16.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:32:16.823 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:32:16.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:32:16.823 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:32:16.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:32:16.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:32:16.825 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:32:16.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:32:16.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:32:16.826 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:32:16.826 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:32:16.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:32:16.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:32:16.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-09-18T14:32:16.833 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:32:16.833 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:32:16.833 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162020 2024-09-18T14:32:16.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.162020 2024-09-18T14:32:16.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:32:16.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:32:16.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2024-09-18T14:32:16.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2024-09-18T14:32:16.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2024-09-18T14:32:16.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2024-09-18T14:32:16.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-09-18T14:32:16.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:32:16.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-09-18T14:32:16.836 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:32:16.836 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:32:16.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:32:16.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:32:16.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:32:16.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:32:16.836 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:32:16.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:32:16.837 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:32:16.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:32:16.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:32:16.839 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:32:16.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:32:16.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:32:16.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:32:16.840 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:32:16.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:32:16.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:32:16.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-09-18T14:32:16.843 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:32:16.843 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:32:16.843 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162020 2024-09-18T14:32:16.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.162020 2024-09-18T14:32:16.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:32:16.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:32:16.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2024-09-18T14:32:16.845 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T14:32:16.845 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:32:16.845 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162020 2024-09-18T14:32:16.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.162020 2024-09-18T14:32:16.846 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T14:32:16.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T14:32:16.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T14:32:16.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2024-09-18T14:32:16.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_default_adjustment td/osd-config 2024-09-18T14:32:16.847 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:78: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin 2024-09-18T14:32:16.860 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:78: TEST_default_adjustment: a= 2024-09-18T14:32:16.860 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:79: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin --default-rgw-torrent-origin default 2024-09-18T14:32:16.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:79: TEST_default_adjustment: b=default 2024-09-18T14:32:16.875 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:80: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin --default-rgw-torrent-origin arg 2024-09-18T14:32:16.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:80: TEST_default_adjustment: c=arg 2024-09-18T14:32:16.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:81: TEST_default_adjustment: '[' '' '!=' default ']' 2024-09-18T14:32:16.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:82: TEST_default_adjustment: '[' default = default ']' 2024-09-18T14:32:16.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:83: TEST_default_adjustment: '[' arg = arg ']' 2024-09-18T14:32:16.891 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:85: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file 2024-09-18T14:32:16.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:85: TEST_default_adjustment: a=true 2024-09-18T14:32:16.904 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:86: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file --default-log-to-file=false 2024-09-18T14:32:16.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:86: TEST_default_adjustment: b=false 2024-09-18T14:32:16.918 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:87: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file --default-log-to-file=false --log-to-file 2024-09-18T14:32:16.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:87: TEST_default_adjustment: c=true 2024-09-18T14:32:16.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:88: TEST_default_adjustment: '[' true = true ']' 2024-09-18T14:32:16.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:89: TEST_default_adjustment: '[' false = false ']' 2024-09-18T14:32:16.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:90: TEST_default_adjustment: '[' true = true ']' 2024-09-18T14:32:16.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2024-09-18T14:32:16.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-09-18T14:32:16.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:32:16.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-09-18T14:32:16.933 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:32:16.933 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:32:16.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:32:16.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:32:16.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:32:16.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:32:16.934 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:32:16.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:32:16.935 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:32:16.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:32:16.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:32:16.937 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:32:16.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:32:16.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:32:16.938 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:32:16.938 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:32:16.938 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:32:16.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:32:16.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-09-18T14:32:16.941 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:32:16.941 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:32:16.941 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162020 2024-09-18T14:32:16.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.162020 2024-09-18T14:32:16.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:32:16.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:32:16.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T14:32:16.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/osd-config 0 2024-09-18T14:32:16.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-09-18T14:32:16.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T14:32:16.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-09-18T14:32:16.943 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:32:16.943 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:32:16.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:32:16.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:32:16.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:32:16.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:32:16.944 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:32:16.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:32:16.945 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:32:16.946 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:32:16.946 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:32:16.946 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:32:16.947 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:32:16.947 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:32:16.947 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:32:16.947 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:32:16.948 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:32:16.949 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T14:32:16.949 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-09-18T14:32:16.950 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:32:16.950 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:32:16.950 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162020 2024-09-18T14:32:16.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.162020 2024-09-18T14:32:16.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:32:16.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:32:16.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T14:32:16.952 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T14:32:16.953 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T14:32:17.018 INFO:tasks.workunit:Running workunit osd/osd-copy-from.sh... 2024-09-18T14:32:17.020 DEBUG:teuthology.orchestra.run.smithi047:workunit test osd/osd-copy-from.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh 2024-09-18T14:32:17.080 INFO:tasks.workunit.client.0.smithi047.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T14:32:17.085 INFO:tasks.workunit.client.0.smithi047.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T14:32:17.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T14:32:17.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T14:32:17.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T14:32:17.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T14:32:17.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T14:32:17.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T14:32:17.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T14:32:17.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T14:32:17.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/osd-copy-from 2024-09-18T14:32:17.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:23: run: local dir=td/osd-copy-from 2024-09-18T14:32:17.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:24: run: shift 2024-09-18T14:32:17.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:26: run: export CEPH_MON=127.0.0.1:7111 2024-09-18T14:32:17.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:26: run: CEPH_MON=127.0.0.1:7111 2024-09-18T14:32:17.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:27: run: export CEPH_ARGS 2024-09-18T14:32:17.086 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:28: run: uuidgen 2024-09-18T14:32:17.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:28: run: CEPH_ARGS+='--fsid=3dccb8e1-1005-4c3c-a555-33db2e71c2be --auth-supported=none ' 2024-09-18T14:32:17.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:29: run: CEPH_ARGS+='--mon-host=127.0.0.1:7111 ' 2024-09-18T14:32:17.088 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: set 2024-09-18T14:32:17.088 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-18T14:32:17.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: local funcs=TEST_copy_from 2024-09-18T14:32:17.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:32: run: for func in $funcs 2024-09-18T14:32:17.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:33: run: setup td/osd-copy-from 2024-09-18T14:32:17.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-copy-from 2024-09-18T14:32:17.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-copy-from 2024-09-18T14:32:17.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2024-09-18T14:32:17.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:32:17.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2024-09-18T14:32:17.089 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:32:17.089 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:32:17.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:32:17.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:32:17.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:32:17.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:32:17.090 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:32:17.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:32:17.091 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:32:17.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:32:17.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:32:17.092 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:32:17.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:32:17.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:32:17.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:32:17.093 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:32:17.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:32:17.095 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:32:17.095 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2024-09-18T14:32:17.096 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:32:17.096 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:32:17.096 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.166178 2024-09-18T14:32:17.096 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.166178 2024-09-18T14:32:17.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:32:17.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:32:17.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-copy-from 2024-09-18T14:32:17.098 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T14:32:17.098 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:32:17.099 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.166178 2024-09-18T14:32:17.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.166178 2024-09-18T14:32:17.101 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T14:32:17.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T14:32:17.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T14:32:17.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T14:32:17.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-copy-from 1' TERM HUP INT 2024-09-18T14:32:17.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:34: run: TEST_copy_from td/osd-copy-from 2024-09-18T14:32:17.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:40: TEST_copy_from: local dir=td/osd-copy-from 2024-09-18T14:32:17.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:42: TEST_copy_from: run_mon td/osd-copy-from a 2024-09-18T14:32:17.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-copy-from 2024-09-18T14:32:17.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T14:32:17.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T14:32:17.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T14:32:17.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-copy-from/a 2024-09-18T14:32:17.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-copy-from/a --run-dir=td/osd-copy-from 2024-09-18T14:32:17.465 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T14:32:17.466 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:32:17.466 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:32:17.466 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:32:17.466 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:32:17.466 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.166178 2024-09-18T14:32:17.466 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.166178/$cluster-$name.asok' 2024-09-18T14:32:17.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-copy-from/a '--log-file=td/osd-copy-from/$name.log' '--admin-socket=/tmp/ceph-asok.166178/$cluster-$name.asok' --mon-cluster-log-file=td/osd-copy-from/log --run-dir=td/osd-copy-from '--pid-file=td/osd-copy-from/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T14:32:17.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T14:32:17.710 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T14:32:17.710 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:32:17.710 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:32:17.710 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T14:32:17.710 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T14:32:17.710 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:32:17.710 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:32:17.710 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:32:17.711 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:32:17.711 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:32:17.711 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.166178 2024-09-18T14:32:17.711 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.166178/ceph-mon.a.asok 2024-09-18T14:32:17.711 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:32:17.711 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.166178/ceph-mon.a.asok config get fsid 2024-09-18T14:32:17.792 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T14:32:17.792 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:32:17.792 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:32:17.793 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T14:32:17.793 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T14:32:17.793 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:32:17.793 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:32:17.793 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:32:17.793 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:32:17.793 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:32:17.793 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.166178 2024-09-18T14:32:17.793 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.166178/ceph-mon.a.asok 2024-09-18T14:32:17.793 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:32:17.793 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.166178/ceph-mon.a.asok config get mon_host 2024-09-18T14:32:17.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:43: TEST_copy_from: run_mgr td/osd-copy-from x 2024-09-18T14:32:17.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-copy-from 2024-09-18T14:32:17.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T14:32:17.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T14:32:17.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T14:32:17.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-copy-from/x 2024-09-18T14:32:17.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T14:32:18.722 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T14:32:18.722 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:32:18.723 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:32:18.723 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:32:18.723 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:32:18.723 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.166178 2024-09-18T14:32:18.723 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.166178/$cluster-$name.asok' 2024-09-18T14:32:18.723 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:32:18.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-copy-from/x '--log-file=td/osd-copy-from/$name.log' '--admin-socket=/tmp/ceph-asok.166178/$cluster-$name.asok' --run-dir=td/osd-copy-from '--pid-file=td/osd-copy-from/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:32:18.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:44: TEST_copy_from: run_osd td/osd-copy-from 0 2024-09-18T14:32:18.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-copy-from 2024-09-18T14:32:18.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:32:18.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T14:32:18.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:32:18.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-copy-from/0 2024-09-18T14:32:18.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3dccb8e1-1005-4c3c-a555-33db2e71c2be --auth-supported=none --mon-host=127.0.0.1:7111 ' 2024-09-18T14:32:18.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:32:18.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:32:18.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:32:18.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-copy-from/0' 2024-09-18T14:32:18.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-copy-from/0/journal' 2024-09-18T14:32:18.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:32:18.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:32:18.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-copy-from' 2024-09-18T14:32:18.749 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:32:18.749 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:32:18.749 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:32:18.749 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:32:18.750 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:32:18.750 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.166178 2024-09-18T14:32:18.750 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.166178/$cluster-$name.asok' 2024-09-18T14:32:18.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.166178/$cluster-$name.asok' 2024-09-18T14:32:18.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:32:18.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:32:18.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:32:18.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-copy-from/$name.log' 2024-09-18T14:32:18.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-copy-from/$name.pid' 2024-09-18T14:32:18.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:32:18.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:32:18.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:32:18.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:32:18.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:32:18.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:32:18.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-copy-from/0 2024-09-18T14:32:18.751 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:32:18.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6d9efd26-737b-424c-af8a-0af2a5456a91 2024-09-18T14:32:18.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6d9efd26-737b-424c-af8a-0af2a5456a91' 2024-09-18T14:32:18.752 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 6d9efd26-737b-424c-af8a-0af2a5456a91 2024-09-18T14:32:18.752 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:32:18.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBy5Opm4FdlLRAAjOlhNj3ztpCDuWXkMaQH+A== 2024-09-18T14:32:18.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBy5Opm4FdlLRAAjOlhNj3ztpCDuWXkMaQH+A=="}' 2024-09-18T14:32:18.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6d9efd26-737b-424c-af8a-0af2a5456a91 -i td/osd-copy-from/0/new.json 2024-09-18T14:32:19.064 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:32:19.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-copy-from/0/new.json 2024-09-18T14:32:19.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=3dccb8e1-1005-4c3c-a555-33db2e71c2be --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/0 --osd-journal=td/osd-copy-from/0/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.166178/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBy5Opm4FdlLRAAjOlhNj3ztpCDuWXkMaQH+A== --osd-uuid 6d9efd26-737b-424c-af8a-0af2a5456a91 2024-09-18T14:32:19.097 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:32:19.096+0000 7f18b5622440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:32:19.109 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:32:19.108+0000 7f18b5622440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:32:19.110 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:32:19.109+0000 7f18b5622440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:32:19.110 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:32:19.109+0000 7f18b5622440 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2024-09-18T14:32:19.110 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:32:19.109+0000 7f18b5622440 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2024-09-18T14:32:19.110 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:32:19.109+0000 7f18b5622440 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2024-09-18T14:32:19.149 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:32:19.148+0000 7f18b5622440 -1 bluestore(td/osd-copy-from/0) _read_fsid unparsable uuid 2024-09-18T14:32:24.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-copy-from/0/keyring 2024-09-18T14:32:24.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:32:24.531 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T14:32:24.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T14:32:24.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-copy-from/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:32:25.055 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T14:32:25.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T14:32:25.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=3dccb8e1-1005-4c3c-a555-33db2e71c2be --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/0 --osd-journal=td/osd-copy-from/0/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.166178/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:32:25.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:32:25.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:32:25.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:32:25.077 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:32:25.076+0000 7fca8d723440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:32:25.097 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:32:25.096+0000 7fca8d723440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:32:25.098 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:32:25.097+0000 7fca8d723440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:32:25.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T14:32:25.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:32:25.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:32:25.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:32:25.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:32:25.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:25.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:32:25.494 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:32:25.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:25.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:32:25.638 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:32:25.637+0000 7fca8d723440 -1 Falling back to public interface 2024-09-18T14:32:25.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:26.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:26.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:26.904 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:32:26.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:32:26.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:26.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:32:27.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:28.184 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:32:28.183+0000 7fca8d723440 -1 osd.0 0 log_to_monitors true 2024-09-18T14:32:28.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:28.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:28.322 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:32:28.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:32:28.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:28.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:32:28.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:29.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:29.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:29.738 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:32:29.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:32:29.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:29.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:32:30.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:31.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:31.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:31.139 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:32:31.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:32:31.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:31.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:32:31.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:32.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:32.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:32.536 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:32:32.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:32:32.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:32.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:32:32.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:33.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:33.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:33.985 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:32:33.985 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:32:33.985 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:33.985 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:32:34.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:35.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:35.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:35.367 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:32:35.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:32:35.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:35.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:32:35.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:36.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:36.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:36.856 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:32:36.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:32:36.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:36.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:32:37.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:38.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:38.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:38.251 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:32:38.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:32:38.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:38.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:32:38.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:39.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:39.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:39.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:32:39.634 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:32:39.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:39.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:32:40.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:41.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:41.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:41.511 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:32:41.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:32:41.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:41.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:32:42.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:43.052 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:43.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:43.053 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:32:43.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:32:43.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:43.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:32:43.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:44.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:44.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:44.726 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:32:44.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:32:44.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:44.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:32:45.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:46.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:46.612 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:46.612 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:32:46.612 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:32:46.612 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:46.612 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:32:47.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:48.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:48.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:48.070 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:32:48.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:32:48.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:48.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:32:48.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:49.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:49.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:49.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:32:49.519 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:32:49.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:49.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:32:50.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:32:51.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:32:51.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:51.144 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T14:32:51.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:32:51.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:51.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:32:51.522 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/167269,v1:127.0.0.1:6803/167269] [v2:127.0.0.1:6804/167269,v1:127.0.0.1:6805/167269] exists,up 6d9efd26-737b-424c-af8a-0af2a5456a91 2024-09-18T14:32:51.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:32:51.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:32:51.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:32:51.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:45: TEST_copy_from: run_osd td/osd-copy-from 1 2024-09-18T14:32:51.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-copy-from 2024-09-18T14:32:51.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:32:51.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T14:32:51.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:32:51.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-copy-from/1 2024-09-18T14:32:51.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3dccb8e1-1005-4c3c-a555-33db2e71c2be --auth-supported=none --mon-host=127.0.0.1:7111 ' 2024-09-18T14:32:51.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:32:51.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:32:51.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:32:51.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-copy-from/1' 2024-09-18T14:32:51.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-copy-from/1/journal' 2024-09-18T14:32:51.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:32:51.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:32:51.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-copy-from' 2024-09-18T14:32:51.530 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:32:51.530 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:32:51.530 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:32:51.530 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:32:51.530 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:32:51.530 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.166178 2024-09-18T14:32:51.530 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.166178/$cluster-$name.asok' 2024-09-18T14:32:51.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.166178/$cluster-$name.asok' 2024-09-18T14:32:51.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:32:51.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:32:51.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:32:51.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-copy-from/$name.log' 2024-09-18T14:32:51.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-copy-from/$name.pid' 2024-09-18T14:32:51.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:32:51.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:32:51.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:32:51.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:32:51.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:32:51.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:32:51.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-copy-from/1 2024-09-18T14:32:51.531 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:32:51.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0b5c44f3-3a3a-4e47-96e0-65179fc07367 2024-09-18T14:32:51.532 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 0b5c44f3-3a3a-4e47-96e0-65179fc07367 2024-09-18T14:32:51.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 0b5c44f3-3a3a-4e47-96e0-65179fc07367' 2024-09-18T14:32:51.532 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:32:51.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCT5Opm2AcRIBAAk5hpqg7dkfzodeH1LAGDIQ== 2024-09-18T14:32:51.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCT5Opm2AcRIBAAk5hpqg7dkfzodeH1LAGDIQ=="}' 2024-09-18T14:32:51.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0b5c44f3-3a3a-4e47-96e0-65179fc07367 -i td/osd-copy-from/1/new.json 2024-09-18T14:32:52.063 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:32:52.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-copy-from/1/new.json 2024-09-18T14:32:52.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=3dccb8e1-1005-4c3c-a555-33db2e71c2be --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/1 --osd-journal=td/osd-copy-from/1/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.166178/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCT5Opm2AcRIBAAk5hpqg7dkfzodeH1LAGDIQ== --osd-uuid 0b5c44f3-3a3a-4e47-96e0-65179fc07367 2024-09-18T14:32:52.098 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:32:52.097+0000 7f8d4a31d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:32:52.100 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:32:52.099+0000 7f8d4a31d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:32:52.101 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:32:52.100+0000 7f8d4a31d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:32:52.101 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:32:52.100+0000 7f8d4a31d440 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2024-09-18T14:32:52.102 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:32:52.100+0000 7f8d4a31d440 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2024-09-18T14:32:52.102 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:32:52.100+0000 7f8d4a31d440 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2024-09-18T14:32:52.178 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:32:52.178+0000 7f8d4a31d440 -1 bluestore(td/osd-copy-from/1) _read_fsid unparsable uuid 2024-09-18T14:32:58.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-copy-from/1/keyring 2024-09-18T14:32:58.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:32:58.050 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T14:32:58.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T14:32:58.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-copy-from/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:32:58.559 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T14:32:58.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T14:32:58.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=3dccb8e1-1005-4c3c-a555-33db2e71c2be --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/1 --osd-journal=td/osd-copy-from/1/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.166178/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:32:58.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:32:58.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:32:58.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:32:58.580 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:32:58.579+0000 7f4cd6408440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:32:58.592 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:32:58.591+0000 7f4cd6408440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:32:58.593 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:32:58.593+0000 7f4cd6408440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:32:58.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T14:32:58.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:32:58.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:32:58.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:32:58.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:32:58.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:32:58.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:32:58.954 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:32:58.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:32:58.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:32:59.115 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:32:59.114+0000 7f4cd6408440 -1 Falling back to public interface 2024-09-18T14:32:59.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:00.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:00.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:00.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:33:00.338 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:33:00.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:00.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:33:00.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:01.608 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:01.607+0000 7f4cd6408440 -1 osd.1 0 log_to_monitors true 2024-09-18T14:33:01.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:01.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:01.718 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:33:01.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:33:01.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:01.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:33:02.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:03.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:03.285 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:03.285 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:33:03.285 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:33:03.285 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:03.285 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:33:03.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:04.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:04.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:04.706 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:33:04.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:33:04.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:04.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:33:05.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:06.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:06.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:06.159 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:33:06.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:33:06.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:06.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:33:06.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:07.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:07.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:07.752 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:33:07.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:33:07.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:07.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:33:08.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:09.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:09.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:09.130 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:33:09.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:33:09.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:09.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:33:09.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:10.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:10.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:10.509 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:33:10.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:33:10.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:10.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:33:10.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:11.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:11.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:11.899 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:33:11.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:33:11.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:11.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:33:12.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:13.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:13.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:13.352 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:33:13.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:33:13.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:13.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:33:13.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:14.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:14.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:14.725 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:33:14.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:33:14.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:14.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:33:15.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:16.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:16.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:16.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:33:16.795 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:33:16.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:16.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:33:17.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:18.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:18.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:18.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:33:18.574 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:33:18.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:18.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:33:19.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:20.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:20.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:20.336 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:33:20.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:33:20.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:20.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:33:20.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:21.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:21.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:21.919 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:33:21.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:33:21.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:21.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:33:22.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:23.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:23.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:23.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:33:23.631 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:33:23.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:23.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:33:24.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:25.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:25.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:25.133 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T14:33:25.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:33:25.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:25.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:33:25.545 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/169023,v1:127.0.0.1:6811/169023] [v2:127.0.0.1:6812/169023,v1:127.0.0.1:6813/169023] exists,up 0b5c44f3-3a3a-4e47-96e0-65179fc07367 2024-09-18T14:33:25.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:33:25.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:33:25.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:33:25.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:46: TEST_copy_from: create_rbd_pool 2024-09-18T14:33:25.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T14:33:25.944 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'rbd' does not exist 2024-09-18T14:33:25.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T14:33:25.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T14:33:26.446 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'rbd' already exists 2024-09-18T14:33:26.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T14:33:27.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T14:33:29.147 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:49: TEST_copy_from: which rados 2024-09-18T14:33:29.147 INFO:tasks.workunit.client.0.smithi047.stderr:eenvironment:0: which: alias 2024-09-18T14:33:29.147 INFO:tasks.workunit.client.0.smithi047.stderr:eenvironment:1: which: eval declare -f 2024-09-18T14:33:29.147 INFO:tasks.workunit.client.0.smithi047.stderr:eeenvironment:1: which: declare -f 2024-09-18T14:33:29.147 INFO:tasks.workunit.client.0.smithi047.stderr:eenvironment:1: which: /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot rados 2024-09-18T14:33:29.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:49: TEST_copy_from: rados -p rbd put foo /usr/bin/rados 2024-09-18T14:33:29.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:50: TEST_copy_from: rados -p rbd cp foo foo2 2024-09-18T14:33:29.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:51: TEST_copy_from: rados -p rbd stat foo2 2024-09-18T14:33:29.573 INFO:tasks.workunit.client.0.smithi047.stdout:rbd/foo2 mtime 2024-09-18T14:33:29.000000+0000, size 595872 2024-09-18T14:33:29.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:54: TEST_copy_from: ceph tell 'osd.*' injectargs -- --osd-debug-inject-copyfrom-error 2024-09-18T14:33:29.683 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: {} 2024-09-18T14:33:29.709 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: {} 2024-09-18T14:33:29.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:55: TEST_copy_from: rados -p rbd cp foo foo3 2024-09-18T14:33:29.755 INFO:tasks.workunit.client.0.smithi047.stderr:error copying rbd/foo => rbd/foo3: (5) Input/output error 2024-09-18T14:33:29.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:56: TEST_copy_from: rados -p rbd stat foo3 2024-09-18T14:33:29.810 INFO:tasks.workunit.client.0.smithi047.stderr: error stat-ing rbd/foo3: (2) No such file or directory 2024-09-18T14:33:29.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:59: TEST_copy_from: ceph tell 'osd.*' injectargs -- --no-osd-debug-inject-copyfrom-error 2024-09-18T14:33:29.911 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: {} 2024-09-18T14:33:29.936 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: {} 2024-09-18T14:33:29.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:60: TEST_copy_from: rados -p rbd cp foo foo3 2024-09-18T14:33:30.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:61: TEST_copy_from: rados -p rbd stat foo3 2024-09-18T14:33:30.171 INFO:tasks.workunit.client.0.smithi047.stdout:rbd/foo3 mtime 2024-09-18T14:33:29.000000+0000, size 595872 2024-09-18T14:33:30.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:35: run: teardown td/osd-copy-from 2024-09-18T14:33:30.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2024-09-18T14:33:30.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:33:30.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2024-09-18T14:33:30.173 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:33:30.174 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:33:30.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:33:30.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:33:30.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:33:30.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:33:30.281 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:33:30.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:33:30.282 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:33:30.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:33:30.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:33:30.283 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:33:30.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:33:30.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:33:30.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:33:30.284 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:33:30.285 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:33:30.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:33:30.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2024-09-18T14:33:30.296 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:33:30.296 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:33:30.296 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.166178 2024-09-18T14:33:30.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.166178 2024-09-18T14:33:30.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:33:30.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:33:30.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T14:33:30.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/osd-copy-from 0 2024-09-18T14:33:30.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2024-09-18T14:33:30.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T14:33:30.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2024-09-18T14:33:30.298 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:33:30.298 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:33:30.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:33:30.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:33:30.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:33:30.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:33:30.299 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:33:30.299 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:33:30.300 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:33:30.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:33:30.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:33:30.301 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:33:30.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:33:30.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:33:30.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:33:30.302 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:33:30.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:33:30.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T14:33:30.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2024-09-18T14:33:30.305 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:33:30.305 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:33:30.305 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.166178 2024-09-18T14:33:30.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.166178 2024-09-18T14:33:30.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:33:30.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:33:30.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T14:33:30.307 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T14:33:30.308 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T14:33:30.372 INFO:tasks.workunit:Running workunit osd/osd-dup.sh... 2024-09-18T14:33:30.372 DEBUG:teuthology.orchestra.run.smithi047:workunit test osd/osd-dup.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh 2024-09-18T14:33:30.435 INFO:tasks.workunit.client.0.smithi047.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T14:33:30.442 INFO:tasks.workunit.client.0.smithi047.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T14:33:30.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T14:33:30.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T14:33:30.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T14:33:30.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T14:33:30.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T14:33:30.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T14:33:30.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T14:33:30.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T14:33:30.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/osd-dup 2024-09-18T14:33:30.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:8: run: local dir=td/osd-dup 2024-09-18T14:33:30.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:9: run: shift 2024-09-18T14:33:30.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:11: run: export CEPH_MON=127.0.0.1:7146 2024-09-18T14:33:30.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:11: run: CEPH_MON=127.0.0.1:7146 2024-09-18T14:33:30.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:12: run: export CEPH_ARGS 2024-09-18T14:33:30.443 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:13: run: uuidgen 2024-09-18T14:33:30.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:13: run: CEPH_ARGS+='--fsid=900aae69-8049-40e4-a548-ae813fbb3731 --auth-supported=none ' 2024-09-18T14:33:30.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-18T14:33:30.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:16: run: CEPH_ARGS+='--filestore_wbthrottle_xfs_ios_hard_limit=900 ' 2024-09-18T14:33:30.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:17: run: CEPH_ARGS+='--filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2024-09-18T14:33:30.443 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: set 2024-09-18T14:33:30.443 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-18T14:33:30.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: local funcs=TEST_filestore_to_bluestore 2024-09-18T14:33:30.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:19: run: for func in $funcs 2024-09-18T14:33:30.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:20: run: setup td/osd-dup 2024-09-18T14:33:30.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-dup 2024-09-18T14:33:30.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-dup 2024-09-18T14:33:30.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-dup 2024-09-18T14:33:30.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:33:30.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-dup KILL 2024-09-18T14:33:30.445 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:33:30.445 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:33:30.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:33:30.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:33:30.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:33:30.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:33:30.446 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:33:30.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:33:30.447 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:33:30.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:33:30.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:33:30.448 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:33:30.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:33:30.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:33:30.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:33:30.449 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:33:30.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:33:30.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:33:30.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-dup 2024-09-18T14:33:30.452 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:33:30.452 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:33:30.452 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.170279 2024-09-18T14:33:30.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.170279 2024-09-18T14:33:30.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:33:30.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:33:30.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-dup 2024-09-18T14:33:30.454 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T14:33:30.454 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:33:30.454 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.170279 2024-09-18T14:33:30.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.170279 2024-09-18T14:33:30.457 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T14:33:30.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T14:33:30.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T14:33:30.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T14:33:30.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-dup 1' TERM HUP INT 2024-09-18T14:33:30.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:21: run: TEST_filestore_to_bluestore td/osd-dup 2024-09-18T14:33:30.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:27: TEST_filestore_to_bluestore: local dir=td/osd-dup 2024-09-18T14:33:30.457 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:29: TEST_filestore_to_bluestore: ulimit -n 2024-09-18T14:33:30.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:29: TEST_filestore_to_bluestore: local flimit=4096 2024-09-18T14:33:30.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:30: TEST_filestore_to_bluestore: '[' 4096 -lt 1536 ']' 2024-09-18T14:33:30.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:34: TEST_filestore_to_bluestore: run_mon td/osd-dup a 2024-09-18T14:33:30.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-dup 2024-09-18T14:33:30.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T14:33:30.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T14:33:30.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T14:33:30.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-dup/a 2024-09-18T14:33:30.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-dup/a --run-dir=td/osd-dup 2024-09-18T14:33:30.814 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T14:33:30.814 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:33:30.815 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:33:30.815 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:33:30.815 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:33:30.815 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.170279 2024-09-18T14:33:30.815 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.170279/$cluster-$name.asok' 2024-09-18T14:33:30.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-dup/a '--log-file=td/osd-dup/$name.log' '--admin-socket=/tmp/ceph-asok.170279/$cluster-$name.asok' --mon-cluster-log-file=td/osd-dup/log --run-dir=td/osd-dup '--pid-file=td/osd-dup/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T14:33:31.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T14:33:31.058 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T14:33:31.058 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:33:31.058 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:33:31.058 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T14:33:31.058 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T14:33:31.058 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:33:31.058 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:33:31.058 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:33:31.059 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:33:31.059 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:33:31.059 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.170279 2024-09-18T14:33:31.059 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.170279/ceph-mon.a.asok 2024-09-18T14:33:31.059 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:33:31.059 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.170279/ceph-mon.a.asok config get fsid 2024-09-18T14:33:31.138 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T14:33:31.139 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:33:31.139 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:33:31.139 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T14:33:31.139 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T14:33:31.139 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:33:31.139 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:33:31.139 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:33:31.139 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:33:31.139 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:33:31.139 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.170279 2024-09-18T14:33:31.140 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.170279/ceph-mon.a.asok 2024-09-18T14:33:31.140 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:33:31.140 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.170279/ceph-mon.a.asok config get mon_host 2024-09-18T14:33:31.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:35: TEST_filestore_to_bluestore: run_mgr td/osd-dup x 2024-09-18T14:33:31.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-dup 2024-09-18T14:33:31.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T14:33:31.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T14:33:31.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T14:33:31.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-dup/x 2024-09-18T14:33:31.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T14:33:32.061 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T14:33:32.061 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:33:32.061 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:33:32.061 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:33:32.062 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:33:32.062 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.170279 2024-09-18T14:33:32.062 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.170279/$cluster-$name.asok' 2024-09-18T14:33:32.062 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:33:32.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-dup/x '--log-file=td/osd-dup/$name.log' '--admin-socket=/tmp/ceph-asok.170279/$cluster-$name.asok' --run-dir=td/osd-dup '--pid-file=td/osd-dup/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:33:32.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:36: TEST_filestore_to_bluestore: run_osd_filestore td/osd-dup 0 2024-09-18T14:33:32.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-dup 2024-09-18T14:33:32.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-18T14:33:32.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=0 2024-09-18T14:33:32.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-18T14:33:32.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-dup/0 2024-09-18T14:33:32.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=900aae69-8049-40e4-a548-ae813fbb3731 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2024-09-18T14:33:32.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:33:32.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:33:32.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:33:32.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-dup/0' 2024-09-18T14:33:32.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-dup/0/journal' 2024-09-18T14:33:32.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-18T14:33:32.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-18T14:33:32.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-dup' 2024-09-18T14:33:32.090 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-18T14:33:32.090 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:33:32.090 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:33:32.090 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:33:32.090 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:33:32.090 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.170279 2024-09-18T14:33:32.091 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.170279/$cluster-$name.asok' 2024-09-18T14:33:32.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.170279/$cluster-$name.asok' 2024-09-18T14:33:32.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-18T14:33:32.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-18T14:33:32.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-18T14:33:32.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-dup/$name.log' 2024-09-18T14:33:32.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-dup/$name.pid' 2024-09-18T14:33:32.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:33:32.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:33:32.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:33:32.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-18T14:33:32.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-18T14:33:32.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-dup/0 2024-09-18T14:33:32.091 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-18T14:33:32.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=d5ff9deb-56b7-4855-a4be-d8096bdafafa 2024-09-18T14:33:32.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd d5ff9deb-56b7-4855-a4be-d8096bdafafa' 2024-09-18T14:33:32.092 INFO:tasks.workunit.client.0.smithi047.stdout:add osd d5ff9deb-56b7-4855-a4be-d8096bdafafa 2024-09-18T14:33:32.093 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-18T14:33:32.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQC85Opm0X4tBhAAjwFsqHg8BZryFRPR7mv1ww== 2024-09-18T14:33:32.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQC85Opm0X4tBhAAjwFsqHg8BZryFRPR7mv1ww=="}' 2024-09-18T14:33:32.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new d5ff9deb-56b7-4855-a4be-d8096bdafafa -i td/osd-dup/0/new.json 2024-09-18T14:33:32.405 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:33:32.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-dup/0/new.json 2024-09-18T14:33:32.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 0 --fsid=900aae69-8049-40e4-a548-ae813fbb3731 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/0 --osd-journal=td/osd-dup/0/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.170279/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC85Opm0X4tBhAAjwFsqHg8BZryFRPR7mv1ww== --osd-uuid d5ff9deb-56b7-4855-a4be-d8096bdafafa --osd-objectstore=filestore 2024-09-18T14:33:32.439 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:32.438+0000 7fd80460e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:33:32.449 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:32.448+0000 7fd80460e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:33:32.450 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:32.449+0000 7fd80460e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:33:32.876 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:32.875+0000 7fd80460e440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T14:33:33.347 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:33.345+0000 7fd80460e440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T14:33:33.347 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:33.346+0000 7fd80460e440 -1 journal do_read_entry(4096): bad header magic 2024-09-18T14:33:33.347 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:33.346+0000 7fd80460e440 -1 journal do_read_entry(4096): bad header magic 2024-09-18T14:33:33.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-dup/0/keyring 2024-09-18T14:33:33.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-18T14:33:33.929 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T14:33:33.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd0 key to auth repository 2024-09-18T14:33:33.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-dup/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:33:34.075 INFO:tasks.workunit.client.0.smithi047.stderr:Error EINVAL: key for osd.0 not found in provided keyring 2024-09-18T14:33:34.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.0 2024-09-18T14:33:34.077 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T14:33:34.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 0 --fsid=900aae69-8049-40e4-a548-ae813fbb3731 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/0 --osd-journal=td/osd-dup/0/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.170279/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:33:34.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-18T14:33:34.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-18T14:33:34.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-18T14:33:34.100 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:34.099+0000 7f107c608440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:33:34.102 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:34.101+0000 7f107c608440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:33:34.104 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:34.103+0000 7f107c608440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:33:34.105 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:34.104+0000 7f107c608440 -1 Falling back to public interface 2024-09-18T14:33:34.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 0 2024-09-18T14:33:34.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:33:34.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:33:34.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:33:34.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:33:34.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:34.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:33:34.346 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:33:34.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:34.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:33:34.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:34.588 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:34.587+0000 7f107c608440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T14:33:34.594 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:34.594+0000 7f107c608440 -1 journal do_read_entry(8192): bad header magic 2024-09-18T14:33:34.595 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:34.594+0000 7f107c608440 -1 journal do_read_entry(8192): bad header magic 2024-09-18T14:33:34.599 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:34.598+0000 7f107c608440 -1 osd.0 0 log_to_monitors true 2024-09-18T14:33:35.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:35.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:35.573 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:33:35.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:33:35.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:35.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:33:35.813 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/170581,v1:127.0.0.1:6801/170581] [v2:127.0.0.1:6802/170581,v1:127.0.0.1:6803/170581] exists,up d5ff9deb-56b7-4855-a4be-d8096bdafafa 2024-09-18T14:33:35.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:33:35.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:33:35.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:33:35.814 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:37: TEST_filestore_to_bluestore: cat td/osd-dup/osd.0.pid 2024-09-18T14:33:35.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:37: TEST_filestore_to_bluestore: osd_pid=170581 2024-09-18T14:33:35.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:38: TEST_filestore_to_bluestore: run_osd_filestore td/osd-dup 1 2024-09-18T14:33:35.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-dup 2024-09-18T14:33:35.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-18T14:33:35.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=1 2024-09-18T14:33:35.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-18T14:33:35.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-dup/1 2024-09-18T14:33:35.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=900aae69-8049-40e4-a548-ae813fbb3731 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2024-09-18T14:33:35.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:33:35.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:33:35.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:33:35.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-dup/1' 2024-09-18T14:33:35.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-dup/1/journal' 2024-09-18T14:33:35.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-18T14:33:35.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-18T14:33:35.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-dup' 2024-09-18T14:33:35.818 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-18T14:33:35.818 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:33:35.818 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:33:35.818 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:33:35.819 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:33:35.819 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.170279 2024-09-18T14:33:35.819 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.170279/$cluster-$name.asok' 2024-09-18T14:33:35.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.170279/$cluster-$name.asok' 2024-09-18T14:33:35.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-18T14:33:35.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-18T14:33:35.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-18T14:33:35.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-dup/$name.log' 2024-09-18T14:33:35.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-dup/$name.pid' 2024-09-18T14:33:35.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:33:35.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:33:35.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:33:35.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-18T14:33:35.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-18T14:33:35.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-dup/1 2024-09-18T14:33:35.820 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-18T14:33:35.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=1223c4be-0cbd-42c4-b881-c47ca5e34a5c 2024-09-18T14:33:35.820 INFO:tasks.workunit.client.0.smithi047.stdout:add osd 1223c4be-0cbd-42c4-b881-c47ca5e34a5c 2024-09-18T14:33:35.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd 1223c4be-0cbd-42c4-b881-c47ca5e34a5c' 2024-09-18T14:33:35.821 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-18T14:33:35.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQC/5OpmcR+sMRAASBh0WYGCV4brcsGg2QQ8Lg== 2024-09-18T14:33:35.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQC/5OpmcR+sMRAASBh0WYGCV4brcsGg2QQ8Lg=="}' 2024-09-18T14:33:35.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new 1223c4be-0cbd-42c4-b881-c47ca5e34a5c -i td/osd-dup/1/new.json 2024-09-18T14:33:36.205 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:33:36.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-dup/1/new.json 2024-09-18T14:33:36.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 1 --fsid=900aae69-8049-40e4-a548-ae813fbb3731 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/1 --osd-journal=td/osd-dup/1/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.170279/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC/5OpmcR+sMRAASBh0WYGCV4brcsGg2QQ8Lg== --osd-uuid 1223c4be-0cbd-42c4-b881-c47ca5e34a5c --osd-objectstore=filestore 2024-09-18T14:33:36.239 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:36.238+0000 7f1da4d1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:33:36.241 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:36.240+0000 7f1da4d1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:33:36.242 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:36.241+0000 7f1da4d1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:33:36.708 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:36.707+0000 7f1da4d1f440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T14:33:37.153 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:37.152+0000 7f1da4d1f440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T14:33:37.153 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:37.152+0000 7f1da4d1f440 -1 journal do_read_entry(4096): bad header magic 2024-09-18T14:33:37.153 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:37.152+0000 7f1da4d1f440 -1 journal do_read_entry(4096): bad header magic 2024-09-18T14:33:37.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-dup/1/keyring 2024-09-18T14:33:37.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-18T14:33:37.771 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T14:33:37.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd1 key to auth repository 2024-09-18T14:33:37.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-dup/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:33:38.107 INFO:tasks.workunit.client.0.smithi047.stderr:Error EINVAL: key for osd.1 not found in provided keyring 2024-09-18T14:33:38.111 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T14:33:38.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.1 2024-09-18T14:33:38.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 1 --fsid=900aae69-8049-40e4-a548-ae813fbb3731 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/1 --osd-journal=td/osd-dup/1/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.170279/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:33:38.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-18T14:33:38.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-18T14:33:38.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-18T14:33:38.134 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:38.133+0000 7fdc47b7a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:33:38.149 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:38.148+0000 7fdc47b7a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:33:38.150 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:38.149+0000 7fdc47b7a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:33:38.151 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:38.151+0000 7fdc47b7a440 -1 Falling back to public interface 2024-09-18T14:33:38.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 1 2024-09-18T14:33:38.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:33:38.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:33:38.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:33:38.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:33:38.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:38.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:33:38.479 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:33:38.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:38.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:33:38.648 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:38.647+0000 7fdc47b7a440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T14:33:38.660 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:38.660+0000 7fdc47b7a440 -1 journal do_read_entry(8192): bad header magic 2024-09-18T14:33:38.660 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:38.660+0000 7fdc47b7a440 -1 journal do_read_entry(8192): bad header magic 2024-09-18T14:33:38.665 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:38.664+0000 7fdc47b7a440 -1 osd.1 0 log_to_monitors true 2024-09-18T14:33:38.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:39.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:39.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:39.902 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:33:39.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:33:39.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:39.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:33:40.315 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/170943,v1:127.0.0.1:6811/170943] [v2:127.0.0.1:6812/170943,v1:127.0.0.1:6813/170943] exists,up 1223c4be-0cbd-42c4-b881-c47ca5e34a5c 2024-09-18T14:33:40.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:33:40.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:33:40.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:33:40.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:39: TEST_filestore_to_bluestore: run_osd_filestore td/osd-dup 2 2024-09-18T14:33:40.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-dup 2024-09-18T14:33:40.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-18T14:33:40.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=2 2024-09-18T14:33:40.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-18T14:33:40.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-dup/2 2024-09-18T14:33:40.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=900aae69-8049-40e4-a548-ae813fbb3731 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2024-09-18T14:33:40.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:33:40.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:33:40.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:33:40.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-dup/2' 2024-09-18T14:33:40.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-dup/2/journal' 2024-09-18T14:33:40.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-18T14:33:40.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-18T14:33:40.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-dup' 2024-09-18T14:33:40.320 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-18T14:33:40.320 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:33:40.320 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:33:40.321 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:33:40.321 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:33:40.321 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.170279 2024-09-18T14:33:40.321 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.170279/$cluster-$name.asok' 2024-09-18T14:33:40.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.170279/$cluster-$name.asok' 2024-09-18T14:33:40.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-18T14:33:40.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-18T14:33:40.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-18T14:33:40.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-dup/$name.log' 2024-09-18T14:33:40.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-dup/$name.pid' 2024-09-18T14:33:40.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:33:40.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:33:40.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:33:40.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-18T14:33:40.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-18T14:33:40.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-dup/2 2024-09-18T14:33:40.322 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-18T14:33:40.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=004f0f62-1453-4542-ac0a-af0d8adf7318 2024-09-18T14:33:40.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd 004f0f62-1453-4542-ac0a-af0d8adf7318' 2024-09-18T14:33:40.323 INFO:tasks.workunit.client.0.smithi047.stdout:add osd 004f0f62-1453-4542-ac0a-af0d8adf7318 2024-09-18T14:33:40.323 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-18T14:33:40.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQDE5OpmEdnfExAASKYW7Pj0gI84TfHt48aTTQ== 2024-09-18T14:33:40.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQDE5OpmEdnfExAASKYW7Pj0gI84TfHt48aTTQ=="}' 2024-09-18T14:33:40.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new 004f0f62-1453-4542-ac0a-af0d8adf7318 -i td/osd-dup/2/new.json 2024-09-18T14:33:40.789 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:33:40.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-dup/2/new.json 2024-09-18T14:33:40.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 2 --fsid=900aae69-8049-40e4-a548-ae813fbb3731 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/2 --osd-journal=td/osd-dup/2/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.170279/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDE5OpmEdnfExAASKYW7Pj0gI84TfHt48aTTQ== --osd-uuid 004f0f62-1453-4542-ac0a-af0d8adf7318 --osd-objectstore=filestore 2024-09-18T14:33:40.824 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:40.823+0000 7f952e72b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:33:40.826 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:40.825+0000 7f952e72b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:33:40.827 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:40.826+0000 7f952e72b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:33:41.359 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:41.359+0000 7f952e72b440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T14:33:41.898 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:41.897+0000 7f952e72b440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T14:33:41.905 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:41.905+0000 7f952e72b440 -1 journal do_read_entry(4096): bad header magic 2024-09-18T14:33:41.905 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:41.905+0000 7f952e72b440 -1 journal do_read_entry(4096): bad header magic 2024-09-18T14:33:42.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-dup/2/keyring 2024-09-18T14:33:42.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-18T14:33:42.685 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd2 key to auth repository 2024-09-18T14:33:42.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd2 key to auth repository 2024-09-18T14:33:42.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-dup/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:33:43.014 INFO:tasks.workunit.client.0.smithi047.stderr:Error EINVAL: key for osd.2 not found in provided keyring 2024-09-18T14:33:43.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.2 2024-09-18T14:33:43.017 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T14:33:43.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 2 --fsid=900aae69-8049-40e4-a548-ae813fbb3731 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/2 --osd-journal=td/osd-dup/2/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.170279/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:33:43.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-18T14:33:43.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-18T14:33:43.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-18T14:33:43.039 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:43.039+0000 7f61a260e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:33:43.041 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:43.041+0000 7f61a260e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:33:43.043 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:43.042+0000 7f61a260e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:33:43.044 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:43.043+0000 7f61a260e440 -1 Falling back to public interface 2024-09-18T14:33:43.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 2 2024-09-18T14:33:43.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:33:43.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T14:33:43.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:33:43.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:33:43.444 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:33:43.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:43.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:33:43.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:43.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:33:43.539 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:43.538+0000 7f61a260e440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T14:33:43.547 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:43.546+0000 7f61a260e440 -1 journal do_read_entry(8192): bad header magic 2024-09-18T14:33:43.547 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:43.546+0000 7f61a260e440 -1 journal do_read_entry(8192): bad header magic 2024-09-18T14:33:43.552 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:33:43.551+0000 7f61a260e440 -1 osd.2 0 log_to_monitors true 2024-09-18T14:33:43.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:33:44.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:33:44.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:33:44.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:33:44.851 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:33:44.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:33:44.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:33:45.269 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/171225,v1:127.0.0.1:6819/171225] [v2:127.0.0.1:6820/171225,v1:127.0.0.1:6821/171225] exists,up 004f0f62-1453-4542-ac0a-af0d8adf7318 2024-09-18T14:33:45.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:33:45.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:33:45.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:33:45.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:41: TEST_filestore_to_bluestore: sleep 5 2024-09-18T14:33:50.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:43: TEST_filestore_to_bluestore: create_pool foo 16 2024-09-18T14:33:50.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-09-18T14:33:50.770 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'foo' already exists 2024-09-18T14:33:50.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T14:33:51.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:46: TEST_filestore_to_bluestore: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2024-09-18T14:33:51.815 INFO:tasks.workunit.client.0.smithi047.stdout:hints = 1 2024-09-18T14:33:51.815 INFO:tasks.workunit.client.0.smithi047.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2024-09-18T14:33:51.815 INFO:tasks.workunit.client.0.smithi047.stdout:Object prefix: benchmark_data_smithi047_171405 2024-09-18T14:33:51.815 INFO:tasks.workunit.client.0.smithi047.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T14:33:51.815 INFO:tasks.workunit.client.0.smithi047.stdout: 0 0 0 0 0 0 - 0 2024-09-18T14:33:52.815 INFO:tasks.workunit.client.0.smithi047.stdout: 1 16 16 0 0 0 - 0 2024-09-18T14:33:53.815 INFO:tasks.workunit.client.0.smithi047.stdout: 2 16 16 0 0 0 - 0 2024-09-18T14:33:54.815 INFO:tasks.workunit.client.0.smithi047.stdout: 3 16 17 1 0.00130193 0.00130208 2.74096 2.74096 2024-09-18T14:33:55.815 INFO:tasks.workunit.client.0.smithi047.stdout: 4 16 17 1 0.00097645 0 - 2.74096 2024-09-18T14:33:56.815 INFO:tasks.workunit.client.0.smithi047.stdout: 5 16 17 1 0.000781156 0 - 2.74096 2024-09-18T14:33:57.815 INFO:tasks.workunit.client.0.smithi047.stdout: 6 16 66 50 0.032548 0.0638021 0.113614 1.90539 2024-09-18T14:33:58.815 INFO:tasks.workunit.client.0.smithi047.stdout: 7 16 195 179 0.0998761 0.503906 0.112152 0.618368 2024-09-18T14:33:59.816 INFO:tasks.workunit.client.0.smithi047.stdout: 8 16 291 275 0.13426 0.375 0.423826 0.443686 2024-09-18T14:34:00.816 INFO:tasks.workunit.client.0.smithi047.stdout: 9 16 371 355 0.154061 0.3125 0.13701 0.401672 2024-09-18T14:34:01.816 INFO:tasks.workunit.client.0.smithi047.stdout: 10 15 465 450 0.17576 0.371094 0.13768 0.355353 2024-09-18T14:34:02.816 INFO:tasks.workunit.client.0.smithi047.stdout:Total time run: 10.1114 2024-09-18T14:34:02.816 INFO:tasks.workunit.client.0.smithi047.stdout:Total writes made: 465 2024-09-18T14:34:02.816 INFO:tasks.workunit.client.0.smithi047.stdout:Write size: 4096 2024-09-18T14:34:02.816 INFO:tasks.workunit.client.0.smithi047.stdout:Object size: 4096 2024-09-18T14:34:02.816 INFO:tasks.workunit.client.0.smithi047.stdout:Bandwidth (MB/sec): 0.17964 2024-09-18T14:34:02.817 INFO:tasks.workunit.client.0.smithi047.stdout:Stddev Bandwidth: 0.202505 2024-09-18T14:34:02.817 INFO:tasks.workunit.client.0.smithi047.stdout:Max bandwidth (MB/sec): 0.503906 2024-09-18T14:34:02.817 INFO:tasks.workunit.client.0.smithi047.stdout:Min bandwidth (MB/sec): 0 2024-09-18T14:34:02.817 INFO:tasks.workunit.client.0.smithi047.stdout:Average IOPS: 45 2024-09-18T14:34:02.817 INFO:tasks.workunit.client.0.smithi047.stdout:Stddev IOPS: 51.8892 2024-09-18T14:34:02.817 INFO:tasks.workunit.client.0.smithi047.stdout:Max IOPS: 129 2024-09-18T14:34:02.817 INFO:tasks.workunit.client.0.smithi047.stdout:Min IOPS: 0 2024-09-18T14:34:02.817 INFO:tasks.workunit.client.0.smithi047.stdout:Average Latency(s): 0.347548 2024-09-18T14:34:02.817 INFO:tasks.workunit.client.0.smithi047.stdout:Stddev Latency(s): 1.00212 2024-09-18T14:34:02.817 INFO:tasks.workunit.client.0.smithi047.stdout:Max latency(s): 5.7369 2024-09-18T14:34:02.817 INFO:tasks.workunit.client.0.smithi047.stdout:Min latency(s): 0.0954042 2024-09-18T14:34:02.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:49: TEST_filestore_to_bluestore: kill 170581 2024-09-18T14:34:02.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:49: TEST_filestore_to_bluestore: sleep 1 2024-09-18T14:34:03.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:49: TEST_filestore_to_bluestore: kill 170581 2024-09-18T14:34:03.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh: line 49: kill: (170581) - No such process 2024-09-18T14:34:03.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:50: TEST_filestore_to_bluestore: ceph osd down 0 2024-09-18T14:34:04.295 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already down. 2024-09-18T14:34:04.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:52: TEST_filestore_to_bluestore: mv td/osd-dup/0 td/osd-dup/0.old 2024-09-18T14:34:04.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:53: TEST_filestore_to_bluestore: mkdir td/osd-dup/0 2024-09-18T14:34:04.308 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:54: TEST_filestore_to_bluestore: cat td/osd-dup/0.old/fsid 2024-09-18T14:34:04.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:54: TEST_filestore_to_bluestore: ofsid=d5ff9deb-56b7-4855-a4be-d8096bdafafa 2024-09-18T14:34:04.309 INFO:tasks.workunit.client.0.smithi047.stdout:osd fsid d5ff9deb-56b7-4855-a4be-d8096bdafafa 2024-09-18T14:34:04.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:55: TEST_filestore_to_bluestore: echo 'osd fsid d5ff9deb-56b7-4855-a4be-d8096bdafafa' 2024-09-18T14:34:04.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:56: TEST_filestore_to_bluestore: O='--fsid=900aae69-8049-40e4-a548-ae813fbb3731 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2024-09-18T14:34:04.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:57: TEST_filestore_to_bluestore: CEPH_ARGS+='--log-file td/osd-dup/cot.log --log-max-recent 0 ' 2024-09-18T14:34:04.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:58: TEST_filestore_to_bluestore: ceph-objectstore-tool --type bluestore --data-path td/osd-dup/0 --fsid d5ff9deb-56b7-4855-a4be-d8096bdafafa --op mkfs --no-mon-config 2024-09-18T14:34:08.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:60: TEST_filestore_to_bluestore: ceph-objectstore-tool --data-path td/osd-dup/0.old --target-data-path td/osd-dup/0 --op dup 2024-09-18T14:34:08.522 INFO:tasks.workunit.client.0.smithi047.stdout:dup from filestore: td/osd-dup/0.old 2024-09-18T14:34:08.522 INFO:tasks.workunit.client.0.smithi047.stdout: to bluestore: td/osd-dup/0 2024-09-18T14:34:09.639 INFO:tasks.workunit.client.0.smithi047.stdout:fsid d5ff9deb-56b7-4855-a4be-d8096bdafafa 2024-09-18T14:34:09.640 INFO:tasks.workunit.client.0.smithi047.stdout:17 collections 2024-09-18T14:34:09.640 INFO:tasks.workunit.client.0.smithi047.stdout:1/17 1.3_head 2024-09-18T14:34:10.047 INFO:tasks.workunit.client.0.smithi047.stdout: 0 objects, 0 bytes, 0 keys 36 objects, 143360 bytes, 41 keys 2024-09-18T14:34:10.047 INFO:tasks.workunit.client.0.smithi047.stdout:2/17 1.1_head 2024-09-18T14:34:10.260 INFO:tasks.workunit.client.0.smithi047.stdout: 0 objects, 0 bytes, 0 keys 31 objects, 122880 bytes, 36 keys 2024-09-18T14:34:10.260 INFO:tasks.workunit.client.0.smithi047.stdout:3/17 1.8_head 2024-09-18T14:34:10.452 INFO:tasks.workunit.client.0.smithi047.stdout: 0 objects, 0 bytes, 0 keys 32 objects, 126976 bytes, 37 keys 2024-09-18T14:34:10.452 INFO:tasks.workunit.client.0.smithi047.stdout:4/17 1.d_head 2024-09-18T14:34:10.638 INFO:tasks.workunit.client.0.smithi047.stdout: 0 objects, 0 bytes, 0 keys 30 objects, 114712 bytes, 35 keys 2024-09-18T14:34:10.638 INFO:tasks.workunit.client.0.smithi047.stdout:5/17 1.f_head 2024-09-18T14:34:10.848 INFO:tasks.workunit.client.0.smithi047.stdout: 0 objects, 0 bytes, 0 keys 31 objects, 122880 bytes, 36 keys 2024-09-18T14:34:10.848 INFO:tasks.workunit.client.0.smithi047.stdout:6/17 1.7_head 2024-09-18T14:34:11.028 INFO:tasks.workunit.client.0.smithi047.stdout: 0 objects, 0 bytes, 0 keys 28 objects, 110592 bytes, 33 keys 2024-09-18T14:34:11.028 INFO:tasks.workunit.client.0.smithi047.stdout:7/17 1.5_head 2024-09-18T14:34:11.218 INFO:tasks.workunit.client.0.smithi047.stdout: 0 objects, 0 bytes, 0 keys 30 objects, 118784 bytes, 35 keys 2024-09-18T14:34:11.218 INFO:tasks.workunit.client.0.smithi047.stdout:8/17 1.e_head 2024-09-18T14:34:11.413 INFO:tasks.workunit.client.0.smithi047.stdout: 0 objects, 0 bytes, 0 keys 32 objects, 126976 bytes, 37 keys 2024-09-18T14:34:11.413 INFO:tasks.workunit.client.0.smithi047.stdout:9/17 1.6_head 2024-09-18T14:34:11.588 INFO:tasks.workunit.client.0.smithi047.stdout: 0 objects, 0 bytes, 0 keys 34 objects, 135168 bytes, 39 keys 2024-09-18T14:34:11.588 INFO:tasks.workunit.client.0.smithi047.stdout:10/17 meta 2024-09-18T14:34:11.592 INFO:tasks.workunit.client.0.smithi047.stdout: 0 objects, 0 bytes, 0 keys 46 objects, 47132 bytes, 1 keys 2024-09-18T14:34:11.592 INFO:tasks.workunit.client.0.smithi047.stdout:11/17 1.a_head 2024-09-18T14:34:11.692 INFO:tasks.workunit.client.0.smithi047.stdout: 0 objects, 0 bytes, 0 keys 22 objects, 86016 bytes, 27 keys 2024-09-18T14:34:11.692 INFO:tasks.workunit.client.0.smithi047.stdout:12/17 1.c_head 2024-09-18T14:34:11.834 INFO:tasks.workunit.client.0.smithi047.stdout: 0 objects, 0 bytes, 0 keys 30 objects, 118784 bytes, 35 keys 2024-09-18T14:34:11.834 INFO:tasks.workunit.client.0.smithi047.stdout:13/17 1.9_head 2024-09-18T14:34:11.993 INFO:tasks.workunit.client.0.smithi047.stdout: 0 objects, 0 bytes, 0 keys 24 objects, 94208 bytes, 29 keys 2024-09-18T14:34:11.993 INFO:tasks.workunit.client.0.smithi047.stdout:14/17 1.0_head 2024-09-18T14:34:12.203 INFO:tasks.workunit.client.0.smithi047.stdout: 0 objects, 0 bytes, 0 keys 37 objects, 147456 bytes, 42 keys 2024-09-18T14:34:12.203 INFO:tasks.workunit.client.0.smithi047.stdout:15/17 1.b_head 2024-09-18T14:34:12.338 INFO:tasks.workunit.client.0.smithi047.stdout: 0 objects, 0 bytes, 0 keys 25 objects, 98304 bytes, 30 keys 2024-09-18T14:34:12.338 INFO:tasks.workunit.client.0.smithi047.stdout:16/17 1.4_head 2024-09-18T14:34:12.515 INFO:tasks.workunit.client.0.smithi047.stdout: 0 objects, 0 bytes, 0 keys 30 objects, 118784 bytes, 35 keys 2024-09-18T14:34:12.515 INFO:tasks.workunit.client.0.smithi047.stdout:17/17 1.2_head 2024-09-18T14:34:12.518 INFO:tasks.workunit.client.0.smithi047.stdout: 0 objects, 0 bytes, 0 keys 30 objects, 118784 bytes, 35 keys 2024-09-18T14:34:12.518 INFO:tasks.workunit.client.0.smithi047.stdout:keyring 2024-09-18T14:34:12.518 INFO:tasks.workunit.client.0.smithi047.stdout:duping osd metadata 2024-09-18T14:34:12.935 INFO:tasks.workunit.client.0.smithi047.stdout:done. 2024-09-18T14:34:13.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:62: TEST_filestore_to_bluestore: CEPH_ARGS='--fsid=900aae69-8049-40e4-a548-ae813fbb3731 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2024-09-18T14:34:13.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:64: TEST_filestore_to_bluestore: activate_osd td/osd-dup 0 2024-09-18T14:34:13.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-dup 2024-09-18T14:34:13.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:34:13.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T14:34:13.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:34:13.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-dup/0 2024-09-18T14:34:13.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=900aae69-8049-40e4-a548-ae813fbb3731 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2024-09-18T14:34:13.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:34:13.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:34:13.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:34:13.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-dup/0' 2024-09-18T14:34:13.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-dup/0/journal' 2024-09-18T14:34:13.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:34:13.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:34:13.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-dup' 2024-09-18T14:34:13.708 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:34:13.708 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:34:13.708 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:34:13.708 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:34:13.709 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:34:13.709 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.170279 2024-09-18T14:34:13.709 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.170279/$cluster-$name.asok' 2024-09-18T14:34:13.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.170279/$cluster-$name.asok' 2024-09-18T14:34:13.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:34:13.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-dup/$name.log' 2024-09-18T14:34:13.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-dup/$name.pid' 2024-09-18T14:34:13.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:34:13.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:34:13.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:34:13.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:34:13.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:34:13.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:34:13.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-dup/0 2024-09-18T14:34:13.711 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T14:34:13.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T14:34:13.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=900aae69-8049-40e4-a548-ae813fbb3731 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/0 --osd-journal=td/osd-dup/0/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.170279/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:34:13.713 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-dup/0/whoami 2024-09-18T14:34:13.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T14:34:13.714 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:34:13.714 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:34:13.714 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:34:13.728 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:34:13.727+0000 7f4913637440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:34:13.730 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:34:13.729+0000 7f4913637440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:34:13.731 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:34:13.731+0000 7f4913637440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:34:14.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T14:34:14.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:34:14.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:34:14.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:34:14.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:34:14.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:34:14.093 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:34:14.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:34:14.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:34:14.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:34:14.249 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:34:14.248+0000 7f4913637440 -1 Falling back to public interface 2024-09-18T14:34:14.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:34:15.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:34:15.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:34:15.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:34:15.463 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:34:15.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:34:15.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:34:15.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:34:16.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:34:16.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:34:16.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:34:16.904 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:34:16.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:34:16.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:34:17.053 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:34:17.052+0000 7f4913637440 -1 osd.0 21 log_to_monitors true 2024-09-18T14:34:17.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:34:18.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:34:18.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:34:18.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:34:18.512 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:34:18.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:34:18.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:34:18.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:34:19.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:34:19.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:34:19.941 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:34:19.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:34:19.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:34:19.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:34:20.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:34:21.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:34:21.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:34:21.398 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:34:21.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:34:21.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:34:21.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:34:21.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:34:22.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:34:22.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:34:22.818 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:34:22.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:34:22.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:34:22.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:34:23.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:34:24.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:34:24.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:34:24.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:34:24.224 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:34:24.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:34:24.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:34:24.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:34:25.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:34:25.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:34:25.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:34:25.705 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:34:25.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:34:25.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:34:26.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:34:27.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:34:27.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:34:27.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:34:27.116 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:34:27.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:34:27.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:34:27.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:34:28.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:34:28.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:34:28.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:34:28.677 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:34:28.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:34:28.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:34:29.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:34:30.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:34:30.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:34:30.405 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:34:30.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:34:30.406 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:34:30.406 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:34:31.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:34:32.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:34:32.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:34:32.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:34:32.512 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:34:32.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:34:32.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:34:32.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:34:33.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:34:33.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:34:33.996 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:34:33.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:34:33.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:34:33.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:34:34.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:34:35.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:34:35.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:34:35.892 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:34:35.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:34:35.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:34:35.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:34:36.437 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:34:37.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:34:37.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:34:37.439 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:34:37.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:34:37.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:34:37.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:34:38.025 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 25 up_thru 25 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6800/172279,v1:127.0.0.1:6801/172279] [v2:127.0.0.1:6802/172279,v1:127.0.0.1:6803/172279] exists,up d5ff9deb-56b7-4855-a4be-d8096bdafafa 2024-09-18T14:34:38.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:34:38.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:34:38.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:34:38.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:66: TEST_filestore_to_bluestore: ceph osd stat 2024-09-18T14:34:38.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:66: TEST_filestore_to_bluestore: grep '3 up' 2024-09-18T14:34:38.580 INFO:tasks.workunit.client.0.smithi047.stdout:3 osds: 3 up (since 1.68874s), 3 in (since 57s); epoch: e27 2024-09-18T14:34:38.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:67: TEST_filestore_to_bluestore: ceph osd metadata 0 2024-09-18T14:34:38.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:67: TEST_filestore_to_bluestore: grep bluestore 2024-09-18T14:34:38.989 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_allocation_from_file": "0", 2024-09-18T14:34:38.989 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_access_mode": "file", 2024-09-18T14:34:38.989 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_block_size": "4096", 2024-09-18T14:34:38.989 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_devices": "sda", 2024-09-18T14:34:38.989 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-18T14:34:38.990 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-18T14:34:38.990 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_path": "td/osd-dup/0/block", 2024-09-18T14:34:38.990 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_rotational": "1", 2024-09-18T14:34:38.990 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_size": "107374182400", 2024-09-18T14:34:38.990 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_support_discard": "1", 2024-09-18T14:34:38.990 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_type": "hdd", 2024-09-18T14:34:38.990 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_min_alloc_size": "4096", 2024-09-18T14:34:38.990 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_objectstore": "bluestore", 2024-09-18T14:34:38.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:69: TEST_filestore_to_bluestore: ceph osd scrub 0 2024-09-18T14:34:39.340 INFO:tasks.workunit.client.0.smithi047.stderr:instructed osd(s) 0 to scrub 2024-09-18T14:34:39.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:72: TEST_filestore_to_bluestore: sleep 15 2024-09-18T14:34:54.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:74: TEST_filestore_to_bluestore: flush_pg_stats 2024-09-18T14:34:54.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:34:54.353 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:34:54.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:34:54.772 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T14:34:54.772 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T14:34:54.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:34:54.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:34:54.773 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:34:54.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182410 2024-09-18T14:34:54.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182410 2024-09-18T14:34:54.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182410' 2024-09-18T14:34:54.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:34:54.881 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:34:54.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672977 2024-09-18T14:34:54.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672977 2024-09-18T14:34:54.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182410 1-42949672977' 2024-09-18T14:34:54.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:34:54.988 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:34:55.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509456 2024-09-18T14:34:55.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509456 2024-09-18T14:34:55.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182410 1-42949672977 2-64424509456' 2024-09-18T14:34:55.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:34:55.094 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-107374182410 2024-09-18T14:34:55.094 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:34:55.095 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:34:55.096 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-107374182410 2024-09-18T14:34:55.096 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:34:55.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182410 2024-09-18T14:34:55.097 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 107374182410 2024-09-18T14:34:55.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 107374182410' 2024-09-18T14:34:55.097 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:34:55.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182408 -lt 107374182410 2024-09-18T14:34:55.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:34:56.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:34:56.463 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:34:56.875 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182410 -lt 107374182410 2024-09-18T14:34:56.875 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:34:56.875 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672977 2024-09-18T14:34:56.875 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:34:56.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:34:56.877 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672977 2024-09-18T14:34:56.877 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:34:56.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672977 2024-09-18T14:34:56.878 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672977 2024-09-18T14:34:56.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-09-18T14:34:56.879 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:34:57.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672978 -lt 42949672977 2024-09-18T14:34:57.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:34:57.284 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509456 2024-09-18T14:34:57.285 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:34:57.285 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:34:57.286 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509456 2024-09-18T14:34:57.286 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:34:57.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509456 2024-09-18T14:34:57.287 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509456 2024-09-18T14:34:57.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-09-18T14:34:57.288 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:34:57.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509457 -lt 64424509456 2024-09-18T14:34:57.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:76: TEST_filestore_to_bluestore: wait_for_clean 2024-09-18T14:34:57.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:34:57.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:34:57.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:34:57.694 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:34:57.695 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:34:57.695 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:34:57.695 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:34:57.695 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:34:57.695 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:34:57.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T14:34:57.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:34:57.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:34:57.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:34:57.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:34:57.783 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:34:58.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:34:58.158 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T14:34:58.158 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T14:34:58.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:34:58.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:34:58.159 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:34:58.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182413 2024-09-18T14:34:58.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182413 2024-09-18T14:34:58.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182413' 2024-09-18T14:34:58.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:34:58.266 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:34:58.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672980 2024-09-18T14:34:58.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672980 2024-09-18T14:34:58.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182413 1-42949672980' 2024-09-18T14:34:58.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:34:58.372 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:34:58.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509459 2024-09-18T14:34:58.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509459 2024-09-18T14:34:58.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182413 1-42949672980 2-64424509459' 2024-09-18T14:34:58.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:34:58.479 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-107374182413 2024-09-18T14:34:58.479 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:34:58.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:34:58.480 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-107374182413 2024-09-18T14:34:58.480 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:34:58.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182413 2024-09-18T14:34:58.481 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 107374182413 2024-09-18T14:34:58.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 107374182413' 2024-09-18T14:34:58.482 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:34:58.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182411 -lt 107374182413 2024-09-18T14:34:58.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:34:59.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:34:59.889 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:35:00.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182413 -lt 107374182413 2024-09-18T14:35:00.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:35:00.291 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672980 2024-09-18T14:35:00.291 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:35:00.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:35:00.292 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672980 2024-09-18T14:35:00.292 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:35:00.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672980 2024-09-18T14:35:00.294 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672980 2024-09-18T14:35:00.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-09-18T14:35:00.294 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:35:00.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672980 -lt 42949672980 2024-09-18T14:35:00.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:35:00.732 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509459 2024-09-18T14:35:00.732 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:35:00.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:35:00.734 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509459 2024-09-18T14:35:00.734 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:35:00.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509459 2024-09-18T14:35:00.735 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509459 2024-09-18T14:35:00.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-09-18T14:35:00.736 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:35:01.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509459 -lt 64424509459 2024-09-18T14:35:01.141 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:35:01.142 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:35:01.142 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:35:01.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-09-18T14:35:01.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:35:01.635 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:35:01.635 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:35:01.635 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:35:01.635 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:35:01.635 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:35:01.635 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:35:02.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-09-18T14:35:02.039 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:35:02.039 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:35:02.039 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:35:02.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-09-18T14:35:02.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:35:02.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:35:02.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:22: run: teardown td/osd-dup 2024-09-18T14:35:02.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-dup 2024-09-18T14:35:02.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:35:02.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-dup KILL 2024-09-18T14:35:02.540 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:35:02.540 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:35:02.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:35:02.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:35:02.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:35:02.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:35:02.649 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:35:02.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:35:02.650 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:35:02.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:35:02.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:35:02.652 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:35:02.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:35:02.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:35:02.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:35:02.653 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:35:02.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:35:02.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:35:02.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-dup 2024-09-18T14:35:02.692 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:35:02.692 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:35:02.692 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.170279 2024-09-18T14:35:02.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.170279 2024-09-18T14:35:02.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:35:02.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:35:02.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T14:35:02.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/osd-dup 0 2024-09-18T14:35:02.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-dup 2024-09-18T14:35:02.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T14:35:02.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-dup KILL 2024-09-18T14:35:02.694 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:35:02.695 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:35:02.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:35:02.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:35:02.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:35:02.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:35:02.696 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:35:02.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:35:02.697 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:35:02.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:35:02.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:35:02.698 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:35:02.699 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:35:02.699 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:35:02.699 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:35:02.699 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:35:02.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:35:02.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T14:35:02.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-dup 2024-09-18T14:35:02.702 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:35:02.702 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:35:02.702 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.170279 2024-09-18T14:35:02.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.170279 2024-09-18T14:35:02.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:35:02.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:35:02.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T14:35:02.705 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T14:35:02.705 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T14:35:02.770 INFO:tasks.workunit:Running workunit osd/osd-fast-mark-down.sh... 2024-09-18T14:35:02.771 DEBUG:teuthology.orchestra.run.smithi047:workunit test osd/osd-fast-mark-down.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh 2024-09-18T14:35:02.833 INFO:tasks.workunit.client.0.smithi047.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T14:35:02.839 INFO:tasks.workunit.client.0.smithi047.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T14:35:02.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T14:35:02.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T14:35:02.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T14:35:02.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T14:35:02.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T14:35:02.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T14:35:02.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T14:35:02.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T14:35:02.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/osd-fast-mark-down 2024-09-18T14:35:02.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:23: run: local dir=td/osd-fast-mark-down 2024-09-18T14:35:02.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:24: run: shift 2024-09-18T14:35:02.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:25: run: rm -f 'td/osd-fast-mark-down/*.pid' 2024-09-18T14:35:02.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:26: run: export CEPH_MON=127.0.0.1:7126 2024-09-18T14:35:02.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:26: run: CEPH_MON=127.0.0.1:7126 2024-09-18T14:35:02.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:27: run: export CEPH_ARGS 2024-09-18T14:35:02.840 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:28: run: uuidgen 2024-09-18T14:35:02.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:28: run: CEPH_ARGS+='--fsid=4c08e6a1-10a6-4a85-9216-0ce33aa35b79 --auth-supported=none ' 2024-09-18T14:35:02.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:30: run: OLD_ARGS='--fsid=4c08e6a1-10a6-4a85-9216-0ce33aa35b79 --auth-supported=none ' 2024-09-18T14:35:02.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:31: run: CEPH_ARGS+='--osd-fast-fail-on-connection-refused=false ' 2024-09-18T14:35:02.841 INFO:tasks.workunit.client.0.smithi047.stdout:Ensuring old behavior is there... 2024-09-18T14:35:02.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:32: run: echo 'Ensuring old behavior is there...' 2024-09-18T14:35:02.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:33: run: test_fast_kill td/osd-fast-mark-down 2024-09-18T14:35:02.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:48: test_fast_kill: setup td/osd-fast-mark-down 2024-09-18T14:35:02.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-fast-mark-down 2024-09-18T14:35:02.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-fast-mark-down 2024-09-18T14:35:02.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2024-09-18T14:35:02.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:35:02.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2024-09-18T14:35:02.843 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:35:02.843 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:35:02.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:35:02.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:35:02.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:35:02.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:35:02.844 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:35:02.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:35:02.844 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:35:02.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:35:02.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:35:02.845 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:35:02.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:35:02.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:35:02.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:35:02.846 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:35:02.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:35:02.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:35:02.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2024-09-18T14:35:02.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:35:02.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:35:02.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173834 2024-09-18T14:35:02.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173834 2024-09-18T14:35:02.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:35:02.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:35:02.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-fast-mark-down 2024-09-18T14:35:02.850 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T14:35:02.851 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:35:02.851 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173834 2024-09-18T14:35:02.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.173834 2024-09-18T14:35:02.852 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T14:35:02.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T14:35:02.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T14:35:02.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T14:35:02.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-fast-mark-down 1' TERM HUP INT 2024-09-18T14:35:02.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: run_mon td/osd-fast-mark-down a --osd_pool_default_size=3 2024-09-18T14:35:02.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-fast-mark-down 2024-09-18T14:35:02.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T14:35:02.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T14:35:02.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T14:35:02.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-fast-mark-down/a 2024-09-18T14:35:02.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-fast-mark-down/a --run-dir=td/osd-fast-mark-down --osd_pool_default_size=3 2024-09-18T14:35:02.871 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:35:02.870+0000 7f34744ce900 -1 failed for service _ceph-mon._tcp 2024-09-18T14:35:02.871 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:35:02.870+0000 7f34744ce900 -1 unable to get monitor info from DNS SRV with service name: ceph-mon 2024-09-18T14:35:02.871 INFO:tasks.workunit.client.0.smithi047.stderr: 2024-09-18T14:35:02.871 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:35:02.870+0000 7f34744ce900 -1 ceph-mon: warning: no initial monitors; must use admin socket to feed hints 2024-09-18T14:35:03.218 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T14:35:03.218 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:35:03.218 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:35:03.218 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:35:03.218 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:35:03.218 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173834 2024-09-18T14:35:03.218 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173834/$cluster-$name.asok' 2024-09-18T14:35:03.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-fast-mark-down/a '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.173834/$cluster-$name.asok' --mon-cluster-log-file=td/osd-fast-mark-down/log --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-18T14:35:03.491 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:35:03.490+0000 7f32332c2900 -1 failed for service _ceph-mon._tcp 2024-09-18T14:35:03.491 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:35:03.490+0000 7f32332c2900 -1 unable to get monitor info from DNS SRV with service name: ceph-mon 2024-09-18T14:35:03.491 INFO:tasks.workunit.client.0.smithi047.stderr: 2024-09-18T14:35:03.491 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:35:03.490+0000 7f32332c2900 -1 ceph-mon: error generating initial monmap: (2) No such file or directory 2024-09-18T14:35:03.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: return 1 2024-09-18T14:35:03.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: return 1 2024-09-18T14:35:03.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:35: run: CEPH_ARGS='--fsid=4c08e6a1-10a6-4a85-9216-0ce33aa35b79 --auth-supported=none --osd-fast-fail-on-connection-refused=true ' 2024-09-18T14:35:03.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:36: run: OLD_ARGS='--fsid=4c08e6a1-10a6-4a85-9216-0ce33aa35b79 --auth-supported=none --osd-fast-fail-on-connection-refused=true ' 2024-09-18T14:35:03.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:38: run: CEPH_ARGS='--fsid=4c08e6a1-10a6-4a85-9216-0ce33aa35b79 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2024-09-18T14:35:03.495 INFO:tasks.workunit.client.0.smithi047.stdout:Testing async msgr... 2024-09-18T14:35:03.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:39: run: echo 'Testing async msgr...' 2024-09-18T14:35:03.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:40: run: test_fast_kill td/osd-fast-mark-down 2024-09-18T14:35:03.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:48: test_fast_kill: setup td/osd-fast-mark-down 2024-09-18T14:35:03.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-fast-mark-down 2024-09-18T14:35:03.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-fast-mark-down 2024-09-18T14:35:03.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2024-09-18T14:35:03.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:35:03.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2024-09-18T14:35:03.497 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:35:03.497 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:35:03.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:35:03.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:35:03.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:35:03.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:35:03.498 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:35:03.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:35:03.499 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:35:03.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:35:03.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:35:03.501 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:35:03.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:35:03.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:35:03.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:35:03.503 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:35:03.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:35:03.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:35:03.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2024-09-18T14:35:03.506 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:35:03.506 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:35:03.507 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173834 2024-09-18T14:35:03.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173834 2024-09-18T14:35:03.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:35:03.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:35:03.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-fast-mark-down 2024-09-18T14:35:03.509 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T14:35:03.509 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:35:03.509 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173834 2024-09-18T14:35:03.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.173834 2024-09-18T14:35:03.511 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T14:35:03.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T14:35:03.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T14:35:03.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-fast-mark-down 1' TERM HUP INT 2024-09-18T14:35:03.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: run_mon td/osd-fast-mark-down a --osd_pool_default_size=3 2024-09-18T14:35:03.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-fast-mark-down 2024-09-18T14:35:03.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T14:35:03.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T14:35:03.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T14:35:03.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-fast-mark-down/a 2024-09-18T14:35:03.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-fast-mark-down/a --run-dir=td/osd-fast-mark-down --osd_pool_default_size=3 2024-09-18T14:35:03.864 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T14:35:03.864 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:35:03.864 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:35:03.864 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:35:03.864 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:35:03.864 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173834 2024-09-18T14:35:03.864 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173834/$cluster-$name.asok' 2024-09-18T14:35:03.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-fast-mark-down/a '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.173834/$cluster-$name.asok' --mon-cluster-log-file=td/osd-fast-mark-down/log --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-18T14:35:04.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T14:35:04.124 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T14:35:04.124 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:35:04.124 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:35:04.125 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T14:35:04.126 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T14:35:04.126 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:35:04.126 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:35:04.126 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:35:04.126 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:35:04.126 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:35:04.126 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173834 2024-09-18T14:35:04.126 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.173834/ceph-mon.a.asok 2024-09-18T14:35:04.126 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:35:04.127 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.173834/ceph-mon.a.asok config get fsid 2024-09-18T14:35:04.212 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T14:35:04.213 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:35:04.213 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:35:04.213 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T14:35:04.213 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T14:35:04.213 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:35:04.213 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:35:04.213 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:35:04.213 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:35:04.213 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:35:04.213 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173834 2024-09-18T14:35:04.213 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.173834/ceph-mon.a.asok 2024-09-18T14:35:04.213 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:35:04.214 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.173834/ceph-mon.a.asok config get mon_host 2024-09-18T14:35:04.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:50: test_fast_kill: run_mgr td/osd-fast-mark-down x 2024-09-18T14:35:04.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-fast-mark-down 2024-09-18T14:35:04.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T14:35:04.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T14:35:04.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T14:35:04.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-fast-mark-down/x 2024-09-18T14:35:04.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T14:35:05.145 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T14:35:05.145 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:35:05.146 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:35:05.146 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:35:05.146 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:35:05.146 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173834 2024-09-18T14:35:05.146 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173834/$cluster-$name.asok' 2024-09-18T14:35:05.146 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:35:05.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-fast-mark-down/x '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.173834/$cluster-$name.asok' --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:35:05.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2024-09-18T14:35:05.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 0 2024-09-18T14:35:05.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2024-09-18T14:35:05.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:35:05.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T14:35:05.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:35:05.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/0 2024-09-18T14:35:05.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4c08e6a1-10a6-4a85-9216-0ce33aa35b79 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2024-09-18T14:35:05.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:35:05.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:35:05.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:35:05.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/0' 2024-09-18T14:35:05.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/0/journal' 2024-09-18T14:35:05.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:35:05.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:35:05.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2024-09-18T14:35:05.175 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:35:05.175 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:35:05.175 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:35:05.175 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:35:05.175 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:35:05.175 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173834 2024-09-18T14:35:05.175 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173834/$cluster-$name.asok' 2024-09-18T14:35:05.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173834/$cluster-$name.asok' 2024-09-18T14:35:05.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:35:05.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:35:05.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:35:05.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2024-09-18T14:35:05.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2024-09-18T14:35:05.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:35:05.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:35:05.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:35:05.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:35:05.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:35:05.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:35:05.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-fast-mark-down/0 2024-09-18T14:35:05.176 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:35:05.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=912da02d-7ffe-4dee-b5b1-7611891351a1 2024-09-18T14:35:05.177 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 912da02d-7ffe-4dee-b5b1-7611891351a1 2024-09-18T14:35:05.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 912da02d-7ffe-4dee-b5b1-7611891351a1' 2024-09-18T14:35:05.177 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:35:05.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAZ5epmIe9KCxAAFftNgPAzarvlWomQB8yRog== 2024-09-18T14:35:05.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAZ5epmIe9KCxAAFftNgPAzarvlWomQB8yRog=="}' 2024-09-18T14:35:05.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 912da02d-7ffe-4dee-b5b1-7611891351a1 -i td/osd-fast-mark-down/0/new.json 2024-09-18T14:35:05.487 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:35:05.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-fast-mark-down/0/new.json 2024-09-18T14:35:05.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=4c08e6a1-10a6-4a85-9216-0ce33aa35b79 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/0 --osd-journal=td/osd-fast-mark-down/0/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.173834/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAZ5epmIe9KCxAAFftNgPAzarvlWomQB8yRog== --osd-uuid 912da02d-7ffe-4dee-b5b1-7611891351a1 2024-09-18T14:35:05.524 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:35:05.523+0000 7ff49a521440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:35:05.532 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:35:05.531+0000 7ff49a521440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:35:05.533 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:35:05.532+0000 7ff49a521440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:35:05.533 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:35:05.532+0000 7ff49a521440 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2024-09-18T14:35:05.533 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:35:05.532+0000 7ff49a521440 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2024-09-18T14:35:05.533 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:35:05.532+0000 7ff49a521440 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2024-09-18T14:35:05.572 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:35:05.571+0000 7ff49a521440 -1 bluestore(td/osd-fast-mark-down/0) _read_fsid unparsable uuid 2024-09-18T14:35:10.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-fast-mark-down/0/keyring 2024-09-18T14:35:10.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:35:10.894 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T14:35:10.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T14:35:10.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-fast-mark-down/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:35:11.408 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T14:35:11.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T14:35:11.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=4c08e6a1-10a6-4a85-9216-0ce33aa35b79 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/0 --osd-journal=td/osd-fast-mark-down/0/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.173834/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:35:11.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:35:11.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:35:11.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:35:11.433 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:35:11.431+0000 7f18cfc34440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:35:11.434 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:35:11.434+0000 7f18cfc34440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:35:11.435 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:35:11.435+0000 7f18cfc34440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:35:11.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T14:35:11.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:35:11.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:35:11.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:35:11.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:35:11.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:11.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:35:11.858 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:35:11.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:11.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:35:11.974 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:35:11.974+0000 7f18cfc34440 -1 Falling back to public interface 2024-09-18T14:35:12.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:35:13.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:35:13.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:13.269 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:35:13.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:35:13.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:13.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:35:13.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:35:14.457 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:35:14.456+0000 7f18cfc34440 -1 osd.0 0 log_to_monitors true 2024-09-18T14:35:14.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:35:14.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:14.687 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:35:14.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:35:14.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:14.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:35:15.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:35:16.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:35:16.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:16.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:35:16.150 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:35:16.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:16.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:35:16.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:35:17.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:35:17.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:17.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:35:17.532 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:35:17.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:17.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:35:17.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:35:18.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:35:18.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:18.915 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:35:18.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:35:18.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:18.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:35:19.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:35:20.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:35:20.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:20.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:35:20.293 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:35:20.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:20.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:35:20.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:35:21.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:35:21.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:21.705 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:35:21.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:35:21.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:21.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:35:22.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:35:23.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:35:23.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:23.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:35:23.149 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:35:23.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:23.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:35:23.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:35:24.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:35:24.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:24.574 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:35:24.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:35:24.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:24.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:35:24.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:35:25.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:35:25.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:25.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:35:25.955 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:35:25.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:25.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:35:26.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:35:27.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:35:27.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:27.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:35:27.632 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:35:27.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:27.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:35:28.247 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:35:29.248 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:35:29.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:29.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:35:29.249 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:35:29.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:29.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:35:30.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:35:31.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:35:31.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:31.210 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:35:31.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:35:31.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:31.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:35:32.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:35:33.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:35:33.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:33.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:35:33.031 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:35:33.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:33.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:35:34.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:35:35.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:35:35.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:35.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:35:35.300 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:35:35.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:35.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:35:36.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:35:37.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:35:37.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:37.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:35:37.365 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:35:37.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:37.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:35:37.753 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/174990,v1:127.0.0.1:6803/174990] [v2:127.0.0.1:6804/174990,v1:127.0.0.1:6805/174990] exists,up 912da02d-7ffe-4dee-b5b1-7611891351a1 2024-09-18T14:35:37.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:35:37.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:35:37.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:35:37.753 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.0.pid 2024-09-18T14:35:37.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=174990 2024-09-18T14:35:37.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2024-09-18T14:35:37.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 1 2024-09-18T14:35:37.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2024-09-18T14:35:37.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:35:37.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T14:35:37.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:35:37.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/1 2024-09-18T14:35:37.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4c08e6a1-10a6-4a85-9216-0ce33aa35b79 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2024-09-18T14:35:37.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:35:37.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:35:37.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:35:37.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/1' 2024-09-18T14:35:37.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/1/journal' 2024-09-18T14:35:37.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:35:37.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:35:37.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2024-09-18T14:35:37.758 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:35:37.758 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:35:37.758 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:35:37.758 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:35:37.758 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:35:37.759 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173834 2024-09-18T14:35:37.759 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173834/$cluster-$name.asok' 2024-09-18T14:35:37.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173834/$cluster-$name.asok' 2024-09-18T14:35:37.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:35:37.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:35:37.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:35:37.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2024-09-18T14:35:37.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2024-09-18T14:35:37.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:35:37.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:35:37.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:35:37.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:35:37.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:35:37.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:35:37.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-fast-mark-down/1 2024-09-18T14:35:37.760 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:35:37.760 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f8485a9a-7bfe-48c8-b7a1-312c778f9558 2024-09-18T14:35:37.760 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f8485a9a-7bfe-48c8-b7a1-312c778f9558' 2024-09-18T14:35:37.760 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 f8485a9a-7bfe-48c8-b7a1-312c778f9558 2024-09-18T14:35:37.760 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:35:37.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA55epm29QMLhAA8lmCqM78VGaBDvrfQ9vQFw== 2024-09-18T14:35:37.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA55epm29QMLhAA8lmCqM78VGaBDvrfQ9vQFw=="}' 2024-09-18T14:35:37.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f8485a9a-7bfe-48c8-b7a1-312c778f9558 -i td/osd-fast-mark-down/1/new.json 2024-09-18T14:35:38.247 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:35:38.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-fast-mark-down/1/new.json 2024-09-18T14:35:38.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=4c08e6a1-10a6-4a85-9216-0ce33aa35b79 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/1 --osd-journal=td/osd-fast-mark-down/1/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.173834/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA55epm29QMLhAA8lmCqM78VGaBDvrfQ9vQFw== --osd-uuid f8485a9a-7bfe-48c8-b7a1-312c778f9558 2024-09-18T14:35:38.285 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:35:38.284+0000 7fc507535440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:35:38.287 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:35:38.286+0000 7fc507535440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:35:38.288 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:35:38.287+0000 7fc507535440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:35:38.288 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:35:38.287+0000 7fc507535440 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2024-09-18T14:35:38.288 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:35:38.287+0000 7fc507535440 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2024-09-18T14:35:38.288 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:35:38.287+0000 7fc507535440 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2024-09-18T14:35:38.337 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:35:38.337+0000 7fc507535440 -1 bluestore(td/osd-fast-mark-down/1) _read_fsid unparsable uuid 2024-09-18T14:35:43.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-fast-mark-down/1/keyring 2024-09-18T14:35:43.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:35:43.960 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T14:35:43.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T14:35:43.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-fast-mark-down/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:35:44.469 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T14:35:44.469 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T14:35:44.470 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=4c08e6a1-10a6-4a85-9216-0ce33aa35b79 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/1 --osd-journal=td/osd-fast-mark-down/1/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.173834/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:35:44.470 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:35:44.470 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:35:44.470 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:35:44.493 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:35:44.492+0000 7f28eac1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:35:44.495 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:35:44.495+0000 7f28eac1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:35:44.496 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:35:44.496+0000 7f28eac1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:35:44.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T14:35:44.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:35:44.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:35:44.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:35:44.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:35:44.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:44.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:35:44.874 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:35:44.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:44.875 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:35:45.014 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:35:45.013+0000 7f28eac1f440 -1 Falling back to public interface 2024-09-18T14:35:45.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:35:46.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:35:46.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:46.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:35:46.246 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:35:46.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:46.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:35:46.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:35:47.619 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:35:47.618+0000 7f28eac1f440 -1 osd.1 0 log_to_monitors true 2024-09-18T14:35:47.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:35:47.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:47.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:35:47.664 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:35:47.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:47.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:35:48.247 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:35:49.248 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:35:49.248 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:49.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:35:49.249 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:35:49.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:49.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:35:49.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:35:50.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:35:50.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:50.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:35:50.697 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:35:50.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:50.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:35:51.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:35:52.079 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:35:52.080 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:52.080 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:35:52.080 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:35:52.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:52.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:35:52.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:35:53.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:35:53.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:53.488 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:35:53.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:35:53.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:53.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:35:53.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:35:54.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:35:54.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:54.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:35:54.899 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:35:54.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:54.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:35:55.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:35:56.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:35:56.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:56.332 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:35:56.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:35:56.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:56.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:35:56.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:35:57.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:35:57.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:57.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:35:57.709 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:35:57.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:57.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:35:58.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:35:59.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:35:59.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:35:59.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:35:59.118 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:35:59.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:35:59.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:35:59.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:00.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:00.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:00.619 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:36:00.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:36:00.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:00.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:36:01.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:02.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:02.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:02.105 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:36:02.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:36:02.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:02.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:36:02.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:03.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:03.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:03.688 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:36:03.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:36:03.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:03.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:36:04.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:05.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:05.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:05.448 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:36:05.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:36:05.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:05.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:36:06.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:07.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:07.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:07.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:36:07.166 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:36:07.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:07.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:36:08.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:09.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:09.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:09.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:36:09.255 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:36:09.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:09.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:36:09.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:10.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:10.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:10.753 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T14:36:10.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:36:10.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:10.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:36:11.131 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/176721,v1:127.0.0.1:6811/176721] [v2:127.0.0.1:6812/176721,v1:127.0.0.1:6813/176721] exists,up f8485a9a-7bfe-48c8-b7a1-312c778f9558 2024-09-18T14:36:11.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:36:11.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:36:11.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:36:11.132 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.1.pid 2024-09-18T14:36:11.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=176721 2024-09-18T14:36:11.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2024-09-18T14:36:11.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 2 2024-09-18T14:36:11.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2024-09-18T14:36:11.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:36:11.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T14:36:11.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:36:11.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/2 2024-09-18T14:36:11.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4c08e6a1-10a6-4a85-9216-0ce33aa35b79 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2024-09-18T14:36:11.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:36:11.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:36:11.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:36:11.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/2' 2024-09-18T14:36:11.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/2/journal' 2024-09-18T14:36:11.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:36:11.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:36:11.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2024-09-18T14:36:11.137 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:36:11.137 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:36:11.137 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:36:11.137 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:36:11.137 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:36:11.137 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173834 2024-09-18T14:36:11.137 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173834/$cluster-$name.asok' 2024-09-18T14:36:11.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173834/$cluster-$name.asok' 2024-09-18T14:36:11.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:36:11.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:36:11.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:36:11.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2024-09-18T14:36:11.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2024-09-18T14:36:11.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:36:11.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:36:11.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:36:11.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:36:11.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:36:11.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:36:11.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-fast-mark-down/2 2024-09-18T14:36:11.138 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:36:11.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f13bdbee-0351-46d4-94c5-a58ece4c3b10 2024-09-18T14:36:11.139 INFO:tasks.workunit.client.0.smithi047.stdout:add osd2 f13bdbee-0351-46d4-94c5-a58ece4c3b10 2024-09-18T14:36:11.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f13bdbee-0351-46d4-94c5-a58ece4c3b10' 2024-09-18T14:36:11.140 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:36:11.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBb5epmdrwJCRAAtNie+uY4AL+sThBZDht9bg== 2024-09-18T14:36:11.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBb5epmdrwJCRAAtNie+uY4AL+sThBZDht9bg=="}' 2024-09-18T14:36:11.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f13bdbee-0351-46d4-94c5-a58ece4c3b10 -i td/osd-fast-mark-down/2/new.json 2024-09-18T14:36:11.669 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:36:11.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-fast-mark-down/2/new.json 2024-09-18T14:36:11.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=4c08e6a1-10a6-4a85-9216-0ce33aa35b79 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/2 --osd-journal=td/osd-fast-mark-down/2/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.173834/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBb5epmdrwJCRAAtNie+uY4AL+sThBZDht9bg== --osd-uuid f13bdbee-0351-46d4-94c5-a58ece4c3b10 2024-09-18T14:36:11.706 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:36:11.706+0000 7f1b1668b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:36:11.708 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:36:11.708+0000 7f1b1668b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:36:11.709 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:36:11.709+0000 7f1b1668b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:36:11.709 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:36:11.710+0000 7f1b1668b440 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2024-09-18T14:36:11.709 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:36:11.710+0000 7f1b1668b440 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2024-09-18T14:36:11.710 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:36:11.710+0000 7f1b1668b440 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2024-09-18T14:36:11.759 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:36:11.759+0000 7f1b1668b440 -1 bluestore(td/osd-fast-mark-down/2) _read_fsid unparsable uuid 2024-09-18T14:36:17.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-fast-mark-down/2/keyring 2024-09-18T14:36:17.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:36:17.847 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd2 key to auth repository 2024-09-18T14:36:17.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T14:36:17.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-fast-mark-down/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:36:18.357 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T14:36:18.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T14:36:18.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=4c08e6a1-10a6-4a85-9216-0ce33aa35b79 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/2 --osd-journal=td/osd-fast-mark-down/2/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.173834/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:36:18.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:36:18.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:36:18.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:36:18.382 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:36:18.381+0000 7f3995019440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:36:18.384 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:36:18.383+0000 7f3995019440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:36:18.385 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:36:18.384+0000 7f3995019440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:36:18.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T14:36:18.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:36:18.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T14:36:18.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:36:18.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:36:18.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:18.726 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:36:18.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:36:18.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:18.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:36:18.900 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:36:18.899+0000 7f3995019440 -1 Falling back to public interface 2024-09-18T14:36:19.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:20.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:20.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:20.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:36:20.100 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:36:20.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:20.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:36:20.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:21.513 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:36:21.512+0000 7f3995019440 -1 osd.2 0 log_to_monitors true 2024-09-18T14:36:21.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:21.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:21.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:36:21.517 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:36:21.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:21.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:36:21.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:22.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:22.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:22.972 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:36:22.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:36:22.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:22.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:36:23.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:24.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:24.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:24.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:36:24.394 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:36:24.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:24.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:36:24.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:25.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:25.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:25.819 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:36:25.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:36:25.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:25.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:36:26.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:27.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:27.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:27.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:36:27.258 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:36:27.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:27.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:36:27.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:28.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:28.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:28.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:36:28.752 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:36:28.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:28.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:36:29.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:30.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:30.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:30.129 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:36:30.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:36:30.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:30.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:36:30.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:31.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:31.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:31.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:36:31.583 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:36:31.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:31.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:36:32.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:33.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:33.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:33.076 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:36:33.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:36:33.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:33.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:36:33.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:34.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:34.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:34.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:36:34.954 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:36:34.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:34.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:36:35.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:36.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:36.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:36.652 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:36:36.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:36:36.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:36.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:36:37.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:38.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:38.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:38.245 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:36:38.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:36:38.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:38.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:36:39.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:40.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:40.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:40.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:36:40.012 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:36:40.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:40.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:36:40.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:41.956 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:41.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:41.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:36:41.957 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:36:41.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:41.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:36:42.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:43.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:43.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:43.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:36:43.687 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:36:43.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:43.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:36:44.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:36:45.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:36:45.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:36:45.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:36:45.189 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T14:36:45.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:36:45.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:36:45.622 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/178476,v1:127.0.0.1:6819/178476] [v2:127.0.0.1:6820/178476,v1:127.0.0.1:6821/178476] exists,up f13bdbee-0351-46d4-94c5-a58ece4c3b10 2024-09-18T14:36:45.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:36:45.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:36:45.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:36:45.623 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.2.pid 2024-09-18T14:36:45.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=178476 2024-09-18T14:36:45.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:56: test_fast_kill: create_rbd_pool 2024-09-18T14:36:45.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T14:36:46.006 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'rbd' does not exist 2024-09-18T14:36:46.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T14:36:46.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T14:36:46.527 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'rbd' already exists 2024-09-18T14:36:46.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T14:36:47.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T14:36:49.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:59: test_fast_kill: timeout 20 rados -p rbd bench 10 write -b 4096 --max-objects 128 --no-cleanup 2024-09-18T14:36:49.600 INFO:tasks.workunit.client.0.smithi047.stdout:hints = 1 2024-09-18T14:36:49.600 INFO:tasks.workunit.client.0.smithi047.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 128 objects 2024-09-18T14:36:49.600 INFO:tasks.workunit.client.0.smithi047.stdout:Object prefix: benchmark_data_smithi047_179456 2024-09-18T14:36:49.600 INFO:tasks.workunit.client.0.smithi047.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T14:36:49.601 INFO:tasks.workunit.client.0.smithi047.stdout: 0 0 0 0 0 0 - 0 2024-09-18T14:36:50.600 INFO:tasks.workunit.client.0.smithi047.stdout: 1 16 80 64 0.249972 0.25 0.271673 0.221659 2024-09-18T14:36:51.600 INFO:tasks.workunit.client.0.smithi047.stdout:Total time run: 1.91875 2024-09-18T14:36:51.600 INFO:tasks.workunit.client.0.smithi047.stdout:Total writes made: 128 2024-09-18T14:36:51.601 INFO:tasks.workunit.client.0.smithi047.stdout:Write size: 4096 2024-09-18T14:36:51.601 INFO:tasks.workunit.client.0.smithi047.stdout:Object size: 4096 2024-09-18T14:36:51.601 INFO:tasks.workunit.client.0.smithi047.stdout:Bandwidth (MB/sec): 0.260586 2024-09-18T14:36:51.601 INFO:tasks.workunit.client.0.smithi047.stdout:Stddev Bandwidth: 0 2024-09-18T14:36:51.601 INFO:tasks.workunit.client.0.smithi047.stdout:Max bandwidth (MB/sec): 0.25 2024-09-18T14:36:51.601 INFO:tasks.workunit.client.0.smithi047.stdout:Min bandwidth (MB/sec): 0.25 2024-09-18T14:36:51.601 INFO:tasks.workunit.client.0.smithi047.stdout:Average IOPS: 66 2024-09-18T14:36:51.601 INFO:tasks.workunit.client.0.smithi047.stdout:Stddev IOPS: 0 2024-09-18T14:36:51.601 INFO:tasks.workunit.client.0.smithi047.stdout:Max IOPS: 64 2024-09-18T14:36:51.601 INFO:tasks.workunit.client.0.smithi047.stdout:Min IOPS: 64 2024-09-18T14:36:51.601 INFO:tasks.workunit.client.0.smithi047.stdout:Average Latency(s): 0.237797 2024-09-18T14:36:51.601 INFO:tasks.workunit.client.0.smithi047.stdout:Stddev Latency(s): 0.0398286 2024-09-18T14:36:51.601 INFO:tasks.workunit.client.0.smithi047.stdout:Max latency(s): 0.281479 2024-09-18T14:36:51.601 INFO:tasks.workunit.client.0.smithi047.stdout:Min latency(s): 0.130719 2024-09-18T14:36:51.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:60: test_fast_kill: sleep 1 2024-09-18T14:36:52.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:62: test_fast_kill: killid=0 2024-09-18T14:36:52.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:63: test_fast_kill: previd=0 2024-09-18T14:36:52.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:66: test_fast_kill: for i in {1..2} 2024-09-18T14:36:52.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 0 -eq 0 ']' 2024-09-18T14:36:52.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:68: test_fast_kill: killid=174990 2024-09-18T14:36:52.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 174990 -eq 0 ']' 2024-09-18T14:36:52.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:70: test_fast_kill: previd=174990 2024-09-18T14:36:52.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:72: test_fast_kill: kill -9 174990 2024-09-18T14:36:52.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:73: test_fast_kill: time_left=30 2024-09-18T14:36:52.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:74: test_fast_kill: down_osds=0 2024-09-18T14:36:52.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:76: test_fast_kill: '[' 30 -gt 0 ']' 2024-09-18T14:36:52.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:77: test_fast_kill: sleep 1 2024-09-18T14:36:53.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:78: test_fast_kill: time_left=29 2024-09-18T14:36:53.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:80: test_fast_kill: grep -m 1 -c -F ms_handle_refused td/osd-fast-mark-down/osd.0.log td/osd-fast-mark-down/osd.1.log td/osd-fast-mark-down/osd.2.log 2024-09-18T14:36:53.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:81: test_fast_kill: '[' 0 -ne 0 ']' 2024-09-18T14:36:53.728 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: ceph osd tree 2024-09-18T14:36:53.728 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: grep -c down 2024-09-18T14:36:54.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: down_osds=1 2024-09-18T14:36:54.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:86: test_fast_kill: '[' 1 -lt 1 ']' 2024-09-18T14:36:54.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:89: test_fast_kill: '[' 1 -gt 1 ']' 2024-09-18T14:36:54.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:93: test_fast_kill: break 2024-09-18T14:36:54.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:97: test_fast_kill: '[' 1 -lt 1 ']' 2024-09-18T14:36:54.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:66: test_fast_kill: for i in {1..2} 2024-09-18T14:36:54.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 174990 -eq 174990 ']' 2024-09-18T14:36:54.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:68: test_fast_kill: killid=176721 2024-09-18T14:36:54.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 176721 -eq 174990 ']' 2024-09-18T14:36:54.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:70: test_fast_kill: previd=176721 2024-09-18T14:36:54.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:72: test_fast_kill: kill -9 176721 2024-09-18T14:36:54.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:73: test_fast_kill: time_left=30 2024-09-18T14:36:54.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:74: test_fast_kill: down_osds=0 2024-09-18T14:36:54.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:76: test_fast_kill: '[' 30 -gt 0 ']' 2024-09-18T14:36:54.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:77: test_fast_kill: sleep 1 2024-09-18T14:36:55.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:78: test_fast_kill: time_left=29 2024-09-18T14:36:55.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:80: test_fast_kill: grep -m 1 -c -F ms_handle_refused td/osd-fast-mark-down/osd.0.log td/osd-fast-mark-down/osd.1.log td/osd-fast-mark-down/osd.2.log 2024-09-18T14:36:55.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:81: test_fast_kill: '[' 0 -ne 0 ']' 2024-09-18T14:36:55.114 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: ceph osd tree 2024-09-18T14:36:55.114 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: grep -c down 2024-09-18T14:36:55.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: down_osds=2 2024-09-18T14:36:55.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:86: test_fast_kill: '[' 2 -lt 2 ']' 2024-09-18T14:36:55.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:89: test_fast_kill: '[' 2 -gt 2 ']' 2024-09-18T14:36:55.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:93: test_fast_kill: break 2024-09-18T14:36:55.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:97: test_fast_kill: '[' 2 -lt 2 ']' 2024-09-18T14:36:55.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:103: test_fast_kill: pkill -SIGTERM rados 2024-09-18T14:36:55.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:104: test_fast_kill: teardown td/osd-fast-mark-down 2024-09-18T14:36:55.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2024-09-18T14:36:55.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:36:55.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2024-09-18T14:36:55.574 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:36:55.574 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:36:55.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:36:55.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:36:55.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:36:55.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:36:55.682 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:36:55.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:36:55.682 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:36:55.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:36:55.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:36:55.684 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:36:55.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:36:55.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:36:55.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:36:55.685 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:36:55.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:36:55.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:36:55.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2024-09-18T14:36:55.701 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:36:55.701 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:36:55.701 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173834 2024-09-18T14:36:55.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173834 2024-09-18T14:36:55.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:36:55.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:36:55.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:42: run: return 0 2024-09-18T14:36:55.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T14:36:55.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/osd-fast-mark-down 0 2024-09-18T14:36:55.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2024-09-18T14:36:55.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T14:36:55.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2024-09-18T14:36:55.704 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:36:55.704 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:36:55.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:36:55.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:36:55.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:36:55.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:36:55.705 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:36:55.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:36:55.706 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:36:55.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:36:55.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:36:55.707 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:36:55.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:36:55.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:36:55.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:36:55.709 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:36:55.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:36:55.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T14:36:55.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2024-09-18T14:36:55.711 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:36:55.712 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:36:55.712 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173834 2024-09-18T14:36:55.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173834 2024-09-18T14:36:55.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:36:55.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:36:55.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T14:36:55.714 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T14:36:55.714 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T14:36:55.779 INFO:tasks.workunit:Running workunit osd/osd-force-create-pg.sh... 2024-09-18T14:36:55.780 DEBUG:teuthology.orchestra.run.smithi047:workunit test osd/osd-force-create-pg.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh 2024-09-18T14:36:55.842 INFO:tasks.workunit.client.0.smithi047.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T14:36:55.848 INFO:tasks.workunit.client.0.smithi047.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T14:36:55.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T14:36:55.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T14:36:55.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T14:36:55.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T14:36:55.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T14:36:55.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T14:36:55.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T14:36:55.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T14:36:55.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/osd-force-create-pg 2024-09-18T14:36:55.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:5: run: local dir=td/osd-force-create-pg 2024-09-18T14:36:55.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:6: run: shift 2024-09-18T14:36:55.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:8: run: export CEPH_MON=127.0.0.1:7145 2024-09-18T14:36:55.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:8: run: CEPH_MON=127.0.0.1:7145 2024-09-18T14:36:55.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:9: run: export CEPH_ARGS 2024-09-18T14:36:55.849 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:10: run: uuidgen 2024-09-18T14:36:55.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:10: run: CEPH_ARGS+='--fsid=5a52c0c3-b4b9-4ac8-989f-3bde2adf854b --auth-supported=none ' 2024-09-18T14:36:55.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:11: run: CEPH_ARGS+='--mon-host=127.0.0.1:7145 ' 2024-09-18T14:36:55.850 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: set 2024-09-18T14:36:55.851 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-18T14:36:55.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: local funcs=TEST_reuse_id 2024-09-18T14:36:55.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:14: run: for func in $funcs 2024-09-18T14:36:55.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:15: run: setup td/osd-force-create-pg 2024-09-18T14:36:55.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-force-create-pg 2024-09-18T14:36:55.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-force-create-pg 2024-09-18T14:36:55.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2024-09-18T14:36:55.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:36:55.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2024-09-18T14:36:55.851 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:36:55.851 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:36:55.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:36:55.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:36:55.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:36:55.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:36:55.852 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:36:55.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:36:55.853 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:36:55.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:36:55.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:36:55.854 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:36:55.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:36:55.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:36:55.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:36:55.855 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:36:55.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:36:55.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:36:55.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2024-09-18T14:36:55.858 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:36:55.858 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:36:55.858 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.179618 2024-09-18T14:36:55.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.179618 2024-09-18T14:36:55.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:36:55.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:36:55.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-force-create-pg 2024-09-18T14:36:55.860 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T14:36:55.861 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:36:55.861 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.179618 2024-09-18T14:36:55.861 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.179618 2024-09-18T14:36:55.865 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T14:36:55.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T14:36:55.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T14:36:55.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T14:36:55.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-force-create-pg 1' TERM HUP INT 2024-09-18T14:36:55.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:16: run: TEST_reuse_id td/osd-force-create-pg 2024-09-18T14:36:55.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:22: TEST_reuse_id: local dir=td/osd-force-create-pg 2024-09-18T14:36:55.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:24: TEST_reuse_id: run_mon td/osd-force-create-pg a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T14:36:55.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-force-create-pg 2024-09-18T14:36:55.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T14:36:55.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T14:36:55.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T14:36:55.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-force-create-pg/a 2024-09-18T14:36:55.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-force-create-pg/a --run-dir=td/osd-force-create-pg --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T14:36:56.218 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T14:36:56.219 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:36:56.219 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:36:56.219 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:36:56.219 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:36:56.219 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.179618 2024-09-18T14:36:56.219 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.179618/$cluster-$name.asok' 2024-09-18T14:36:56.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-force-create-pg/a '--log-file=td/osd-force-create-pg/$name.log' '--admin-socket=/tmp/ceph-asok.179618/$cluster-$name.asok' --mon-cluster-log-file=td/osd-force-create-pg/log --run-dir=td/osd-force-create-pg '--pid-file=td/osd-force-create-pg/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T14:36:56.470 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T14:36:56.471 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T14:36:56.471 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:36:56.471 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:36:56.471 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T14:36:56.471 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T14:36:56.472 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:36:56.472 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:36:56.472 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:36:56.472 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:36:56.472 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:36:56.473 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.179618 2024-09-18T14:36:56.473 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.179618/ceph-mon.a.asok 2024-09-18T14:36:56.473 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:36:56.473 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.179618/ceph-mon.a.asok config get fsid 2024-09-18T14:36:56.556 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T14:36:56.556 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:36:56.557 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:36:56.557 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T14:36:56.557 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T14:36:56.557 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:36:56.557 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:36:56.557 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:36:56.557 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:36:56.557 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:36:56.557 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.179618 2024-09-18T14:36:56.557 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.179618/ceph-mon.a.asok 2024-09-18T14:36:56.557 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:36:56.557 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.179618/ceph-mon.a.asok config get mon_host 2024-09-18T14:36:56.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:25: TEST_reuse_id: run_mgr td/osd-force-create-pg x 2024-09-18T14:36:56.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-force-create-pg 2024-09-18T14:36:56.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T14:36:56.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T14:36:56.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T14:36:56.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-force-create-pg/x 2024-09-18T14:36:56.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T14:36:57.467 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T14:36:57.468 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:36:57.468 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:36:57.468 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:36:57.468 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:36:57.468 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.179618 2024-09-18T14:36:57.468 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.179618/$cluster-$name.asok' 2024-09-18T14:36:57.468 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:36:57.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-force-create-pg/x '--log-file=td/osd-force-create-pg/$name.log' '--admin-socket=/tmp/ceph-asok.179618/$cluster-$name.asok' --run-dir=td/osd-force-create-pg '--pid-file=td/osd-force-create-pg/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:36:57.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:26: TEST_reuse_id: run_osd td/osd-force-create-pg 0 2024-09-18T14:36:57.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2024-09-18T14:36:57.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:36:57.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T14:36:57.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:36:57.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/0 2024-09-18T14:36:57.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a52c0c3-b4b9-4ac8-989f-3bde2adf854b --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-09-18T14:36:57.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:36:57.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:36:57.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:36:57.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/0' 2024-09-18T14:36:57.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/0/journal' 2024-09-18T14:36:57.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:36:57.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:36:57.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-09-18T14:36:57.495 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:36:57.495 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:36:57.495 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:36:57.495 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:36:57.495 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:36:57.495 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.179618 2024-09-18T14:36:57.495 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.179618/$cluster-$name.asok' 2024-09-18T14:36:57.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.179618/$cluster-$name.asok' 2024-09-18T14:36:57.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:36:57.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:36:57.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:36:57.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-09-18T14:36:57.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-09-18T14:36:57.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:36:57.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:36:57.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:36:57.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:36:57.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:36:57.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:36:57.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-force-create-pg/0 2024-09-18T14:36:57.496 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:36:57.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=642de96c-78e4-40b9-aa86-e9e7f6687e72 2024-09-18T14:36:57.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 642de96c-78e4-40b9-aa86-e9e7f6687e72' 2024-09-18T14:36:57.497 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 642de96c-78e4-40b9-aa86-e9e7f6687e72 2024-09-18T14:36:57.498 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:36:57.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCJ5epm+WhCHhAAnBvMGWXkxBHv5arrt8/nww== 2024-09-18T14:36:57.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCJ5epm+WhCHhAAnBvMGWXkxBHv5arrt8/nww=="}' 2024-09-18T14:36:57.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 642de96c-78e4-40b9-aa86-e9e7f6687e72 -i td/osd-force-create-pg/0/new.json 2024-09-18T14:36:57.801 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:36:57.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-force-create-pg/0/new.json 2024-09-18T14:36:57.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=5a52c0c3-b4b9-4ac8-989f-3bde2adf854b --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.179618/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCJ5epm+WhCHhAAnBvMGWXkxBHv5arrt8/nww== --osd-uuid 642de96c-78e4-40b9-aa86-e9e7f6687e72 2024-09-18T14:36:57.835 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:36:57.835+0000 7fead9b35440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:36:57.838 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:36:57.838+0000 7fead9b35440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:36:57.838 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:36:57.839+0000 7fead9b35440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:36:57.839 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:36:57.839+0000 7fead9b35440 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2024-09-18T14:36:57.839 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:36:57.839+0000 7fead9b35440 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2024-09-18T14:36:57.839 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:36:57.839+0000 7fead9b35440 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2024-09-18T14:36:57.878 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:36:57.878+0000 7fead9b35440 -1 bluestore(td/osd-force-create-pg/0) _read_fsid unparsable uuid 2024-09-18T14:37:03.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-force-create-pg/0/keyring 2024-09-18T14:37:03.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:37:03.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T14:37:03.197 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T14:37:03.197 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-force-create-pg/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:37:03.705 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T14:37:03.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T14:37:03.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=5a52c0c3-b4b9-4ac8-989f-3bde2adf854b --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.179618/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:37:03.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:37:03.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:37:03.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:37:03.727 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:37:03.726+0000 7fb4e7c17440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:37:03.729 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:37:03.728+0000 7fb4e7c17440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:37:03.730 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:37:03.729+0000 7fb4e7c17440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:37:04.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T14:37:04.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:37:04.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:37:04.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:37:04.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:37:04.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:37:04.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:37:04.069 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:37:04.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:37:04.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:37:04.251 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:37:04.250+0000 7fb4e7c17440 -1 Falling back to public interface 2024-09-18T14:37:04.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:37:05.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:37:05.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:37:05.467 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:37:05.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:37:05.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:37:05.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:37:05.875 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:37:06.747 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:37:06.746+0000 7fb4e7c17440 -1 osd.0 0 log_to_monitors true 2024-09-18T14:37:06.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:37:06.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:37:06.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:37:06.877 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:37:06.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:37:06.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:37:07.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:37:08.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:37:08.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:37:08.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:37:08.350 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:37:08.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:37:08.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:37:08.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:37:09.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:37:09.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:37:09.773 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:37:09.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:37:09.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:37:09.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:37:10.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:37:11.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:37:11.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:37:11.151 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:37:11.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:37:11.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:37:11.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:37:11.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:37:12.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:37:12.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:37:12.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:37:12.607 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:37:12.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:37:12.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:37:13.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:37:14.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:37:14.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:37:14.014 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:37:14.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:37:14.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:37:14.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:37:14.470 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:37:15.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:37:15.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:37:15.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:37:15.472 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:37:15.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:37:15.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:37:15.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:37:16.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:37:16.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:37:16.851 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:37:16.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:37:16.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:37:16.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:37:17.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:37:18.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:37:18.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:37:18.227 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:37:18.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:37:18.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:37:18.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:37:18.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:37:19.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:37:19.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:37:19.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:37:19.873 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:37:19.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:37:19.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:37:20.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:37:21.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:37:21.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:37:21.543 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:37:21.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:37:21.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:37:21.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:37:22.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:37:23.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:37:23.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:37:23.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:37:23.291 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:37:23.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:37:23.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:37:23.666 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:37:24.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:37:24.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:37:24.668 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:37:24.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:37:24.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:37:24.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:37:25.432 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:37:26.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:37:26.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:37:26.434 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:37:26.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:37:26.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:37:26.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:37:27.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:37:28.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:37:28.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:37:28.109 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:37:28.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:37:28.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:37:28.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:37:28.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:37:29.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:37:29.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:37:29.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:37:29.682 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T14:37:29.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:37:29.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:37:30.087 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/180709,v1:127.0.0.1:6803/180709] [v2:127.0.0.1:6804/180709,v1:127.0.0.1:6805/180709] exists,up 642de96c-78e4-40b9-aa86-e9e7f6687e72 2024-09-18T14:37:30.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:37:30.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:37:30.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:37:30.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:27: TEST_reuse_id: run_osd td/osd-force-create-pg 1 2024-09-18T14:37:30.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2024-09-18T14:37:30.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:37:30.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T14:37:30.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:37:30.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/1 2024-09-18T14:37:30.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a52c0c3-b4b9-4ac8-989f-3bde2adf854b --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-09-18T14:37:30.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:37:30.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:37:30.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:37:30.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/1' 2024-09-18T14:37:30.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/1/journal' 2024-09-18T14:37:30.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:37:30.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:37:30.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-09-18T14:37:30.092 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:37:30.092 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:37:30.092 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:37:30.092 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:37:30.092 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:37:30.092 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.179618 2024-09-18T14:37:30.093 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.179618/$cluster-$name.asok' 2024-09-18T14:37:30.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.179618/$cluster-$name.asok' 2024-09-18T14:37:30.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:37:30.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:37:30.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:37:30.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-09-18T14:37:30.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-09-18T14:37:30.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:37:30.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:37:30.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:37:30.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:37:30.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:37:30.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:37:30.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-force-create-pg/1 2024-09-18T14:37:30.093 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:37:30.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=19018653-3b60-447d-87e1-f557a48e913f 2024-09-18T14:37:30.094 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 19018653-3b60-447d-87e1-f557a48e913f 2024-09-18T14:37:30.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 19018653-3b60-447d-87e1-f557a48e913f' 2024-09-18T14:37:30.094 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:37:30.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCq5epmj/0lBhAAMeKujKqfQhPpiWkpyMkbtQ== 2024-09-18T14:37:30.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCq5epmj/0lBhAAMeKujKqfQhPpiWkpyMkbtQ=="}' 2024-09-18T14:37:30.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 19018653-3b60-447d-87e1-f557a48e913f -i td/osd-force-create-pg/1/new.json 2024-09-18T14:37:30.616 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:37:30.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-force-create-pg/1/new.json 2024-09-18T14:37:30.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=5a52c0c3-b4b9-4ac8-989f-3bde2adf854b --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.179618/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCq5epmj/0lBhAAMeKujKqfQhPpiWkpyMkbtQ== --osd-uuid 19018653-3b60-447d-87e1-f557a48e913f 2024-09-18T14:37:30.650 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:37:30.650+0000 7f854a609440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:37:30.652 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:37:30.652+0000 7f854a609440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:37:30.653 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:37:30.653+0000 7f854a609440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:37:30.654 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:37:30.653+0000 7f854a609440 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2024-09-18T14:37:30.654 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:37:30.653+0000 7f854a609440 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2024-09-18T14:37:30.654 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:37:30.653+0000 7f854a609440 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2024-09-18T14:37:30.698 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:37:30.698+0000 7f854a609440 -1 bluestore(td/osd-force-create-pg/1) _read_fsid unparsable uuid 2024-09-18T14:37:36.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-force-create-pg/1/keyring 2024-09-18T14:37:36.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:37:36.871 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T14:37:36.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T14:37:36.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-force-create-pg/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:37:37.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T14:37:37.378 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T14:37:37.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=5a52c0c3-b4b9-4ac8-989f-3bde2adf854b --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.179618/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:37:37.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:37:37.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:37:37.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:37:37.398 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:37:37.397+0000 7fc9f2725440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:37:37.400 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:37:37.399+0000 7fc9f2725440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:37:37.401 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:37:37.401+0000 7fc9f2725440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:37:37.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T14:37:37.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:37:37.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:37:37.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:37:37.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:37:37.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:37:37.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:37:37.773 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:37:37.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:37:37.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:37:37.916 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:37:37.915+0000 7fc9f2725440 -1 Falling back to public interface 2024-09-18T14:37:38.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:37:39.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:37:39.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:37:39.138 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:37:39.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:37:39.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:37:39.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:37:39.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:37:40.410 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:37:40.409+0000 7fc9f2725440 -1 osd.1 0 log_to_monitors true 2024-09-18T14:37:40.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:37:40.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:37:40.564 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:37:40.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:37:40.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:37:40.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:37:41.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:37:42.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:37:42.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:37:42.150 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:37:42.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:37:42.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:37:42.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:37:42.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:37:43.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:37:43.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:37:43.567 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:37:43.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:37:43.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:37:43.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:37:43.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:37:44.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:37:44.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:37:44.998 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:37:44.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:37:44.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:37:44.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:37:45.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:37:46.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:37:46.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:37:46.374 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:37:46.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:37:46.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:37:46.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:37:46.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:37:47.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:37:47.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:37:47.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:37:47.759 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:37:47.760 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:37:47.760 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:37:48.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:37:49.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:37:49.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:37:49.245 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:37:49.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:37:49.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:37:49.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:37:49.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:37:50.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:37:50.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:37:50.784 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:37:50.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:37:50.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:37:50.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:37:51.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:37:52.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:37:52.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:37:52.282 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:37:52.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:37:52.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:37:52.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:37:52.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:37:53.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:37:53.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:37:53.835 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:37:53.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:37:53.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:37:53.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:37:54.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:37:55.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:37:55.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:37:55.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:37:55.537 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:37:55.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:37:55.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:37:56.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:37:57.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:37:57.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:37:57.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:37:57.234 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:37:57.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:37:57.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:37:57.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:37:58.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:37:58.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:37:58.627 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:37:58.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:37:58.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:37:58.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:37:59.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:38:00.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:38:00.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:38:00.289 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:38:00.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:38:00.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:38:00.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:38:01.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:38:02.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:38:02.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:38:02.046 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:38:02.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:38:02.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:38:02.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:38:02.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:38:03.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:38:03.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:38:03.837 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T14:38:03.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:38:03.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:38:03.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:38:04.212 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/182463,v1:127.0.0.1:6811/182463] [v2:127.0.0.1:6812/182463,v1:127.0.0.1:6813/182463] exists,up 19018653-3b60-447d-87e1-f557a48e913f 2024-09-18T14:38:04.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:38:04.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:38:04.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:38:04.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:28: TEST_reuse_id: run_osd td/osd-force-create-pg 2 2024-09-18T14:38:04.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2024-09-18T14:38:04.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:38:04.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T14:38:04.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:38:04.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/2 2024-09-18T14:38:04.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5a52c0c3-b4b9-4ac8-989f-3bde2adf854b --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-09-18T14:38:04.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:38:04.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:38:04.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:38:04.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/2' 2024-09-18T14:38:04.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/2/journal' 2024-09-18T14:38:04.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:38:04.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:38:04.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-09-18T14:38:04.217 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:38:04.217 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:38:04.217 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:38:04.217 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:38:04.217 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:38:04.217 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.179618 2024-09-18T14:38:04.217 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.179618/$cluster-$name.asok' 2024-09-18T14:38:04.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.179618/$cluster-$name.asok' 2024-09-18T14:38:04.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:38:04.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:38:04.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:38:04.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-09-18T14:38:04.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-09-18T14:38:04.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:38:04.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:38:04.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:38:04.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:38:04.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:38:04.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:38:04.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-force-create-pg/2 2024-09-18T14:38:04.218 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:38:04.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f13f0332-2fd5-4b62-bdbf-698fa31927f4 2024-09-18T14:38:04.219 INFO:tasks.workunit.client.0.smithi047.stdout:add osd2 f13f0332-2fd5-4b62-bdbf-698fa31927f4 2024-09-18T14:38:04.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f13f0332-2fd5-4b62-bdbf-698fa31927f4' 2024-09-18T14:38:04.220 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:38:04.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDM5epmXzGgDRAAIe2RZjJkiV5KXcZITsZb6g== 2024-09-18T14:38:04.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDM5epmXzGgDRAAIe2RZjJkiV5KXcZITsZb6g=="}' 2024-09-18T14:38:04.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f13f0332-2fd5-4b62-bdbf-698fa31927f4 -i td/osd-force-create-pg/2/new.json 2024-09-18T14:38:04.732 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:38:04.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-force-create-pg/2/new.json 2024-09-18T14:38:04.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=5a52c0c3-b4b9-4ac8-989f-3bde2adf854b --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.179618/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDM5epmXzGgDRAAIe2RZjJkiV5KXcZITsZb6g== --osd-uuid f13f0332-2fd5-4b62-bdbf-698fa31927f4 2024-09-18T14:38:04.768 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:38:04.767+0000 7f5367125440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:38:04.769 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:38:04.769+0000 7f5367125440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:38:04.770 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:38:04.770+0000 7f5367125440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:38:04.771 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:38:04.770+0000 7f5367125440 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2024-09-18T14:38:04.771 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:38:04.770+0000 7f5367125440 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2024-09-18T14:38:04.771 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:38:04.770+0000 7f5367125440 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2024-09-18T14:38:04.855 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:38:04.855+0000 7f5367125440 -1 bluestore(td/osd-force-create-pg/2) _read_fsid unparsable uuid 2024-09-18T14:38:10.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-force-create-pg/2/keyring 2024-09-18T14:38:10.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:38:10.832 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd2 key to auth repository 2024-09-18T14:38:10.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T14:38:10.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-force-create-pg/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:38:11.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T14:38:11.347 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T14:38:11.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=5a52c0c3-b4b9-4ac8-989f-3bde2adf854b --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.179618/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:38:11.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:38:11.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:38:11.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:38:11.368 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:38:11.367+0000 7f9663920440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:38:11.395 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:38:11.394+0000 7f9663920440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:38:11.397 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:38:11.396+0000 7f9663920440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:38:11.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T14:38:11.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:38:11.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T14:38:11.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:38:11.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:38:11.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:38:11.777 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:38:11.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:38:11.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:38:11.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:38:11.935 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:38:11.934+0000 7f9663920440 -1 Falling back to public interface 2024-09-18T14:38:12.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:38:13.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:38:13.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:38:13.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:38:13.179 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:38:13.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:38:13.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:38:13.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:38:14.426 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:38:14.425+0000 7f9663920440 -1 osd.2 0 log_to_monitors true 2024-09-18T14:38:14.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:38:14.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:38:14.596 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:38:14.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:38:14.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:38:14.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:38:15.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:38:16.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:38:16.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:38:16.063 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:38:16.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:38:16.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:38:16.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:38:16.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:38:17.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:38:17.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:38:17.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:38:17.475 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:38:17.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:38:17.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:38:17.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:38:18.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:38:18.953 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:38:18.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:38:18.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:38:18.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:38:18.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:38:19.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:38:20.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:38:20.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:38:20.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:38:20.505 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:38:20.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:38:20.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:38:20.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:38:21.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:38:21.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:38:21.895 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:38:21.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:38:21.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:38:21.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:38:22.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:38:23.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:38:23.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:38:23.303 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:38:23.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:38:23.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:38:23.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:38:23.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:38:24.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:38:24.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:38:24.733 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:38:24.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:38:24.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:38:24.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:38:25.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:38:26.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:38:26.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:38:26.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:38:26.167 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:38:26.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:38:26.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:38:26.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:38:27.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:38:27.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:38:27.711 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:38:27.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:38:27.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:38:27.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:38:28.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:38:29.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:38:29.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:38:29.536 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:38:29.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:38:29.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:38:29.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:38:30.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:38:31.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:38:31.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:38:31.292 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:38:31.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:38:31.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:38:31.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:38:31.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:38:32.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:38:32.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:38:32.745 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:38:32.746 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:38:32.746 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:38:32.746 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:38:33.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:38:34.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:38:34.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:38:34.565 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:38:34.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:38:34.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:38:34.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:38:35.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:38:36.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:38:36.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:38:36.314 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:38:36.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:38:36.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:38:36.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:38:36.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:38:37.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:38:37.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:38:37.920 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T14:38:37.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:38:37.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:38:37.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:38:38.295 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/184217,v1:127.0.0.1:6819/184217] [v2:127.0.0.1:6820/184217,v1:127.0.0.1:6821/184217] exists,up f13f0332-2fd5-4b62-bdbf-698fa31927f4 2024-09-18T14:38:38.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:38:38.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:38:38.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:38:38.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:30: TEST_reuse_id: ceph osd pool create foo 50 2024-09-18T14:38:38.786 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'foo' already exists 2024-09-18T14:38:38.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:31: TEST_reuse_id: wait_for_clean 2024-09-18T14:38:38.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:38:38.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:38:38.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:38:38.799 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:38:38.799 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:38:38.799 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:38:38.799 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:38:38.799 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:38:38.799 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:38:38.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T14:38:38.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:38:38.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:38:38.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:38:38.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:38:38.887 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:38:39.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:38:39.336 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T14:38:39.336 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T14:38:39.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:38:39.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:38:39.337 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:38:39.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836500 2024-09-18T14:38:39.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836500 2024-09-18T14:38:39.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500' 2024-09-18T14:38:39.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:38:39.599 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:38:39.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672971 2024-09-18T14:38:39.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672971 2024-09-18T14:38:39.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500 1-42949672971' 2024-09-18T14:38:39.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:38:39.822 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:38:39.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509445 2024-09-18T14:38:39.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509445 2024-09-18T14:38:39.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500 1-42949672971 2-64424509445' 2024-09-18T14:38:39.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:38:39.980 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836500 2024-09-18T14:38:39.980 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:38:39.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:38:39.981 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836500 2024-09-18T14:38:39.981 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:38:39.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836500 2024-09-18T14:38:39.982 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836500 2024-09-18T14:38:39.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-09-18T14:38:39.983 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:38:40.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836498 -lt 21474836500 2024-09-18T14:38:40.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:38:41.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:38:41.504 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:38:41.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836500 -lt 21474836500 2024-09-18T14:38:41.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:38:41.875 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672971 2024-09-18T14:38:41.875 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:38:41.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:38:41.877 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672971 2024-09-18T14:38:41.877 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:38:41.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672971 2024-09-18T14:38:41.878 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672971 2024-09-18T14:38:41.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-09-18T14:38:41.878 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:38:42.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672971 -lt 42949672971 2024-09-18T14:38:42.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:38:42.292 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509445 2024-09-18T14:38:42.292 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:38:42.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:38:42.294 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509445 2024-09-18T14:38:42.294 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:38:42.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509445 2024-09-18T14:38:42.294 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509445 2024-09-18T14:38:42.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-09-18T14:38:42.294 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:38:42.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509445 -lt 64424509445 2024-09-18T14:38:42.656 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:38:42.656 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:38:42.656 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:38:43.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 50 == 0 2024-09-18T14:38:43.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:38:43.149 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:38:43.149 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:38:43.150 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:38:43.150 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:38:43.150 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:38:43.150 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:38:43.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=13 2024-09-18T14:38:43.552 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:38:43.552 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:38:43.552 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:38:44.079 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 13 = 50 2024-09-18T14:38:44.080 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 13 '!=' -1 2024-09-18T14:38:44.080 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T14:38:44.080 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=13 2024-09-18T14:38:44.080 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:38:44.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:38:44.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:38:44.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:38:44.181 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:38:44.181 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:38:44.181 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:38:44.181 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:38:44.181 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:38:44.181 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:38:44.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=13 2024-09-18T14:38:44.546 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:38:44.546 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:38:44.546 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:38:45.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 13 = 50 2024-09-18T14:38:45.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 13 '!=' 13 2024-09-18T14:38:45.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:38:45.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:38:45.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:38:45.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:38:45.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:38:45.069 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:38:45.069 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:38:45.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T14:38:45.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T14:38:45.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-09-18T14:38:45.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:38:45.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-18T14:38:45.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:38:45.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:38:45.802 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:38:45.803 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:38:45.803 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:38:45.803 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:38:45.803 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:38:45.803 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:38:46.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=50 2024-09-18T14:38:46.169 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:38:46.169 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:38:46.169 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:38:46.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 50 = 50 2024-09-18T14:38:46.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:38:46.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:38:46.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:33: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.0 2024-09-18T14:38:46.661 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:38:46.661 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:38:46.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:38:46.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:38:46.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:38:46.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:38:46.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:34: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.1 2024-09-18T14:38:46.968 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:38:46.968 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:38:46.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:38:46.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:38:46.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:38:47.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:38:47.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:35: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.2 2024-09-18T14:38:47.275 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:38:47.275 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:38:47.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:38:47.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:38:47.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:38:47.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:38:47.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:36: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/0 --op remove --pgid 1.0 --force 2024-09-18T14:38:48.831 INFO:tasks.workunit.client.0.smithi047.stderr:PG '1.0' not found 2024-09-18T14:38:49.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:37: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/1 --op remove --pgid 1.0 --force 2024-09-18T14:38:50.386 INFO:tasks.workunit.client.0.smithi047.stdout: marking collection for removal 2024-09-18T14:38:50.386 INFO:tasks.workunit.client.0.smithi047.stdout:setting '_remove' omap key 2024-09-18T14:38:50.387 INFO:tasks.workunit.client.0.smithi047.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-09-18T14:38:50.460 INFO:tasks.workunit.client.0.smithi047.stdout:Remove successful 2024-09-18T14:38:50.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:38: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/2 --op remove --pgid 1.0 --force 2024-09-18T14:38:52.230 INFO:tasks.workunit.client.0.smithi047.stderr:PG '1.0' not found 2024-09-18T14:38:52.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:39: TEST_reuse_id: activate_osd td/osd-force-create-pg 0 2024-09-18T14:38:52.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-force-create-pg 2024-09-18T14:38:52.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:38:52.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T14:38:52.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:38:52.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-force-create-pg/0 2024-09-18T14:38:52.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5a52c0c3-b4b9-4ac8-989f-3bde2adf854b --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-09-18T14:38:52.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:38:52.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:38:52.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:38:52.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/0' 2024-09-18T14:38:52.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/0/journal' 2024-09-18T14:38:52.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:38:52.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:38:52.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-09-18T14:38:52.499 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:38:52.499 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:38:52.499 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:38:52.499 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:38:52.499 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:38:52.499 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.179618 2024-09-18T14:38:52.499 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.179618/$cluster-$name.asok' 2024-09-18T14:38:52.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.179618/$cluster-$name.asok' 2024-09-18T14:38:52.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:38:52.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-09-18T14:38:52.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-09-18T14:38:52.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:38:52.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:38:52.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:38:52.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:38:52.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:38:52.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:38:52.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-force-create-pg/0 2024-09-18T14:38:52.500 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T14:38:52.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T14:38:52.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=5a52c0c3-b4b9-4ac8-989f-3bde2adf854b --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.179618/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:38:52.501 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-force-create-pg/0/whoami 2024-09-18T14:38:52.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T14:38:52.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:38:52.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:38:52.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:38:52.522 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:38:52.521+0000 7efd9022d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:38:52.524 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:38:52.523+0000 7efd9022d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:38:52.526 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:38:52.525+0000 7efd9022d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:38:52.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T14:38:52.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:38:52.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:38:52.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:38:52.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:38:52.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:38:52.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:38:52.881 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:38:52.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:38:52.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:38:53.043 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:38:53.042+0000 7efd9022d440 -1 Falling back to public interface 2024-09-18T14:38:53.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:38:54.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:38:54.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:38:54.363 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:38:54.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:38:54.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:38:54.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:38:54.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:38:55.541 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:38:55.540+0000 7efd9022d440 -1 osd.0 22 log_to_monitors true 2024-09-18T14:38:55.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:38:55.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:38:55.774 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:38:55.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:38:55.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:38:55.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:38:56.312 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 27 up_thru 17 down_at 23 last_clean_interval [5,22) [v2:127.0.0.1:6802/186810,v1:127.0.0.1:6803/186810] [v2:127.0.0.1:6804/186810,v1:127.0.0.1:6805/186810] exists,up 642de96c-78e4-40b9-aa86-e9e7f6687e72 2024-09-18T14:38:56.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:38:56.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:38:56.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:38:56.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:40: TEST_reuse_id: activate_osd td/osd-force-create-pg 1 2024-09-18T14:38:56.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-force-create-pg 2024-09-18T14:38:56.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:38:56.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T14:38:56.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:38:56.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-force-create-pg/1 2024-09-18T14:38:56.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5a52c0c3-b4b9-4ac8-989f-3bde2adf854b --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-09-18T14:38:56.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:38:56.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:38:56.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:38:56.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/1' 2024-09-18T14:38:56.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/1/journal' 2024-09-18T14:38:56.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:38:56.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:38:56.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-09-18T14:38:56.317 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:38:56.317 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:38:56.317 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:38:56.317 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:38:56.317 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:38:56.317 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.179618 2024-09-18T14:38:56.318 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.179618/$cluster-$name.asok' 2024-09-18T14:38:56.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.179618/$cluster-$name.asok' 2024-09-18T14:38:56.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:38:56.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-09-18T14:38:56.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-09-18T14:38:56.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:38:56.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:38:56.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:38:56.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:38:56.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:38:56.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:38:56.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-force-create-pg/1 2024-09-18T14:38:56.319 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T14:38:56.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T14:38:56.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=5a52c0c3-b4b9-4ac8-989f-3bde2adf854b --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.179618/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:38:56.320 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-force-create-pg/1/whoami 2024-09-18T14:38:56.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T14:38:56.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:38:56.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:38:56.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:38:56.337 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:38:56.336+0000 7f1de257b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:38:56.368 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:38:56.367+0000 7f1de257b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:38:56.369 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:38:56.369+0000 7f1de257b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:38:56.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T14:38:56.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:38:56.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:38:56.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:38:56.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:38:56.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:38:56.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:38:56.689 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:38:56.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:38:56.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:38:56.891 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:38:56.890+0000 7f1de257b440 -1 Falling back to public interface 2024-09-18T14:38:57.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:38:58.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:38:58.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:38:58.118 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:38:58.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:38:58.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:38:58.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:38:58.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:38:59.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:38:59.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:38:59.496 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:38:59.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:38:59.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:38:59.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:38:59.497 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:38:59.497+0000 7f1de257b440 -1 osd.1 23 log_to_monitors true 2024-09-18T14:38:59.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:39:00.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:39:00.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:39:00.984 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:39:00.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:39:00.985 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:39:00.985 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:39:01.405 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 30 up_thru 30 down_at 24 last_clean_interval [10,23) [v2:127.0.0.1:6810/187363,v1:127.0.0.1:6811/187363] [v2:127.0.0.1:6812/187363,v1:127.0.0.1:6813/187363] exists,up 19018653-3b60-447d-87e1-f557a48e913f 2024-09-18T14:39:01.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:39:01.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:39:01.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:39:01.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:41: TEST_reuse_id: activate_osd td/osd-force-create-pg 2 2024-09-18T14:39:01.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-force-create-pg 2024-09-18T14:39:01.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:39:01.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T14:39:01.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:39:01.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-force-create-pg/2 2024-09-18T14:39:01.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5a52c0c3-b4b9-4ac8-989f-3bde2adf854b --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-09-18T14:39:01.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:39:01.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:39:01.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:39:01.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/2' 2024-09-18T14:39:01.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/2/journal' 2024-09-18T14:39:01.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:39:01.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:39:01.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-09-18T14:39:01.409 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:39:01.409 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:39:01.409 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:39:01.410 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:39:01.410 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:39:01.410 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.179618 2024-09-18T14:39:01.410 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.179618/$cluster-$name.asok' 2024-09-18T14:39:01.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.179618/$cluster-$name.asok' 2024-09-18T14:39:01.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:39:01.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-09-18T14:39:01.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-09-18T14:39:01.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:39:01.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:39:01.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:39:01.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:39:01.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:39:01.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:39:01.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-force-create-pg/2 2024-09-18T14:39:01.411 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T14:39:01.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T14:39:01.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=5a52c0c3-b4b9-4ac8-989f-3bde2adf854b --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.179618/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:39:01.412 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-force-create-pg/2/whoami 2024-09-18T14:39:01.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T14:39:01.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:39:01.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:39:01.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:39:01.428 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:39:01.427+0000 7fa2d3c22440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:39:01.430 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:39:01.429+0000 7fa2d3c22440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:39:01.431 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:39:01.430+0000 7fa2d3c22440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:39:01.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T14:39:01.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:39:01.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T14:39:01.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:39:01.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:39:01.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:39:01.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:39:01.837 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:39:01.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:39:01.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:39:01.945 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:39:01.944+0000 7fa2d3c22440 -1 Falling back to public interface 2024-09-18T14:39:02.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:39:03.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:39:03.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:39:03.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:39:03.203 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:39:03.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:39:03.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:39:03.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:39:04.453 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:39:04.451+0000 7fa2d3c22440 -1 osd.2 24 log_to_monitors true 2024-09-18T14:39:04.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:39:04.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:39:04.704 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:39:04.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:39:04.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:39:04.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:39:05.342 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 33 up_thru 33 down_at 25 last_clean_interval [15,24) [v2:127.0.0.1:6818/187941,v1:127.0.0.1:6819/187941] [v2:127.0.0.1:6820/187941,v1:127.0.0.1:6821/187941] exists,up f13f0332-2fd5-4b62-bdbf-698fa31927f4 2024-09-18T14:39:05.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:39:05.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:39:05.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:39:05.343 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:42: TEST_reuse_id: sleep 10 2024-09-18T14:39:15.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:43: TEST_reuse_id: ceph pg ls 2024-09-18T14:39:15.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:43: TEST_reuse_id: grep 1.0 2024-09-18T14:39:15.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:43: TEST_reuse_id: grep stale 2024-09-18T14:39:15.715 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 0 0 0 0 0 0 0 0 0 stale+active+clean 35s 0'0 21:10 [1]p1 [1]p1 2024-09-18T14:38:38.602378+0000 2024-09-18T14:38:38.602378+0000 0 periodic scrub scheduled @ 2024-09-19T18:51:20.737707+0000 2024-09-18T14:39:15.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:45: TEST_reuse_id: ceph osd force-create-pg 1.0 --yes-i-really-mean-it 2024-09-18T14:39:16.356 INFO:tasks.workunit.client.0.smithi047.stderr:pg 1.0 already creating 2024-09-18T14:39:16.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:46: TEST_reuse_id: wait_for_clean 2024-09-18T14:39:16.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:39:16.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:39:16.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:39:16.368 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:39:16.368 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:39:16.368 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:39:16.368 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:39:16.368 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:39:16.368 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:39:16.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T14:39:16.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:39:16.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:39:16.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:39:16.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:39:16.456 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:39:16.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:39:16.852 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T14:39:16.852 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T14:39:16.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:39:16.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:39:16.852 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:39:16.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=115964116999 2024-09-18T14:39:16.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 115964116999 2024-09-18T14:39:16.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-115964116999' 2024-09-18T14:39:16.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:39:16.958 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:39:17.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018886 2024-09-18T14:39:17.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018886 2024-09-18T14:39:17.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-115964116999 1-128849018886' 2024-09-18T14:39:17.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:39:17.086 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:39:17.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=141733920773 2024-09-18T14:39:17.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 141733920773 2024-09-18T14:39:17.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-115964116999 1-128849018886 2-141733920773' 2024-09-18T14:39:17.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:39:17.193 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-115964116999 2024-09-18T14:39:17.193 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:39:17.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:39:17.194 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-115964116999 2024-09-18T14:39:17.194 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:39:17.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=115964116999 2024-09-18T14:39:17.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 115964116999' 2024-09-18T14:39:17.195 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 115964116999 2024-09-18T14:39:17.195 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:39:17.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 115964116999 -lt 115964116999 2024-09-18T14:39:17.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:39:17.597 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-128849018886 2024-09-18T14:39:17.597 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:39:17.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:39:17.598 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-128849018886 2024-09-18T14:39:17.599 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:39:17.599 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018886 2024-09-18T14:39:17.600 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 128849018886 2024-09-18T14:39:17.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 128849018886' 2024-09-18T14:39:17.600 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:39:17.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018886 -lt 128849018886 2024-09-18T14:39:17.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:39:17.960 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-141733920773 2024-09-18T14:39:17.960 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:39:17.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:39:17.962 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-141733920773 2024-09-18T14:39:17.962 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:39:17.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=141733920773 2024-09-18T14:39:17.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 141733920773' 2024-09-18T14:39:17.963 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 141733920773 2024-09-18T14:39:17.963 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:39:18.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 141733920772 -lt 141733920773 2024-09-18T14:39:18.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:39:19.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:39:19.366 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:39:19.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 141733920773 -lt 141733920773 2024-09-18T14:39:19.769 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:39:19.769 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:39:19.769 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:39:20.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 50 == 0 2024-09-18T14:39:20.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:39:20.261 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:39:20.261 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:39:20.261 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:39:20.261 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:39:20.261 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:39:20.261 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:39:20.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=50 2024-09-18T14:39:20.656 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:39:20.656 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:39:20.657 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:39:21.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 50 = 50 2024-09-18T14:39:21.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:39:21.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:39:21.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:17: run: teardown td/osd-force-create-pg 2024-09-18T14:39:21.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2024-09-18T14:39:21.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:39:21.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2024-09-18T14:39:21.151 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:39:21.151 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:39:21.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:39:21.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:39:21.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:39:21.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:39:21.258 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:39:21.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:39:21.258 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:39:21.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:39:21.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:39:21.260 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:39:21.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:39:21.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:39:21.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:39:21.261 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:39:21.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:39:21.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:39:21.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2024-09-18T14:39:21.279 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:39:21.279 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:39:21.279 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.179618 2024-09-18T14:39:21.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.179618 2024-09-18T14:39:21.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:39:21.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:39:21.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T14:39:21.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/osd-force-create-pg 0 2024-09-18T14:39:21.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2024-09-18T14:39:21.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T14:39:21.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2024-09-18T14:39:21.281 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:39:21.281 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:39:21.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:39:21.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:39:21.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:39:21.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:39:21.282 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:39:21.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:39:21.284 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:39:21.285 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:39:21.285 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:39:21.285 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:39:21.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:39:21.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:39:21.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:39:21.286 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:39:21.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:39:21.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T14:39:21.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2024-09-18T14:39:21.289 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:39:21.289 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:39:21.289 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.179618 2024-09-18T14:39:21.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.179618 2024-09-18T14:39:21.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:39:21.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:39:21.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T14:39:21.291 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T14:39:21.292 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T14:39:21.356 INFO:tasks.workunit:Running workunit osd/osd-markdown.sh... 2024-09-18T14:39:21.356 DEBUG:teuthology.orchestra.run.smithi047:workunit test osd/osd-markdown.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh 2024-09-18T14:39:21.419 INFO:tasks.workunit.client.0.smithi047.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T14:39:21.425 INFO:tasks.workunit.client.0.smithi047.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T14:39:21.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T14:39:21.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T14:39:21.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T14:39:21.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T14:39:21.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T14:39:21.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T14:39:21.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T14:39:21.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T14:39:21.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/osd-markdown 2024-09-18T14:39:21.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:22: run: local dir=td/osd-markdown 2024-09-18T14:39:21.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:23: run: shift 2024-09-18T14:39:21.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:25: run: export CEPH_MON=127.0.0.1:7108 2024-09-18T14:39:21.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:25: run: CEPH_MON=127.0.0.1:7108 2024-09-18T14:39:21.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:26: run: export CEPH_ARGS 2024-09-18T14:39:21.426 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:27: run: uuidgen 2024-09-18T14:39:21.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:27: run: CEPH_ARGS+='--fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none ' 2024-09-18T14:39:21.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7108 ' 2024-09-18T14:39:21.427 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: set 2024-09-18T14:39:21.427 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-18T14:39:21.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: local 'funcs=TEST_markdown_boot 2024-09-18T14:39:21.428 INFO:tasks.workunit.client.0.smithi047.stderr:TEST_markdown_boot_exceed_time 2024-09-18T14:39:21.428 INFO:tasks.workunit.client.0.smithi047.stderr:TEST_markdown_exceed_maxdown_count 2024-09-18T14:39:21.428 INFO:tasks.workunit.client.0.smithi047.stderr:TEST_osd_stop' 2024-09-18T14:39:21.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2024-09-18T14:39:21.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2024-09-18T14:39:21.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2024-09-18T14:39:21.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2024-09-18T14:39:21.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-09-18T14:39:21.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:39:21.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-09-18T14:39:21.429 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:39:21.429 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:39:21.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:39:21.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:39:21.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:39:21.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:39:21.431 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:39:21.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:39:21.431 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:39:21.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:39:21.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:39:21.431 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:39:21.432 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:39:21.432 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:39:21.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:39:21.433 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:39:21.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:39:21.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:39:21.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-09-18T14:39:21.435 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:39:21.436 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:39:21.436 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:39:21.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.188945 2024-09-18T14:39:21.437 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:39:21.437 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:39:21.437 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2024-09-18T14:39:21.438 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T14:39:21.438 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:39:21.438 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:39:21.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.188945 2024-09-18T14:39:21.440 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T14:39:21.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T14:39:21.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T14:39:21.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T14:39:21.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2024-09-18T14:39:21.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_boot td/osd-markdown 2024-09-18T14:39:21.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:82: TEST_markdown_boot: local dir=td/osd-markdown 2024-09-18T14:39:21.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:84: TEST_markdown_boot: run_mon td/osd-markdown a 2024-09-18T14:39:21.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2024-09-18T14:39:21.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T14:39:21.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T14:39:21.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T14:39:21.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2024-09-18T14:39:21.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2024-09-18T14:39:21.826 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T14:39:21.826 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:39:21.826 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:39:21.826 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:39:21.826 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:39:21.826 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:39:21.826 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188945/$cluster-$name.asok' 2024-09-18T14:39:21.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T14:39:22.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T14:39:22.079 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T14:39:22.080 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:39:22.080 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:39:22.080 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T14:39:22.080 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T14:39:22.080 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:39:22.081 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:39:22.081 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:39:22.081 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:39:22.081 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:39:22.081 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:39:22.082 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.188945/ceph-mon.a.asok 2024-09-18T14:39:22.082 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:39:22.082 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.188945/ceph-mon.a.asok config get fsid 2024-09-18T14:39:22.162 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T14:39:22.162 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:39:22.162 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:39:22.162 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T14:39:22.162 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T14:39:22.162 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:39:22.162 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:39:22.162 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:39:22.162 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:39:22.162 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:39:22.162 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:39:22.163 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.188945/ceph-mon.a.asok 2024-09-18T14:39:22.163 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:39:22.163 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.188945/ceph-mon.a.asok config get mon_host 2024-09-18T14:39:22.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:85: TEST_markdown_boot: run_mgr td/osd-markdown x 2024-09-18T14:39:22.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2024-09-18T14:39:22.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T14:39:22.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T14:39:22.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T14:39:22.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2024-09-18T14:39:22.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T14:39:23.083 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T14:39:23.084 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:39:23.084 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:39:23.084 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:39:23.084 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:39:23.084 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:39:23.084 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188945/$cluster-$name.asok' 2024-09-18T14:39:23.084 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:39:23.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:39:23.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:86: TEST_markdown_boot: run_osd td/osd-markdown 0 2024-09-18T14:39:23.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-18T14:39:23.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:39:23.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T14:39:23.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:39:23.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2024-09-18T14:39:23.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-18T14:39:23.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:39:23.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:39:23.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:39:23.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2024-09-18T14:39:23.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2024-09-18T14:39:23.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:39:23.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:39:23.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-18T14:39:23.112 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:39:23.112 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:39:23.112 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:39:23.112 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:39:23.112 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:39:23.112 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:39:23.112 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188945/$cluster-$name.asok' 2024-09-18T14:39:23.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' 2024-09-18T14:39:23.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:39:23.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:39:23.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:39:23.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-18T14:39:23.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-18T14:39:23.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:39:23.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:39:23.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:39:23.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:39:23.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:39:23.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:39:23.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2024-09-18T14:39:23.113 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:39:23.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0770ea77-5902-4a9e-9b34-12816c1976f9 2024-09-18T14:39:23.113 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 0770ea77-5902-4a9e-9b34-12816c1976f9 2024-09-18T14:39:23.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0770ea77-5902-4a9e-9b34-12816c1976f9' 2024-09-18T14:39:23.114 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:39:23.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAb5upmBT9WBxAAieAZNHW3y0AU4mpxT0UGyg== 2024-09-18T14:39:23.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAb5upmBT9WBxAAieAZNHW3y0AU4mpxT0UGyg=="}' 2024-09-18T14:39:23.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0770ea77-5902-4a9e-9b34-12816c1976f9 -i td/osd-markdown/0/new.json 2024-09-18T14:39:23.417 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:39:23.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2024-09-18T14:39:23.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAb5upmBT9WBxAAieAZNHW3y0AU4mpxT0UGyg== --osd-uuid 0770ea77-5902-4a9e-9b34-12816c1976f9 2024-09-18T14:39:23.451 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:39:23.449+0000 7fc29ff23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:39:23.462 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:39:23.461+0000 7fc29ff23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:39:23.464 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:39:23.463+0000 7fc29ff23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:39:23.464 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:39:23.463+0000 7fc29ff23440 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-18T14:39:23.464 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:39:23.463+0000 7fc29ff23440 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-18T14:39:23.464 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:39:23.463+0000 7fc29ff23440 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-18T14:39:23.502 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:39:23.501+0000 7fc29ff23440 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2024-09-18T14:39:28.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2024-09-18T14:39:28.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:39:28.867 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T14:39:28.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T14:39:28.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:39:29.379 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T14:39:29.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T14:39:29.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:39:29.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:39:29.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:39:29.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:39:29.401 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:39:29.400+0000 7fe8a3c13440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:39:29.403 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:39:29.402+0000 7fe8a3c13440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:39:29.404 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:39:29.403+0000 7fe8a3c13440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:39:29.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T14:39:29.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:39:29.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:39:29.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:39:29.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:39:29.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:39:29.745 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:39:29.746 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:39:29.746 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:39:29.746 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:39:29.923 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:39:29.922+0000 7fe8a3c13440 -1 Falling back to public interface 2024-09-18T14:39:30.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:39:31.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:39:31.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:39:31.150 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:39:31.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:39:31.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:39:31.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:39:31.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:39:32.405 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:39:32.404+0000 7fe8a3c13440 -1 osd.0 0 log_to_monitors true 2024-09-18T14:39:32.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:39:32.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:39:32.568 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:39:32.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:39:32.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:39:32.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:39:33.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:39:34.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:39:34.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:39:34.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:39:34.019 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:39:34.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:39:34.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:39:34.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:39:35.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:39:35.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:39:35.429 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:39:35.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:39:35.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:39:35.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:39:35.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:39:36.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:39:36.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:39:36.818 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:39:36.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:39:36.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:39:36.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:39:37.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:39:38.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:39:38.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:39:38.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:39:38.291 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:39:38.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:39:38.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:39:38.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:39:39.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:39:39.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:39:39.697 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:39:39.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:39:39.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:39:39.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:39:40.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:39:41.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:39:41.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:39:41.161 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:39:41.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:39:41.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:39:41.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:39:41.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:39:42.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:39:42.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:39:42.536 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:39:42.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:39:42.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:39:42.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:39:42.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:39:43.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:39:43.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:39:43.939 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:39:43.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:39:43.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:39:43.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:39:44.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:39:45.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:39:45.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:39:45.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:39:45.626 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:39:45.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:39:45.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:39:46.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:39:47.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:39:47.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:39:47.100 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:39:47.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:39:47.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:39:47.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:39:47.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:39:48.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:39:48.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:39:48.870 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:39:48.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:39:48.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:39:48.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:39:49.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:39:50.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:39:50.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:39:50.313 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:39:50.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:39:50.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:39:50.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:39:50.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:39:51.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:39:51.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:39:51.978 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:39:51.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:39:51.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:39:51.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:39:52.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:39:53.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:39:53.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:39:53.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:39:53.740 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:39:53.740 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:39:53.740 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:39:54.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:39:55.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:39:55.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:39:55.315 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T14:39:55.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:39:55.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:39:55.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:39:55.711 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/190036,v1:127.0.0.1:6803/190036] [v2:127.0.0.1:6804/190036,v1:127.0.0.1:6805/190036] exists,up 0770ea77-5902-4a9e-9b34-12816c1976f9 2024-09-18T14:39:55.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:39:55.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:39:55.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:39:55.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:87: TEST_markdown_boot: run_osd td/osd-markdown 1 2024-09-18T14:39:55.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-18T14:39:55.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:39:55.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T14:39:55.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:39:55.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2024-09-18T14:39:55.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-18T14:39:55.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:39:55.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:39:55.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:39:55.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2024-09-18T14:39:55.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2024-09-18T14:39:55.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:39:55.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:39:55.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-18T14:39:55.717 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:39:55.717 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:39:55.717 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:39:55.717 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:39:55.717 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:39:55.717 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:39:55.717 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188945/$cluster-$name.asok' 2024-09-18T14:39:55.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' 2024-09-18T14:39:55.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:39:55.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:39:55.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:39:55.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-18T14:39:55.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-18T14:39:55.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:39:55.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:39:55.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:39:55.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:39:55.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:39:55.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:39:55.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2024-09-18T14:39:55.718 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:39:55.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=29ec6c94-b1b5-4ff2-9fee-dc7860ac55ed 2024-09-18T14:39:55.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 29ec6c94-b1b5-4ff2-9fee-dc7860ac55ed' 2024-09-18T14:39:55.719 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 29ec6c94-b1b5-4ff2-9fee-dc7860ac55ed 2024-09-18T14:39:55.719 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:39:55.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA75upmf6JeKxAADMLFUgJzHhZqQsxCDRg1JA== 2024-09-18T14:39:55.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA75upmf6JeKxAADMLFUgJzHhZqQsxCDRg1JA=="}' 2024-09-18T14:39:55.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 29ec6c94-b1b5-4ff2-9fee-dc7860ac55ed -i td/osd-markdown/1/new.json 2024-09-18T14:39:56.188 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:39:56.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2024-09-18T14:39:56.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA75upmf6JeKxAADMLFUgJzHhZqQsxCDRg1JA== --osd-uuid 29ec6c94-b1b5-4ff2-9fee-dc7860ac55ed 2024-09-18T14:39:56.222 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:39:56.221+0000 7f16d4a0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:39:56.224 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:39:56.223+0000 7f16d4a0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:39:56.225 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:39:56.224+0000 7f16d4a0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:39:56.225 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:39:56.224+0000 7f16d4a0b440 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-18T14:39:56.225 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:39:56.224+0000 7f16d4a0b440 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-18T14:39:56.225 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:39:56.224+0000 7f16d4a0b440 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-18T14:39:56.278 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:39:56.277+0000 7f16d4a0b440 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2024-09-18T14:40:01.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2024-09-18T14:40:01.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:40:01.928 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T14:40:01.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T14:40:01.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:40:02.438 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T14:40:02.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T14:40:02.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:40:02.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:40:02.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:40:02.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:40:02.459 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:40:02.458+0000 7f82f3e0e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:40:02.485 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:40:02.484+0000 7f82f3e0e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:40:02.486 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:40:02.485+0000 7f82f3e0e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:40:02.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T14:40:02.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:40:02.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:40:02.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:40:02.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:40:02.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:40:02.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:40:02.906 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:40:02.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:40:02.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:40:03.013 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:40:03.012+0000 7f82f3e0e440 -1 Falling back to public interface 2024-09-18T14:40:03.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:40:04.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:40:04.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:40:04.312 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:40:04.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:40:04.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:40:04.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:40:04.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:40:05.633 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:40:05.632+0000 7f82f3e0e440 -1 osd.1 0 log_to_monitors true 2024-09-18T14:40:05.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:40:05.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:40:05.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:40:05.689 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:40:05.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:40:05.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:40:06.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:40:07.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:40:07.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:40:07.260 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:40:07.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:40:07.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:40:07.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:40:07.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:40:08.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:40:08.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:40:08.733 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:40:08.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:40:08.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:40:08.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:40:09.135 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:40:10.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:40:10.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:40:10.136 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:40:10.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:40:10.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:40:10.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:40:10.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:40:11.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:40:11.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:40:11.618 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:40:11.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:40:11.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:40:11.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:40:12.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:40:13.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:40:13.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:40:13.030 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:40:13.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:40:13.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:40:13.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:40:13.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:40:14.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:40:14.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:40:14.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:40:14.531 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:40:14.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:40:14.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:40:14.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:40:15.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:40:15.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:40:15.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:40:15.906 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:40:15.907 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:40:15.907 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:40:16.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:40:17.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:40:17.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:40:17.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:40:17.281 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:40:17.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:40:17.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:40:18.155 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:40:19.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:40:19.157 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:40:19.157 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:40:19.157 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:40:19.157 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:40:19.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:40:20.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:40:21.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:40:21.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:40:21.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:40:21.220 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:40:21.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:40:21.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:40:21.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:40:22.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:40:22.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:40:22.690 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:40:22.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:40:22.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:40:22.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:40:23.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:40:24.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:40:24.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:40:24.148 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:40:24.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:40:24.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:40:24.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:40:24.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:40:25.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:40:25.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:40:25.842 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:40:25.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:40:25.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:40:25.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:40:26.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:40:27.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:40:27.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:40:27.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:40:27.823 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:40:27.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:40:27.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:40:28.205 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/191790,v1:127.0.0.1:6811/191790] [v2:127.0.0.1:6812/191790,v1:127.0.0.1:6813/191790] exists,up 29ec6c94-b1b5-4ff2-9fee-dc7860ac55ed 2024-09-18T14:40:28.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:40:28.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:40:28.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:40:28.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:88: TEST_markdown_boot: run_osd td/osd-markdown 2 2024-09-18T14:40:28.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-18T14:40:28.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:40:28.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T14:40:28.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:40:28.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2024-09-18T14:40:28.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-18T14:40:28.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:40:28.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:40:28.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:40:28.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2024-09-18T14:40:28.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2024-09-18T14:40:28.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:40:28.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:40:28.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-18T14:40:28.210 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:40:28.210 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:40:28.210 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:40:28.210 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:40:28.210 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:40:28.210 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:40:28.210 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188945/$cluster-$name.asok' 2024-09-18T14:40:28.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' 2024-09-18T14:40:28.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:40:28.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:40:28.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:40:28.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-18T14:40:28.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-18T14:40:28.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:40:28.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:40:28.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:40:28.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:40:28.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:40:28.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:40:28.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2024-09-18T14:40:28.211 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:40:28.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ab81f389-11d2-44a4-abd6-0005e1f3fb47 2024-09-18T14:40:28.211 INFO:tasks.workunit.client.0.smithi047.stdout:add osd2 ab81f389-11d2-44a4-abd6-0005e1f3fb47 2024-09-18T14:40:28.212 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 ab81f389-11d2-44a4-abd6-0005e1f3fb47' 2024-09-18T14:40:28.212 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:40:28.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBc5upm3OAxDRAAc73jnHnrT5DWQ2qnicE+YQ== 2024-09-18T14:40:28.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBc5upm3OAxDRAAc73jnHnrT5DWQ2qnicE+YQ=="}' 2024-09-18T14:40:28.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ab81f389-11d2-44a4-abd6-0005e1f3fb47 -i td/osd-markdown/2/new.json 2024-09-18T14:40:28.783 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:40:28.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2024-09-18T14:40:28.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBc5upm3OAxDRAAc73jnHnrT5DWQ2qnicE+YQ== --osd-uuid ab81f389-11d2-44a4-abd6-0005e1f3fb47 2024-09-18T14:40:28.818 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:40:28.817+0000 7fd1711fe440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:40:28.833 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:40:28.832+0000 7fd1711fe440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:40:28.834 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:40:28.833+0000 7fd1711fe440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:40:28.834 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:40:28.834+0000 7fd1711fe440 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-18T14:40:28.835 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:40:28.834+0000 7fd1711fe440 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-18T14:40:28.835 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:40:28.834+0000 7fd1711fe440 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-18T14:40:28.904 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:40:28.903+0000 7fd1711fe440 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2024-09-18T14:40:34.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2024-09-18T14:40:34.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:40:34.714 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd2 key to auth repository 2024-09-18T14:40:34.714 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T14:40:34.714 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:40:35.228 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T14:40:35.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T14:40:35.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:40:35.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:40:35.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:40:35.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:40:35.249 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:40:35.248+0000 7f2598c6e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:40:35.281 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:40:35.280+0000 7f2598c6e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:40:35.282 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:40:35.281+0000 7f2598c6e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:40:35.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T14:40:35.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:40:35.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T14:40:35.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:40:35.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:40:35.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:40:35.629 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:40:35.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:40:35.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:40:35.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:40:35.812 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:40:35.811+0000 7f2598c6e440 -1 Falling back to public interface 2024-09-18T14:40:36.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:40:37.022 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:40:37.022 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:40:37.022 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:40:37.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:40:37.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:40:37.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:40:37.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:40:38.271 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:40:38.270+0000 7f2598c6e440 -1 osd.2 0 log_to_monitors true 2024-09-18T14:40:38.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:40:38.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:40:38.425 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:40:38.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:40:38.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:40:38.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:40:38.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:40:39.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:40:39.967 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:40:39.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:40:39.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:40:39.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:40:39.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:40:40.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:40:41.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:40:41.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:40:41.344 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:40:41.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:40:41.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:40:41.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:40:41.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:40:42.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:40:42.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:40:42.805 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:40:42.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:40:42.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:40:42.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:40:43.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:40:44.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:40:44.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:40:44.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:40:44.260 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:40:44.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:40:44.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:40:44.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:40:45.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:40:45.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:40:45.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:40:45.641 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:40:45.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:40:45.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:40:46.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:40:47.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:40:47.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:40:47.131 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:40:47.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:40:47.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:40:47.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:40:47.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:40:48.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:40:48.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:40:48.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:40:48.558 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:40:48.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:40:48.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:40:49.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:40:50.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:40:50.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:40:50.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:40:50.088 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:40:50.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:40:50.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:40:50.599 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:40:51.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:40:51.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:40:51.601 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:40:51.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:40:51.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:40:51.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:40:52.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:40:53.419 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:40:53.420 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:40:53.420 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:40:53.420 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:40:53.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:40:53.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:40:54.483 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:40:55.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:40:55.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:40:55.485 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:40:55.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:40:55.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:40:55.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:40:56.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:40:57.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:40:57.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:40:57.186 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:40:57.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:40:57.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:40:57.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:40:57.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:40:58.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:40:58.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:40:58.934 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:40:58.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:40:58.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:40:58.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:40:59.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:41:00.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:41:00.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:41:00.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:41:00.703 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:41:00.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:41:00.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:41:01.129 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/193521,v1:127.0.0.1:6819/193521] [v2:127.0.0.1:6820/193521,v1:127.0.0.1:6821/193521] exists,up ab81f389-11d2-44a4-abd6-0005e1f3fb47 2024-09-18T14:41:01.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:41:01.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:41:01.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:41:01.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:90: TEST_markdown_boot: create_rbd_pool 2024-09-18T14:41:01.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T14:41:01.562 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'rbd' does not exist 2024-09-18T14:41:01.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T14:41:01.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T14:41:02.064 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'rbd' already exists 2024-09-18T14:41:02.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T14:41:03.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T14:41:05.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:93: TEST_markdown_boot: local count=3 2024-09-18T14:41:05.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:94: TEST_markdown_boot: local sleeptime=10 2024-09-18T14:41:05.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:95: TEST_markdown_boot: local period=120 2024-09-18T14:41:05.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:96: TEST_markdown_boot: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2024-09-18T14:41:05.221 INFO:tasks.workunit.client.0.smithi047.stdout:{} 2024-09-18T14:41:05.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:97: TEST_markdown_boot: ceph tell osd.0 injectargs '--osd_max_markdown_period 120' 2024-09-18T14:41:05.330 INFO:tasks.workunit.client.0.smithi047.stdout:{} 2024-09-18T14:41:05.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:99: TEST_markdown_boot: markdown_N_impl 3 120 10 2024-09-18T14:41:05.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=3 2024-09-18T14:41:05.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=120 2024-09-18T14:41:05.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2024-09-18T14:41:05.340 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 3 2024-09-18T14:41:05.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-09-18T14:41:05.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-09-18T14:41:05.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-09-18T14:41:05.797 INFO:tasks.workunit.client.0.smithi047.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-18T14:41:05.797 INFO:tasks.workunit.client.0.smithi047.stdout:-1 0.29306 root default 2024-09-18T14:41:05.797 INFO:tasks.workunit.client.0.smithi047.stdout:-3 0.29306 host smithi047 2024-09-18T14:41:05.797 INFO:tasks.workunit.client.0.smithi047.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-18T14:41:05.797 INFO:tasks.workunit.client.0.smithi047.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-09-18T14:41:05.797 INFO:tasks.workunit.client.0.smithi047.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-09-18T14:41:05.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-09-18T14:41:05.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-09-18T14:41:05.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-09-18T14:41:06.213 INFO:tasks.workunit.client.0.smithi047.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-18T14:41:06.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-09-18T14:41:06.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-09-18T14:41:06.604 INFO:tasks.workunit.client.0.smithi047.stderr:marked down osd.0. 2024-09-18T14:41:06.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-09-18T14:41:16.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-09-18T14:41:16.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-09-18T14:41:16.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-09-18T14:41:17.089 INFO:tasks.workunit.client.0.smithi047.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-18T14:41:17.089 INFO:tasks.workunit.client.0.smithi047.stdout:-1 0.29306 root default 2024-09-18T14:41:17.089 INFO:tasks.workunit.client.0.smithi047.stdout:-3 0.29306 host smithi047 2024-09-18T14:41:17.089 INFO:tasks.workunit.client.0.smithi047.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-18T14:41:17.090 INFO:tasks.workunit.client.0.smithi047.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-09-18T14:41:17.090 INFO:tasks.workunit.client.0.smithi047.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-09-18T14:41:17.096 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-09-18T14:41:17.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-09-18T14:41:17.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-09-18T14:41:17.458 INFO:tasks.workunit.client.0.smithi047.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-18T14:41:17.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-09-18T14:41:17.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-09-18T14:41:17.852 INFO:tasks.workunit.client.0.smithi047.stderr:marked down osd.0. 2024-09-18T14:41:17.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-09-18T14:41:27.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-09-18T14:41:27.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-09-18T14:41:27.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-09-18T14:41:28.339 INFO:tasks.workunit.client.0.smithi047.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-18T14:41:28.339 INFO:tasks.workunit.client.0.smithi047.stdout:-1 0.29306 root default 2024-09-18T14:41:28.339 INFO:tasks.workunit.client.0.smithi047.stdout:-3 0.29306 host smithi047 2024-09-18T14:41:28.339 INFO:tasks.workunit.client.0.smithi047.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-18T14:41:28.339 INFO:tasks.workunit.client.0.smithi047.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-09-18T14:41:28.339 INFO:tasks.workunit.client.0.smithi047.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-09-18T14:41:28.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-09-18T14:41:28.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-09-18T14:41:28.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-09-18T14:41:28.757 INFO:tasks.workunit.client.0.smithi047.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-18T14:41:28.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-09-18T14:41:28.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-09-18T14:41:29.139 INFO:tasks.workunit.client.0.smithi047.stderr:marked down osd.0. 2024-09-18T14:41:29.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-09-18T14:41:39.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:101: TEST_markdown_boot: sleep 15 2024-09-18T14:41:54.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:102: TEST_markdown_boot: ceph tell osd.0 get_latest_osdmap 2024-09-18T14:41:54.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: ceph osd tree 2024-09-18T14:41:54.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: grep up 2024-09-18T14:41:54.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: grep osd.0 2024-09-18T14:41:54.662 INFO:tasks.workunit.client.0.smithi047.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-18T14:41:54.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2024-09-18T14:41:54.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-09-18T14:41:54.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:41:54.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-09-18T14:41:54.664 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:41:54.664 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:41:54.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:41:54.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:41:54.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:41:54.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:41:54.771 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:41:54.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:41:54.772 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:41:54.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:41:54.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:41:54.773 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:41:54.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:41:54.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:41:54.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:41:54.775 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:41:54.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:41:54.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:41:54.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-09-18T14:41:54.791 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:41:54.791 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:41:54.791 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:41:54.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.188945 2024-09-18T14:41:54.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:41:54.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:41:54.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2024-09-18T14:41:54.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2024-09-18T14:41:54.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2024-09-18T14:41:54.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2024-09-18T14:41:54.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-09-18T14:41:54.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:41:54.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-09-18T14:41:54.794 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:41:54.794 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:41:54.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:41:54.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:41:54.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:41:54.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:41:54.795 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:41:54.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:41:54.796 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:41:54.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:41:54.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:41:54.797 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:41:54.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:41:54.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:41:54.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:41:54.799 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:41:54.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:41:54.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:41:54.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-09-18T14:41:54.801 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:41:54.802 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:41:54.802 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:41:54.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.188945 2024-09-18T14:41:54.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:41:54.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:41:54.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2024-09-18T14:41:54.804 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T14:41:54.804 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:41:54.804 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:41:54.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.188945 2024-09-18T14:41:54.806 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T14:41:54.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T14:41:54.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T14:41:54.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2024-09-18T14:41:54.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_boot_exceed_time td/osd-markdown 2024-09-18T14:41:54.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:107: TEST_markdown_boot_exceed_time: local dir=td/osd-markdown 2024-09-18T14:41:54.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:109: TEST_markdown_boot_exceed_time: run_mon td/osd-markdown a 2024-09-18T14:41:54.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2024-09-18T14:41:54.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T14:41:54.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T14:41:54.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T14:41:54.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2024-09-18T14:41:54.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2024-09-18T14:41:55.166 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T14:41:55.166 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:41:55.166 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:41:55.166 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:41:55.167 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:41:55.167 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:41:55.167 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188945/$cluster-$name.asok' 2024-09-18T14:41:55.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T14:41:55.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T14:41:55.428 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T14:41:55.428 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:41:55.428 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:41:55.428 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T14:41:55.428 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T14:41:55.428 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:41:55.428 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:41:55.428 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:41:55.428 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:41:55.428 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:41:55.428 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:41:55.429 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.188945/ceph-mon.a.asok 2024-09-18T14:41:55.429 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:41:55.429 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.188945/ceph-mon.a.asok config get fsid 2024-09-18T14:41:55.508 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T14:41:55.509 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:41:55.509 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:41:55.509 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T14:41:55.509 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T14:41:55.509 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:41:55.509 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:41:55.509 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:41:55.509 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:41:55.509 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:41:55.509 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:41:55.509 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.188945/ceph-mon.a.asok 2024-09-18T14:41:55.509 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:41:55.509 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.188945/ceph-mon.a.asok config get mon_host 2024-09-18T14:41:55.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:110: TEST_markdown_boot_exceed_time: run_mgr td/osd-markdown x 2024-09-18T14:41:55.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2024-09-18T14:41:55.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T14:41:55.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T14:41:55.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T14:41:55.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2024-09-18T14:41:55.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T14:41:56.431 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T14:41:56.431 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:41:56.431 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:41:56.431 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:41:56.431 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:41:56.431 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:41:56.431 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188945/$cluster-$name.asok' 2024-09-18T14:41:56.432 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:41:56.432 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:41:56.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:111: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 0 2024-09-18T14:41:56.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-18T14:41:56.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:41:56.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T14:41:56.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:41:56.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2024-09-18T14:41:56.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-18T14:41:56.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:41:56.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:41:56.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:41:56.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2024-09-18T14:41:56.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2024-09-18T14:41:56.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:41:56.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:41:56.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-18T14:41:56.460 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:41:56.460 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:41:56.460 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:41:56.460 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:41:56.460 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:41:56.460 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:41:56.460 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188945/$cluster-$name.asok' 2024-09-18T14:41:56.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' 2024-09-18T14:41:56.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:41:56.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:41:56.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:41:56.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-18T14:41:56.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-18T14:41:56.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:41:56.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:41:56.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:41:56.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:41:56.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:41:56.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:41:56.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2024-09-18T14:41:56.461 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:41:56.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4891de38-dd65-4e05-b785-b2c1c915e79a 2024-09-18T14:41:56.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4891de38-dd65-4e05-b785-b2c1c915e79a' 2024-09-18T14:41:56.462 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 4891de38-dd65-4e05-b785-b2c1c915e79a 2024-09-18T14:41:56.462 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:41:56.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC05upm+gYmHBAAfygYR4QSnaA1d/fjWziA5w== 2024-09-18T14:41:56.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC05upm+gYmHBAAfygYR4QSnaA1d/fjWziA5w=="}' 2024-09-18T14:41:56.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4891de38-dd65-4e05-b785-b2c1c915e79a -i td/osd-markdown/0/new.json 2024-09-18T14:41:56.774 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:41:56.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2024-09-18T14:41:56.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC05upm+gYmHBAAfygYR4QSnaA1d/fjWziA5w== --osd-uuid 4891de38-dd65-4e05-b785-b2c1c915e79a 2024-09-18T14:41:56.807 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:41:56.806+0000 7f9674129440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:41:56.810 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:41:56.809+0000 7f9674129440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:41:56.811 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:41:56.810+0000 7f9674129440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:41:56.811 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:41:56.810+0000 7f9674129440 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-18T14:41:56.811 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:41:56.810+0000 7f9674129440 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-18T14:41:56.811 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:41:56.810+0000 7f9674129440 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-18T14:41:56.850 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:41:56.849+0000 7f9674129440 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2024-09-18T14:42:02.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2024-09-18T14:42:02.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:42:02.233 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T14:42:02.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T14:42:02.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:42:02.748 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T14:42:02.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T14:42:02.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:42:02.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:42:02.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:42:02.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:42:02.769 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:42:02.768+0000 7fce98e08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:42:02.771 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:42:02.770+0000 7fce98e08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:42:02.772 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:42:02.771+0000 7fce98e08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:42:03.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T14:42:03.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:42:03.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:42:03.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:42:03.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:42:03.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:42:03.147 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:42:03.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:42:03.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:42:03.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:42:03.287 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:42:03.286+0000 7fce98e08440 -1 Falling back to public interface 2024-09-18T14:42:03.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:42:04.527 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:42:04.527 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:42:04.527 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:42:04.528 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:42:04.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:42:04.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:42:04.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:42:05.804 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:42:05.803+0000 7fce98e08440 -1 osd.0 0 log_to_monitors true 2024-09-18T14:42:05.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:42:05.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:42:05.893 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:42:05.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:42:05.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:42:05.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:42:06.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:42:07.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:42:07.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:42:07.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:42:07.365 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:42:07.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:42:07.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:42:07.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:42:08.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:42:08.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:42:08.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:42:08.823 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:42:08.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:42:08.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:42:09.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:42:10.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:42:10.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:42:10.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:42:10.241 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:42:10.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:42:10.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:42:10.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:42:11.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:42:11.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:42:11.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:42:11.621 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:42:11.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:42:11.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:42:12.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:42:13.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:42:13.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:42:13.020 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:42:13.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:42:13.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:42:13.021 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:42:13.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:42:14.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:42:14.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:42:14.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:42:14.511 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:42:14.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:42:14.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:42:14.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:42:15.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:42:15.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:42:15.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:42:15.910 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:42:15.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:42:15.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:42:16.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:42:17.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:42:17.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:42:17.311 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:42:17.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:42:17.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:42:17.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:42:18.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:42:19.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:42:19.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:42:19.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:42:19.040 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:42:19.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:42:19.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:42:20.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:42:21.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:42:21.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:42:21.090 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:42:21.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:42:21.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:42:21.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:42:21.988 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:42:22.989 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:42:22.989 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:42:22.990 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:42:22.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:42:22.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:42:22.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:42:23.666 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:42:24.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:42:24.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:42:24.668 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:42:24.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:42:24.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:42:24.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:42:25.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:42:26.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:42:26.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:42:26.447 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:42:26.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:42:26.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:42:26.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:42:27.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:42:28.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:42:28.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:42:28.105 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:42:28.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:42:28.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:42:28.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:42:28.535 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/195974,v1:127.0.0.1:6803/195974] [v2:127.0.0.1:6804/195974,v1:127.0.0.1:6805/195974] exists,up 4891de38-dd65-4e05-b785-b2c1c915e79a 2024-09-18T14:42:28.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:42:28.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:42:28.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:42:28.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:112: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 1 2024-09-18T14:42:28.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-18T14:42:28.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:42:28.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T14:42:28.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:42:28.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2024-09-18T14:42:28.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-18T14:42:28.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:42:28.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:42:28.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:42:28.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2024-09-18T14:42:28.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2024-09-18T14:42:28.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:42:28.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:42:28.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-18T14:42:28.541 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:42:28.541 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:42:28.541 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:42:28.541 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:42:28.541 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:42:28.541 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:42:28.541 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188945/$cluster-$name.asok' 2024-09-18T14:42:28.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' 2024-09-18T14:42:28.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:42:28.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:42:28.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:42:28.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-18T14:42:28.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-18T14:42:28.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:42:28.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:42:28.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:42:28.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:42:28.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:42:28.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:42:28.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2024-09-18T14:42:28.542 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:42:28.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ba9de3ab-5682-4311-aa71-fda3c5abc35c 2024-09-18T14:42:28.542 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 ba9de3ab-5682-4311-aa71-fda3c5abc35c 2024-09-18T14:42:28.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ba9de3ab-5682-4311-aa71-fda3c5abc35c' 2024-09-18T14:42:28.543 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:42:28.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDU5upmWaLaIBAAO+LRbfN4DHeVohhsubzG9w== 2024-09-18T14:42:28.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDU5upmWaLaIBAAO+LRbfN4DHeVohhsubzG9w=="}' 2024-09-18T14:42:28.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ba9de3ab-5682-4311-aa71-fda3c5abc35c -i td/osd-markdown/1/new.json 2024-09-18T14:42:29.010 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:42:29.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2024-09-18T14:42:29.021 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDU5upmWaLaIBAAO+LRbfN4DHeVohhsubzG9w== --osd-uuid ba9de3ab-5682-4311-aa71-fda3c5abc35c 2024-09-18T14:42:29.044 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:42:29.044+0000 7f58c9c48440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:42:29.100 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:42:29.101+0000 7f58c9c48440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:42:29.101 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:42:29.102+0000 7f58c9c48440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:42:29.102 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:42:29.102+0000 7f58c9c48440 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-18T14:42:29.102 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:42:29.102+0000 7f58c9c48440 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-18T14:42:29.102 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:42:29.102+0000 7f58c9c48440 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-18T14:42:29.205 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:42:29.204+0000 7f58c9c48440 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2024-09-18T14:42:35.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2024-09-18T14:42:35.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:42:35.133 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T14:42:35.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T14:42:35.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:42:35.657 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T14:42:35.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T14:42:35.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:42:35.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:42:35.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:42:35.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:42:35.678 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:42:35.677+0000 7fa41d822440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:42:35.707 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:42:35.706+0000 7fa41d822440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:42:35.708 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:42:35.707+0000 7fa41d822440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:42:36.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T14:42:36.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:42:36.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:42:36.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:42:36.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:42:36.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:42:36.063 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:42:36.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:42:36.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:42:36.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:42:36.242 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:42:36.241+0000 7fa41d822440 -1 Falling back to public interface 2024-09-18T14:42:36.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:42:37.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:42:37.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:42:37.456 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:42:37.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:42:37.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:42:37.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:42:37.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:42:38.846 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:42:38.845+0000 7fa41d822440 -1 osd.1 0 log_to_monitors true 2024-09-18T14:42:38.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:42:38.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:42:38.874 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:42:38.875 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:42:38.875 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:42:38.875 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:42:39.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:42:40.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:42:40.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:42:40.330 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:42:40.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:42:40.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:42:40.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:42:40.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:42:41.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:42:41.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:42:41.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:42:41.802 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:42:41.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:42:41.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:42:42.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:42:43.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:42:43.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:42:43.216 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:42:43.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:42:43.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:42:43.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:42:43.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:42:44.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:42:44.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:42:44.632 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:42:44.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:42:44.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:42:44.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:42:45.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:42:46.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:42:46.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:42:46.153 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:42:46.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:42:46.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:42:46.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:42:46.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:42:47.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:42:47.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:42:47.661 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:42:47.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:42:47.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:42:47.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:42:48.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:42:49.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:42:49.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:42:49.048 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:42:49.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:42:49.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:42:49.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:42:49.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:42:50.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:42:50.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:42:50.429 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:42:50.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:42:50.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:42:50.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:42:51.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:42:52.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:42:52.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:42:52.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:42:52.056 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:42:52.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:42:52.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:42:52.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:42:53.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:42:53.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:42:53.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:42:53.587 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:42:53.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:42:53.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:42:54.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:42:55.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:42:55.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:42:55.049 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:42:55.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:42:55.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:42:55.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:42:55.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:42:56.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:42:56.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:42:56.778 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:42:56.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:42:56.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:42:56.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:42:57.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:42:58.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:42:58.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:42:58.697 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:42:58.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:42:58.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:42:58.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:42:59.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:43:00.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:43:00.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:43:00.358 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:43:00.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:43:00.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:43:00.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:43:00.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:43:01.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:43:01.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:43:01.952 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T14:43:01.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:43:01.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:43:01.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:43:02.331 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/197703,v1:127.0.0.1:6811/197703] [v2:127.0.0.1:6812/197703,v1:127.0.0.1:6813/197703] exists,up ba9de3ab-5682-4311-aa71-fda3c5abc35c 2024-09-18T14:43:02.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:43:02.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:43:02.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:43:02.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:113: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 2 2024-09-18T14:43:02.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-18T14:43:02.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:43:02.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T14:43:02.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:43:02.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2024-09-18T14:43:02.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-18T14:43:02.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:43:02.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:43:02.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:43:02.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2024-09-18T14:43:02.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2024-09-18T14:43:02.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:43:02.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:43:02.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-18T14:43:02.336 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:43:02.337 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:43:02.337 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:43:02.337 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:43:02.337 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:43:02.337 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:43:02.337 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188945/$cluster-$name.asok' 2024-09-18T14:43:02.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' 2024-09-18T14:43:02.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:43:02.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:43:02.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:43:02.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-18T14:43:02.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-18T14:43:02.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:43:02.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:43:02.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:43:02.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:43:02.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:43:02.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:43:02.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2024-09-18T14:43:02.338 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:43:02.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=113d492b-afbe-40a5-88f3-b63cd2140abe 2024-09-18T14:43:02.338 INFO:tasks.workunit.client.0.smithi047.stdout:add osd2 113d492b-afbe-40a5-88f3-b63cd2140abe 2024-09-18T14:43:02.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 113d492b-afbe-40a5-88f3-b63cd2140abe' 2024-09-18T14:43:02.339 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:43:02.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD25upml5i+FBAAEozXcPt0ZdwvvPLjysyp9A== 2024-09-18T14:43:02.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD25upml5i+FBAAEozXcPt0ZdwvvPLjysyp9A=="}' 2024-09-18T14:43:02.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 113d492b-afbe-40a5-88f3-b63cd2140abe -i td/osd-markdown/2/new.json 2024-09-18T14:43:02.848 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:43:02.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2024-09-18T14:43:02.860 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD25upml5i+FBAAEozXcPt0ZdwvvPLjysyp9A== --osd-uuid 113d492b-afbe-40a5-88f3-b63cd2140abe 2024-09-18T14:43:02.883 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:43:02.881+0000 7ff3e3c0d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:43:02.884 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:43:02.883+0000 7ff3e3c0d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:43:02.885 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:43:02.884+0000 7ff3e3c0d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:43:02.886 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:43:02.885+0000 7ff3e3c0d440 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-18T14:43:02.886 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:43:02.885+0000 7ff3e3c0d440 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-18T14:43:02.886 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:43:02.885+0000 7ff3e3c0d440 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-18T14:43:02.964 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:43:02.963+0000 7ff3e3c0d440 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2024-09-18T14:43:08.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2024-09-18T14:43:08.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:43:08.540 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd2 key to auth repository 2024-09-18T14:43:08.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T14:43:08.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:43:09.046 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T14:43:09.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T14:43:09.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:43:09.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:43:09.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:43:09.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:43:09.067 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:43:09.067+0000 7ff56340d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:43:09.103 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:43:09.102+0000 7ff56340d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:43:09.104 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:43:09.103+0000 7ff56340d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:43:09.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T14:43:09.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:43:09.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T14:43:09.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:43:09.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:43:09.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:43:09.414 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:43:09.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:43:09.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:43:09.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:43:09.641 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:43:09.640+0000 7ff56340d440 -1 Falling back to public interface 2024-09-18T14:43:09.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:43:10.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:43:10.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:43:10.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:43:10.811 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:43:10.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:43:10.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:43:11.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:43:12.134 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:43:12.133+0000 7ff56340d440 -1 osd.2 0 log_to_monitors true 2024-09-18T14:43:12.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:43:12.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:43:12.229 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:43:12.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:43:12.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:43:12.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:43:12.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:43:13.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:43:13.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:43:13.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:43:13.770 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:43:13.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:43:13.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:43:14.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:43:15.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:43:15.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:43:15.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:43:15.216 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:43:15.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:43:15.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:43:15.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:43:16.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:43:16.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:43:16.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:43:16.593 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:43:16.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:43:16.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:43:17.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:43:18.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:43:18.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:43:18.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:43:18.156 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:43:18.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:43:18.157 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:43:18.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:43:19.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:43:19.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:43:19.542 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:43:19.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:43:19.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:43:19.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:43:19.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:43:20.989 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:43:20.989 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:43:20.989 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:43:20.989 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:43:20.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:43:20.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:43:21.406 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:43:22.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:43:22.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:43:22.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:43:22.408 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:43:22.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:43:22.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:43:22.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:43:23.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:43:23.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:43:23.953 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:43:23.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:43:23.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:43:23.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:43:24.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:43:25.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:43:25.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:43:25.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:43:25.435 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:43:25.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:43:25.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:43:26.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:43:27.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:43:27.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:43:27.116 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:43:27.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:43:27.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:43:27.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:43:27.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:43:28.826 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:43:28.826 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:43:28.826 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:43:28.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:43:28.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:43:28.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:43:29.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:43:30.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:43:30.519 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:43:30.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:43:30.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:43:30.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:43:30.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:43:31.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:43:32.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:43:32.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:43:32.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:43:32.113 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:43:32.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:43:32.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:43:32.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:43:33.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:43:33.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:43:33.621 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:43:33.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:43:33.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:43:33.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:43:34.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:43:35.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:43:35.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:43:35.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:43:35.410 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T14:43:35.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:43:35.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:43:35.873 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/199457,v1:127.0.0.1:6819/199457] [v2:127.0.0.1:6820/199457,v1:127.0.0.1:6821/199457] exists,up 113d492b-afbe-40a5-88f3-b63cd2140abe 2024-09-18T14:43:35.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:43:35.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:43:35.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:43:35.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:115: TEST_markdown_boot_exceed_time: create_rbd_pool 2024-09-18T14:43:35.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T14:43:36.236 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'rbd' does not exist 2024-09-18T14:43:36.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T14:43:36.247 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T14:43:36.777 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'rbd' already exists 2024-09-18T14:43:36.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T14:43:37.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T14:43:39.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:118: TEST_markdown_boot_exceed_time: local count=3 2024-09-18T14:43:39.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:119: TEST_markdown_boot_exceed_time: local period=20 2024-09-18T14:43:39.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:120: TEST_markdown_boot_exceed_time: local sleeptime=10 2024-09-18T14:43:39.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:121: TEST_markdown_boot_exceed_time: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2024-09-18T14:43:39.813 INFO:tasks.workunit.client.0.smithi047.stdout:{} 2024-09-18T14:43:39.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:122: TEST_markdown_boot_exceed_time: ceph tell osd.0 injectargs '--osd_max_markdown_period 20' 2024-09-18T14:43:39.922 INFO:tasks.workunit.client.0.smithi047.stdout:{} 2024-09-18T14:43:39.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:124: TEST_markdown_boot_exceed_time: markdown_N_impl 4 20 10 2024-09-18T14:43:39.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=4 2024-09-18T14:43:39.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=20 2024-09-18T14:43:39.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2024-09-18T14:43:39.932 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 4 2024-09-18T14:43:39.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-09-18T14:43:39.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-09-18T14:43:40.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-09-18T14:43:40.419 INFO:tasks.workunit.client.0.smithi047.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-18T14:43:40.419 INFO:tasks.workunit.client.0.smithi047.stdout:-1 0.29306 root default 2024-09-18T14:43:40.419 INFO:tasks.workunit.client.0.smithi047.stdout:-3 0.29306 host smithi047 2024-09-18T14:43:40.419 INFO:tasks.workunit.client.0.smithi047.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-18T14:43:40.419 INFO:tasks.workunit.client.0.smithi047.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-09-18T14:43:40.420 INFO:tasks.workunit.client.0.smithi047.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-09-18T14:43:40.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-09-18T14:43:40.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-09-18T14:43:40.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-09-18T14:43:40.795 INFO:tasks.workunit.client.0.smithi047.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-18T14:43:40.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-09-18T14:43:40.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-09-18T14:43:41.178 INFO:tasks.workunit.client.0.smithi047.stderr:marked down osd.0. 2024-09-18T14:43:41.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-09-18T14:43:51.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-09-18T14:43:51.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-09-18T14:43:51.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-09-18T14:43:51.666 INFO:tasks.workunit.client.0.smithi047.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-18T14:43:51.666 INFO:tasks.workunit.client.0.smithi047.stdout:-1 0.29306 root default 2024-09-18T14:43:51.666 INFO:tasks.workunit.client.0.smithi047.stdout:-3 0.29306 host smithi047 2024-09-18T14:43:51.666 INFO:tasks.workunit.client.0.smithi047.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-18T14:43:51.666 INFO:tasks.workunit.client.0.smithi047.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-09-18T14:43:51.666 INFO:tasks.workunit.client.0.smithi047.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-09-18T14:43:51.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-09-18T14:43:51.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-09-18T14:43:51.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-09-18T14:43:52.088 INFO:tasks.workunit.client.0.smithi047.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-18T14:43:52.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-09-18T14:43:52.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-09-18T14:43:52.470 INFO:tasks.workunit.client.0.smithi047.stderr:marked down osd.0. 2024-09-18T14:43:52.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-09-18T14:44:02.483 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-09-18T14:44:02.483 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-09-18T14:44:02.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-09-18T14:44:03.028 INFO:tasks.workunit.client.0.smithi047.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-18T14:44:03.028 INFO:tasks.workunit.client.0.smithi047.stdout:-1 0.29306 root default 2024-09-18T14:44:03.028 INFO:tasks.workunit.client.0.smithi047.stdout:-3 0.29306 host smithi047 2024-09-18T14:44:03.028 INFO:tasks.workunit.client.0.smithi047.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-18T14:44:03.028 INFO:tasks.workunit.client.0.smithi047.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-09-18T14:44:03.028 INFO:tasks.workunit.client.0.smithi047.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-09-18T14:44:03.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-09-18T14:44:03.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-09-18T14:44:03.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-09-18T14:44:03.439 INFO:tasks.workunit.client.0.smithi047.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-18T14:44:03.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-09-18T14:44:03.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-09-18T14:44:03.821 INFO:tasks.workunit.client.0.smithi047.stderr:marked down osd.0. 2024-09-18T14:44:03.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-09-18T14:44:13.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-09-18T14:44:13.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-09-18T14:44:13.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-09-18T14:44:14.324 INFO:tasks.workunit.client.0.smithi047.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-18T14:44:14.325 INFO:tasks.workunit.client.0.smithi047.stdout:-1 0.29306 root default 2024-09-18T14:44:14.325 INFO:tasks.workunit.client.0.smithi047.stdout:-3 0.29306 host smithi047 2024-09-18T14:44:14.325 INFO:tasks.workunit.client.0.smithi047.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-18T14:44:14.325 INFO:tasks.workunit.client.0.smithi047.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-09-18T14:44:14.325 INFO:tasks.workunit.client.0.smithi047.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-09-18T14:44:14.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-09-18T14:44:14.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-09-18T14:44:14.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-09-18T14:44:14.696 INFO:tasks.workunit.client.0.smithi047.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-18T14:44:14.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-09-18T14:44:14.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-09-18T14:44:15.078 INFO:tasks.workunit.client.0.smithi047.stderr:marked down osd.0. 2024-09-18T14:44:15.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-09-18T14:44:25.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:125: TEST_markdown_boot_exceed_time: sleep 15 2024-09-18T14:44:40.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:126: TEST_markdown_boot_exceed_time: ceph tell osd.0 get_latest_osdmap 2024-09-18T14:44:40.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: ceph osd tree 2024-09-18T14:44:40.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: grep up 2024-09-18T14:44:40.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: grep osd.0 2024-09-18T14:44:40.610 INFO:tasks.workunit.client.0.smithi047.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-18T14:44:40.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2024-09-18T14:44:40.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-09-18T14:44:40.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:44:40.612 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-09-18T14:44:40.612 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:44:40.612 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:44:40.612 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:44:40.612 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:44:40.612 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:44:40.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:44:40.719 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:44:40.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:44:40.720 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:44:40.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:44:40.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:44:40.721 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:44:40.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:44:40.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:44:40.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:44:40.723 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:44:40.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:44:40.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:44:40.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-09-18T14:44:40.740 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:44:40.740 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:44:40.740 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:44:40.740 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.188945 2024-09-18T14:44:40.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:44:40.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:44:40.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2024-09-18T14:44:40.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2024-09-18T14:44:40.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2024-09-18T14:44:40.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2024-09-18T14:44:40.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-09-18T14:44:40.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:44:40.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-09-18T14:44:40.743 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:44:40.743 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:44:40.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:44:40.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:44:40.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:44:40.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:44:40.743 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:44:40.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:44:40.744 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:44:40.746 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:44:40.746 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:44:40.746 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:44:40.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:44:40.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:44:40.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:44:40.747 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:44:40.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:44:40.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:44:40.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-09-18T14:44:40.750 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:44:40.750 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:44:40.750 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:44:40.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.188945 2024-09-18T14:44:40.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:44:40.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:44:40.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2024-09-18T14:44:40.752 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T14:44:40.752 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:44:40.752 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:44:40.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.188945 2024-09-18T14:44:40.754 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T14:44:40.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T14:44:40.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T14:44:40.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2024-09-18T14:44:40.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_exceed_maxdown_count td/osd-markdown 2024-09-18T14:44:40.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:59: TEST_markdown_exceed_maxdown_count: local dir=td/osd-markdown 2024-09-18T14:44:40.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:61: TEST_markdown_exceed_maxdown_count: run_mon td/osd-markdown a 2024-09-18T14:44:40.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2024-09-18T14:44:40.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T14:44:40.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T14:44:40.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T14:44:40.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2024-09-18T14:44:40.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2024-09-18T14:44:41.114 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T14:44:41.114 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:44:41.115 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:44:41.115 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:44:41.115 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:44:41.115 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:44:41.115 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188945/$cluster-$name.asok' 2024-09-18T14:44:41.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T14:44:41.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T14:44:41.366 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T14:44:41.367 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:44:41.367 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:44:41.367 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T14:44:41.367 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T14:44:41.367 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:44:41.367 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:44:41.368 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:44:41.369 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:44:41.369 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:44:41.369 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:44:41.369 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.188945/ceph-mon.a.asok 2024-09-18T14:44:41.369 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:44:41.369 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.188945/ceph-mon.a.asok config get fsid 2024-09-18T14:44:41.453 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T14:44:41.453 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:44:41.453 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:44:41.453 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T14:44:41.453 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T14:44:41.453 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:44:41.453 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:44:41.453 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:44:41.453 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:44:41.453 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:44:41.454 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:44:41.454 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.188945/ceph-mon.a.asok 2024-09-18T14:44:41.454 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:44:41.454 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.188945/ceph-mon.a.asok config get mon_host 2024-09-18T14:44:41.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:62: TEST_markdown_exceed_maxdown_count: run_mgr td/osd-markdown x 2024-09-18T14:44:41.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2024-09-18T14:44:41.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T14:44:41.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T14:44:41.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T14:44:41.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2024-09-18T14:44:41.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T14:44:42.371 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T14:44:42.372 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:44:42.372 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:44:42.372 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:44:42.372 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:44:42.372 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:44:42.372 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188945/$cluster-$name.asok' 2024-09-18T14:44:42.372 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:44:42.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:44:42.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:63: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 0 2024-09-18T14:44:42.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-18T14:44:42.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:44:42.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T14:44:42.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:44:42.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2024-09-18T14:44:42.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-18T14:44:42.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:44:42.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:44:42.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:44:42.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2024-09-18T14:44:42.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2024-09-18T14:44:42.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:44:42.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:44:42.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-18T14:44:42.399 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:44:42.399 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:44:42.399 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:44:42.399 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:44:42.399 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:44:42.399 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:44:42.399 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188945/$cluster-$name.asok' 2024-09-18T14:44:42.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' 2024-09-18T14:44:42.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:44:42.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:44:42.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:44:42.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-18T14:44:42.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-18T14:44:42.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:44:42.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:44:42.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:44:42.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:44:42.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:44:42.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:44:42.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2024-09-18T14:44:42.400 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:44:42.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=90400af4-8586-4635-9f65-34148cee9caf 2024-09-18T14:44:42.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 90400af4-8586-4635-9f65-34148cee9caf' 2024-09-18T14:44:42.401 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 90400af4-8586-4635-9f65-34148cee9caf 2024-09-18T14:44:42.401 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:44:42.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBa5+pmPuqAGBAAuANRzLNTNOFiHK0qln2pFA== 2024-09-18T14:44:42.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBa5+pmPuqAGBAAuANRzLNTNOFiHK0qln2pFA=="}' 2024-09-18T14:44:42.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 90400af4-8586-4635-9f65-34148cee9caf -i td/osd-markdown/0/new.json 2024-09-18T14:44:42.713 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:44:42.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2024-09-18T14:44:42.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBa5+pmPuqAGBAAuANRzLNTNOFiHK0qln2pFA== --osd-uuid 90400af4-8586-4635-9f65-34148cee9caf 2024-09-18T14:44:42.745 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:44:42.744+0000 7fe6ed31d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:44:42.758 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:44:42.757+0000 7fe6ed31d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:44:42.759 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:44:42.758+0000 7fe6ed31d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:44:42.759 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:44:42.758+0000 7fe6ed31d440 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-18T14:44:42.759 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:44:42.758+0000 7fe6ed31d440 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-18T14:44:42.759 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:44:42.758+0000 7fe6ed31d440 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-18T14:44:42.800 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:44:42.797+0000 7fe6ed31d440 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2024-09-18T14:44:48.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2024-09-18T14:44:48.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:44:48.373 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T14:44:48.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T14:44:48.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:44:48.887 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T14:44:48.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T14:44:48.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:44:48.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:44:48.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:44:48.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:44:48.909 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:44:48.907+0000 7fc71d808440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:44:48.911 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:44:48.910+0000 7fc71d808440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:44:48.912 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:44:48.911+0000 7fc71d808440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:44:49.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T14:44:49.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:44:49.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:44:49.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:44:49.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:44:49.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:44:49.335 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:44:49.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:44:49.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:44:49.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:44:49.445 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:44:49.444+0000 7fc71d808440 -1 Falling back to public interface 2024-09-18T14:44:49.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:44:50.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:44:50.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:44:50.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:44:50.695 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:44:50.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:44:50.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:44:51.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:44:51.975 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:44:51.974+0000 7fc71d808440 -1 osd.0 0 log_to_monitors true 2024-09-18T14:44:52.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:44:52.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:44:52.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:44:52.067 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:44:52.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:44:52.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:44:52.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:44:53.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:44:53.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:44:53.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:44:53.586 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:44:53.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:44:53.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:44:53.989 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:44:54.989 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:44:54.989 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:44:54.989 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:44:54.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:44:54.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:44:54.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:44:55.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:44:56.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:44:56.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:44:56.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:44:56.374 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:44:56.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:44:56.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:44:56.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:44:57.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:44:57.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:44:57.816 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:44:57.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:44:57.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:44:57.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:44:58.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:44:59.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:44:59.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:44:59.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:44:59.224 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:44:59.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:44:59.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:44:59.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:45:00.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:45:00.666 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:45:00.666 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:45:00.666 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:45:00.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:45:00.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:45:01.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:45:02.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:45:02.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:45:02.059 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:45:02.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:45:02.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:45:02.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:45:02.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:45:03.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:45:03.450 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:45:03.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:45:03.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:45:03.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:45:03.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:45:04.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:45:05.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:45:05.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:45:05.116 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:45:05.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:45:05.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:45:05.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:45:05.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:45:06.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:45:06.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:45:06.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:45:06.850 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:45:06.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:45:06.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:45:07.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:45:08.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:45:08.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:45:08.623 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:45:08.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:45:08.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:45:08.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:45:09.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:45:10.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:45:10.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:45:10.382 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:45:10.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:45:10.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:45:10.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:45:11.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:45:12.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:45:12.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:45:12.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:45:12.360 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:45:12.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:45:12.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:45:13.194 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:45:14.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:45:14.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:45:14.195 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:45:14.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:45:14.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:45:14.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:45:14.674 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/202029,v1:127.0.0.1:6803/202029] [v2:127.0.0.1:6804/202029,v1:127.0.0.1:6805/202029] exists,up 90400af4-8586-4635-9f65-34148cee9caf 2024-09-18T14:45:14.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:45:14.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:45:14.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:45:14.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:64: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 1 2024-09-18T14:45:14.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-18T14:45:14.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:45:14.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T14:45:14.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:45:14.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2024-09-18T14:45:14.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-18T14:45:14.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:45:14.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:45:14.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:45:14.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2024-09-18T14:45:14.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2024-09-18T14:45:14.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:45:14.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:45:14.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-18T14:45:14.679 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:45:14.679 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:45:14.679 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:45:14.679 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:45:14.679 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:14.680 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:45:14.680 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188945/$cluster-$name.asok' 2024-09-18T14:45:14.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' 2024-09-18T14:45:14.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:45:14.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:45:14.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:45:14.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-18T14:45:14.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-18T14:45:14.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:45:14.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:45:14.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:45:14.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:45:14.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:45:14.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:45:14.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2024-09-18T14:45:14.681 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:45:14.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6ace50e3-2dee-40a8-b4ac-cc2266b41ec3 2024-09-18T14:45:14.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 6ace50e3-2dee-40a8-b4ac-cc2266b41ec3' 2024-09-18T14:45:14.681 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 6ace50e3-2dee-40a8-b4ac-cc2266b41ec3 2024-09-18T14:45:14.681 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:45:14.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB65+pmiochKRAACvur+liLzv9Hapn5PQ+XfA== 2024-09-18T14:45:14.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB65+pmiochKRAACvur+liLzv9Hapn5PQ+XfA=="}' 2024-09-18T14:45:14.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6ace50e3-2dee-40a8-b4ac-cc2266b41ec3 -i td/osd-markdown/1/new.json 2024-09-18T14:45:15.247 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:45:15.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2024-09-18T14:45:15.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB65+pmiochKRAACvur+liLzv9Hapn5PQ+XfA== --osd-uuid 6ace50e3-2dee-40a8-b4ac-cc2266b41ec3 2024-09-18T14:45:15.281 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:45:15.280+0000 7f315b614440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:45:15.314 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:45:15.313+0000 7f315b614440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:45:15.315 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:45:15.314+0000 7f315b614440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:45:15.316 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:45:15.314+0000 7f315b614440 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-18T14:45:15.316 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:45:15.314+0000 7f315b614440 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-18T14:45:15.316 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:45:15.314+0000 7f315b614440 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-18T14:45:15.389 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:45:15.388+0000 7f315b614440 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2024-09-18T14:45:21.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2024-09-18T14:45:21.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:45:21.308 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T14:45:21.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T14:45:21.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:45:21.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T14:45:21.817 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T14:45:21.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:45:21.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:45:21.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:45:21.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:45:21.837 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:45:21.836+0000 7f456e122440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:45:21.870 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:45:21.869+0000 7f456e122440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:45:21.871 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:45:21.871+0000 7f456e122440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:45:22.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T14:45:22.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:45:22.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:45:22.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:45:22.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:45:22.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:45:22.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:45:22.200 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:45:22.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:45:22.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:45:22.416 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:45:22.415+0000 7f456e122440 -1 Falling back to public interface 2024-09-18T14:45:22.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:45:23.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:45:23.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:45:23.577 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:45:23.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:45:23.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:45:23.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:45:23.985 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:45:24.983 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:45:24.982+0000 7f456e122440 -1 osd.1 0 log_to_monitors true 2024-09-18T14:45:24.988 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:45:24.988 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:45:24.988 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:45:24.989 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:45:24.989 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:45:24.989 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:45:25.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:45:26.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:45:26.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:45:26.435 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:45:26.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:45:26.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:45:26.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:45:26.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:45:27.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:45:27.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:45:27.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:45:27.866 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:45:27.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:45:27.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:45:28.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:45:29.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:45:29.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:45:29.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:45:29.329 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:45:29.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:45:29.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:45:29.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:45:30.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:45:30.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:45:30.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:45:30.736 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:45:30.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:45:30.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:45:31.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:45:32.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:45:32.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:45:32.124 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:45:32.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:45:32.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:45:32.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:45:32.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:45:33.549 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:45:33.549 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:45:33.549 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:45:33.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:45:33.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:45:33.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:45:33.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:45:34.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:45:34.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:45:34.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:45:34.960 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:45:34.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:45:34.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:45:35.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:45:36.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:45:36.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:45:36.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:45:36.367 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:45:36.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:45:36.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:45:37.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:45:38.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:45:38.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:45:38.053 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:45:38.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:45:38.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:45:38.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:45:38.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:45:39.746 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:45:39.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:45:39.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:45:39.747 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:45:39.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:45:39.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:45:40.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:45:41.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:45:41.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:45:41.410 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:45:41.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:45:41.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:45:41.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:45:42.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:45:43.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:45:43.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:45:43.130 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:45:43.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:45:43.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:45:43.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:45:43.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:45:44.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:45:44.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:45:44.976 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:45:44.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:45:44.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:45:44.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:45:45.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:45:46.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:45:46.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:45:46.824 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:45:46.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:45:46.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:45:46.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:45:47.248 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:45:48.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:45:48.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:45:48.250 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T14:45:48.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:45:48.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:45:48.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:45:48.665 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/203758,v1:127.0.0.1:6811/203758] [v2:127.0.0.1:6812/203758,v1:127.0.0.1:6813/203758] exists,up 6ace50e3-2dee-40a8-b4ac-cc2266b41ec3 2024-09-18T14:45:48.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:45:48.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:45:48.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:45:48.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:65: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 2 2024-09-18T14:45:48.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-18T14:45:48.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:45:48.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T14:45:48.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:45:48.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2024-09-18T14:45:48.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-18T14:45:48.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:45:48.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:45:48.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:45:48.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2024-09-18T14:45:48.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2024-09-18T14:45:48.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:45:48.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:45:48.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-18T14:45:48.669 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:45:48.669 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:45:48.669 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:45:48.670 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:45:48.670 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:45:48.670 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:45:48.670 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188945/$cluster-$name.asok' 2024-09-18T14:45:48.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' 2024-09-18T14:45:48.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:45:48.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:45:48.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:45:48.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-18T14:45:48.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-18T14:45:48.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:45:48.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:45:48.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:45:48.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:45:48.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:45:48.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:45:48.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2024-09-18T14:45:48.671 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:45:48.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6242151a-1a40-4986-875e-4bed0b9d2df4 2024-09-18T14:45:48.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 6242151a-1a40-4986-875e-4bed0b9d2df4' 2024-09-18T14:45:48.672 INFO:tasks.workunit.client.0.smithi047.stdout:add osd2 6242151a-1a40-4986-875e-4bed0b9d2df4 2024-09-18T14:45:48.672 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:45:48.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCc5+pmoXWYKBAAlBoHvYoI5SDVI5esk4SbqQ== 2024-09-18T14:45:48.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCc5+pmoXWYKBAAlBoHvYoI5SDVI5esk4SbqQ=="}' 2024-09-18T14:45:48.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6242151a-1a40-4986-875e-4bed0b9d2df4 -i td/osd-markdown/2/new.json 2024-09-18T14:45:49.193 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:45:49.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2024-09-18T14:45:49.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCc5+pmoXWYKBAAlBoHvYoI5SDVI5esk4SbqQ== --osd-uuid 6242151a-1a40-4986-875e-4bed0b9d2df4 2024-09-18T14:45:49.227 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:45:49.226+0000 7f5ba9622440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:45:49.229 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:45:49.228+0000 7f5ba9622440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:45:49.230 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:45:49.229+0000 7f5ba9622440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:45:49.230 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:45:49.229+0000 7f5ba9622440 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-18T14:45:49.230 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:45:49.229+0000 7f5ba9622440 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-18T14:45:49.230 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:45:49.229+0000 7f5ba9622440 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-18T14:45:49.308 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:45:49.307+0000 7f5ba9622440 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2024-09-18T14:45:54.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2024-09-18T14:45:54.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:45:54.923 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd2 key to auth repository 2024-09-18T14:45:54.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T14:45:54.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:45:55.435 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T14:45:55.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T14:45:55.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:45:55.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:45:55.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:45:55.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:45:55.456 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:45:55.455+0000 7f50d1c16440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:45:55.481 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:45:55.480+0000 7f50d1c16440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:45:55.483 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:45:55.481+0000 7f50d1c16440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:45:55.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T14:45:55.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:45:55.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T14:45:55.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:45:55.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:45:55.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:45:55.840 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:45:55.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:45:55.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:45:55.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:45:56.022 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:45:56.021+0000 7f50d1c16440 -1 Falling back to public interface 2024-09-18T14:45:56.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:45:57.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:45:57.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:45:57.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:45:57.264 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:45:57.264 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:45:57.264 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:45:57.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:45:58.637 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:45:58.636+0000 7f50d1c16440 -1 osd.2 0 log_to_monitors true 2024-09-18T14:45:58.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:45:58.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:45:58.642 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:45:58.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:45:58.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:45:58.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:45:59.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:46:00.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:46:00.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:46:00.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:46:00.123 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:46:00.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:46:00.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:46:00.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:46:01.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:46:01.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:46:01.602 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:46:01.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:46:01.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:46:01.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:46:01.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:46:02.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:46:02.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:46:02.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:46:02.982 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:46:02.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:46:02.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:46:03.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:46:04.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:46:04.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:46:04.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:46:04.425 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:46:04.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:46:04.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:46:04.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:46:05.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:46:05.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:46:05.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:46:05.904 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:46:05.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:46:05.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:46:06.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:46:07.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:46:07.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:46:07.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:46:07.327 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:46:07.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:46:07.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:46:07.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:46:08.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:46:08.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:46:08.767 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:46:08.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:46:08.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:46:08.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:46:09.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:46:10.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:46:10.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:46:10.258 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:46:10.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:46:10.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:46:10.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:46:11.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:46:12.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:46:12.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:46:12.122 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:46:12.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:46:12.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:46:12.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:46:12.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:46:13.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:46:13.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:46:13.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:46:13.820 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:46:13.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:46:13.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:46:14.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:46:15.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:46:15.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:46:15.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:46:15.401 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:46:15.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:46:15.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:46:16.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:46:17.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:46:17.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:46:17.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:46:17.222 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:46:17.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:46:17.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:46:17.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:46:18.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:46:18.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:46:18.917 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:46:18.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:46:18.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:46:18.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:46:19.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:46:20.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:46:20.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:46:20.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:46:20.636 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:46:20.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:46:20.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:46:21.077 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/205514,v1:127.0.0.1:6819/205514] [v2:127.0.0.1:6820/205514,v1:127.0.0.1:6821/205514] exists,up 6242151a-1a40-4986-875e-4bed0b9d2df4 2024-09-18T14:46:21.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:46:21.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:46:21.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:46:21.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:67: TEST_markdown_exceed_maxdown_count: create_rbd_pool 2024-09-18T14:46:21.079 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T14:46:21.479 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'rbd' does not exist 2024-09-18T14:46:21.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T14:46:21.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T14:46:22.081 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'rbd' already exists 2024-09-18T14:46:22.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T14:46:23.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T14:46:25.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:70: TEST_markdown_exceed_maxdown_count: local count=3 2024-09-18T14:46:25.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:71: TEST_markdown_exceed_maxdown_count: local sleeptime=10 2024-09-18T14:46:25.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:72: TEST_markdown_exceed_maxdown_count: local period=300 2024-09-18T14:46:25.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:73: TEST_markdown_exceed_maxdown_count: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2024-09-18T14:46:25.213 INFO:tasks.workunit.client.0.smithi047.stdout:{} 2024-09-18T14:46:25.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:74: TEST_markdown_exceed_maxdown_count: ceph tell osd.0 injectargs '--osd_max_markdown_period 300' 2024-09-18T14:46:25.323 INFO:tasks.workunit.client.0.smithi047.stdout:{} 2024-09-18T14:46:25.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:76: TEST_markdown_exceed_maxdown_count: markdown_N_impl 4 300 10 2024-09-18T14:46:25.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=4 2024-09-18T14:46:25.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=300 2024-09-18T14:46:25.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2024-09-18T14:46:25.331 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 4 2024-09-18T14:46:25.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-09-18T14:46:25.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-09-18T14:46:25.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-09-18T14:46:25.797 INFO:tasks.workunit.client.0.smithi047.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-18T14:46:25.797 INFO:tasks.workunit.client.0.smithi047.stdout:-1 0.29306 root default 2024-09-18T14:46:25.797 INFO:tasks.workunit.client.0.smithi047.stdout:-3 0.29306 host smithi047 2024-09-18T14:46:25.797 INFO:tasks.workunit.client.0.smithi047.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-18T14:46:25.798 INFO:tasks.workunit.client.0.smithi047.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-09-18T14:46:25.798 INFO:tasks.workunit.client.0.smithi047.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-09-18T14:46:25.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-09-18T14:46:25.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-09-18T14:46:25.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-09-18T14:46:26.175 INFO:tasks.workunit.client.0.smithi047.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-18T14:46:26.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-09-18T14:46:26.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-09-18T14:46:26.589 INFO:tasks.workunit.client.0.smithi047.stderr:marked down osd.0. 2024-09-18T14:46:26.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-09-18T14:46:36.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-09-18T14:46:36.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-09-18T14:46:36.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-09-18T14:46:37.112 INFO:tasks.workunit.client.0.smithi047.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-18T14:46:37.112 INFO:tasks.workunit.client.0.smithi047.stdout:-1 0.29306 root default 2024-09-18T14:46:37.112 INFO:tasks.workunit.client.0.smithi047.stdout:-3 0.29306 host smithi047 2024-09-18T14:46:37.112 INFO:tasks.workunit.client.0.smithi047.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-18T14:46:37.112 INFO:tasks.workunit.client.0.smithi047.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-09-18T14:46:37.112 INFO:tasks.workunit.client.0.smithi047.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-09-18T14:46:37.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-09-18T14:46:37.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-09-18T14:46:37.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-09-18T14:46:37.484 INFO:tasks.workunit.client.0.smithi047.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-18T14:46:37.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-09-18T14:46:37.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-09-18T14:46:37.872 INFO:tasks.workunit.client.0.smithi047.stderr:marked down osd.0. 2024-09-18T14:46:37.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-09-18T14:46:47.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-09-18T14:46:47.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-09-18T14:46:48.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-09-18T14:46:48.381 INFO:tasks.workunit.client.0.smithi047.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-18T14:46:48.381 INFO:tasks.workunit.client.0.smithi047.stdout:-1 0.29306 root default 2024-09-18T14:46:48.381 INFO:tasks.workunit.client.0.smithi047.stdout:-3 0.29306 host smithi047 2024-09-18T14:46:48.381 INFO:tasks.workunit.client.0.smithi047.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-18T14:46:48.381 INFO:tasks.workunit.client.0.smithi047.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-09-18T14:46:48.381 INFO:tasks.workunit.client.0.smithi047.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-09-18T14:46:48.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-09-18T14:46:48.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-09-18T14:46:48.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-09-18T14:46:48.752 INFO:tasks.workunit.client.0.smithi047.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-18T14:46:48.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-09-18T14:46:48.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-09-18T14:46:49.141 INFO:tasks.workunit.client.0.smithi047.stderr:marked down osd.0. 2024-09-18T14:46:49.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-09-18T14:46:59.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-09-18T14:46:59.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-09-18T14:46:59.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-09-18T14:46:59.659 INFO:tasks.workunit.client.0.smithi047.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-18T14:46:59.659 INFO:tasks.workunit.client.0.smithi047.stdout:-1 0.29306 root default 2024-09-18T14:46:59.660 INFO:tasks.workunit.client.0.smithi047.stdout:-3 0.29306 host smithi047 2024-09-18T14:46:59.660 INFO:tasks.workunit.client.0.smithi047.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-18T14:46:59.660 INFO:tasks.workunit.client.0.smithi047.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-09-18T14:46:59.660 INFO:tasks.workunit.client.0.smithi047.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-09-18T14:46:59.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-09-18T14:46:59.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-09-18T14:46:59.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-09-18T14:47:00.089 INFO:tasks.workunit.client.0.smithi047.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-18T14:47:00.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-09-18T14:47:00.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-09-18T14:47:00.479 INFO:tasks.workunit.client.0.smithi047.stderr:marked down osd.0. 2024-09-18T14:47:00.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-09-18T14:47:10.492 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: ceph osd tree 2024-09-18T14:47:10.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: grep down 2024-09-18T14:47:10.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: grep osd.0 2024-09-18T14:47:10.911 INFO:tasks.workunit.client.0.smithi047.stdout: 0 hdd 0.09769 osd.0 down 1.00000 1.00000 2024-09-18T14:47:10.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2024-09-18T14:47:10.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-09-18T14:47:10.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:47:10.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-09-18T14:47:10.912 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:47:10.912 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:47:10.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:47:10.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:47:10.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:47:11.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:47:11.019 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:47:11.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:47:11.020 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:47:11.021 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:47:11.021 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:47:11.021 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:47:11.022 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:47:11.022 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:47:11.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:47:11.023 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:47:11.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:47:11.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:47:11.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-09-18T14:47:11.040 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:47:11.040 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:11.040 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:47:11.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.188945 2024-09-18T14:47:11.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:47:11.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:47:11.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2024-09-18T14:47:11.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2024-09-18T14:47:11.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2024-09-18T14:47:11.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2024-09-18T14:47:11.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-09-18T14:47:11.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:47:11.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-09-18T14:47:11.042 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:47:11.043 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:47:11.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:47:11.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:47:11.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:47:11.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:47:11.043 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:47:11.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:47:11.044 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:47:11.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:47:11.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:47:11.046 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:47:11.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:47:11.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:47:11.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:47:11.047 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:47:11.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:47:11.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:47:11.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-09-18T14:47:11.050 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:47:11.050 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:11.050 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:47:11.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.188945 2024-09-18T14:47:11.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:47:11.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:47:11.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2024-09-18T14:47:11.052 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T14:47:11.052 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:11.053 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:47:11.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.188945 2024-09-18T14:47:11.055 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T14:47:11.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T14:47:11.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T14:47:11.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2024-09-18T14:47:11.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_osd_stop td/osd-markdown 2024-09-18T14:47:11.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:132: TEST_osd_stop: local dir=td/osd-markdown 2024-09-18T14:47:11.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:134: TEST_osd_stop: run_mon td/osd-markdown a 2024-09-18T14:47:11.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2024-09-18T14:47:11.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T14:47:11.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T14:47:11.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T14:47:11.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2024-09-18T14:47:11.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2024-09-18T14:47:11.423 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T14:47:11.423 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:47:11.423 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:47:11.423 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:47:11.423 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:11.423 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:47:11.423 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188945/$cluster-$name.asok' 2024-09-18T14:47:11.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T14:47:11.648 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T14:47:11.650 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T14:47:11.650 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:47:11.650 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:47:11.650 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T14:47:11.650 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T14:47:11.650 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:47:11.650 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:47:11.650 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:47:11.650 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:11.651 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:11.651 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:47:11.651 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.188945/ceph-mon.a.asok 2024-09-18T14:47:11.651 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:47:11.651 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.188945/ceph-mon.a.asok config get fsid 2024-09-18T14:47:11.728 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T14:47:11.728 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:47:11.728 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:47:11.728 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T14:47:11.729 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T14:47:11.729 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:47:11.729 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:47:11.729 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:47:11.729 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:47:11.729 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:11.729 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:47:11.729 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.188945/ceph-mon.a.asok 2024-09-18T14:47:11.729 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:47:11.729 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.188945/ceph-mon.a.asok config get mon_host 2024-09-18T14:47:11.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:135: TEST_osd_stop: run_mgr td/osd-markdown x 2024-09-18T14:47:11.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2024-09-18T14:47:11.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T14:47:11.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T14:47:11.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T14:47:11.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2024-09-18T14:47:11.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T14:47:12.645 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T14:47:12.645 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:47:12.645 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:47:12.646 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:47:12.646 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:12.646 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:47:12.646 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188945/$cluster-$name.asok' 2024-09-18T14:47:12.646 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:47:12.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:47:12.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:136: TEST_osd_stop: run_osd td/osd-markdown 0 2024-09-18T14:47:12.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-18T14:47:12.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:47:12.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T14:47:12.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:47:12.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2024-09-18T14:47:12.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-18T14:47:12.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:47:12.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:47:12.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:47:12.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2024-09-18T14:47:12.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2024-09-18T14:47:12.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:47:12.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:47:12.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-18T14:47:12.674 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:47:12.675 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:47:12.675 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:47:12.675 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:47:12.675 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:12.675 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:47:12.675 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188945/$cluster-$name.asok' 2024-09-18T14:47:12.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' 2024-09-18T14:47:12.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:47:12.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:47:12.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:47:12.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-18T14:47:12.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-18T14:47:12.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:47:12.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:47:12.676 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:47:12.676 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:47:12.676 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:47:12.676 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:47:12.676 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2024-09-18T14:47:12.676 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:47:12.676 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=727b50e8-8001-479c-ba82-ae26d38acd88 2024-09-18T14:47:12.676 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 727b50e8-8001-479c-ba82-ae26d38acd88' 2024-09-18T14:47:12.676 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 727b50e8-8001-479c-ba82-ae26d38acd88 2024-09-18T14:47:12.677 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:47:12.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDw5+pm4d7KKBAA5o/0yKL1Ej1Rl2A1FlHSGA== 2024-09-18T14:47:12.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDw5+pm4d7KKBAA5o/0yKL1Ej1Rl2A1FlHSGA=="}' 2024-09-18T14:47:12.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 727b50e8-8001-479c-ba82-ae26d38acd88 -i td/osd-markdown/0/new.json 2024-09-18T14:47:12.988 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:47:12.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2024-09-18T14:47:12.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDw5+pm4d7KKBAA5o/0yKL1Ej1Rl2A1FlHSGA== --osd-uuid 727b50e8-8001-479c-ba82-ae26d38acd88 2024-09-18T14:47:13.022 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:47:13.021+0000 7f6ff431d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:47:13.033 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:47:13.032+0000 7f6ff431d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:47:13.034 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:47:13.033+0000 7f6ff431d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:47:13.035 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:47:13.034+0000 7f6ff431d440 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-18T14:47:13.035 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:47:13.034+0000 7f6ff431d440 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-18T14:47:13.035 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:47:13.034+0000 7f6ff431d440 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-18T14:47:13.073 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:47:13.072+0000 7f6ff431d440 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2024-09-18T14:47:18.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2024-09-18T14:47:18.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:47:18.412 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T14:47:18.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T14:47:18.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:47:18.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T14:47:18.921 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T14:47:18.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:47:18.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:47:18.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:47:18.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:47:18.941 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:47:18.940+0000 7ff2cd721440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:47:18.943 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:47:18.942+0000 7ff2cd721440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:47:18.944 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:47:18.943+0000 7ff2cd721440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:47:19.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T14:47:19.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:47:19.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:47:19.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:47:19.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:47:19.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:47:19.283 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:47:19.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:47:19.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:47:19.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:47:19.461 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:47:19.460+0000 7ff2cd721440 -1 Falling back to public interface 2024-09-18T14:47:19.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:47:20.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:47:20.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:47:20.695 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:47:20.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:47:20.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:47:20.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:47:21.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:47:21.918 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:47:21.916+0000 7ff2cd721440 -1 osd.0 0 log_to_monitors true 2024-09-18T14:47:22.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:47:22.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:47:22.130 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:47:22.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:47:22.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:47:22.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:47:22.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:47:23.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:47:23.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:47:23.610 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:47:23.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:47:23.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:47:23.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:47:24.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:47:25.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:47:25.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:47:25.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:47:25.015 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:47:25.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:47:25.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:47:25.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:47:26.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:47:26.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:47:26.398 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:47:26.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:47:26.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:47:26.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:47:26.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:47:27.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:47:27.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:47:27.918 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:47:27.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:47:27.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:47:27.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:47:28.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:47:29.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:47:29.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:47:29.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:47:29.313 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:47:29.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:47:29.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:47:29.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:47:30.746 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:47:30.746 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:47:30.747 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:47:30.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:47:30.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:47:30.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:47:31.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:47:32.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:47:32.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:47:32.143 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:47:32.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:47:32.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:47:32.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:47:32.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:47:33.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:47:33.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:47:33.593 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:47:33.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:47:33.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:47:33.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:47:34.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:47:35.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:47:35.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:47:35.379 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:47:35.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:47:35.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:47:35.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:47:35.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:47:36.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:47:36.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:47:36.907 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:47:36.907 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:47:36.907 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:47:36.907 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:47:37.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:47:38.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:47:38.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:47:38.586 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:47:38.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:47:38.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:47:38.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:47:39.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:47:40.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:47:40.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:47:40.030 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:47:40.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:47:40.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:47:40.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:47:40.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:47:41.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:47:41.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:47:41.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:47:41.750 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:47:41.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:47:41.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:47:42.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:47:43.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:47:43.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:47:43.411 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:47:43.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:47:43.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:47:43.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:47:43.806 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/208036,v1:127.0.0.1:6803/208036] [v2:127.0.0.1:6804/208036,v1:127.0.0.1:6805/208036] exists,up 727b50e8-8001-479c-ba82-ae26d38acd88 2024-09-18T14:47:43.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:47:43.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:47:43.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:47:43.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:137: TEST_osd_stop: run_osd td/osd-markdown 1 2024-09-18T14:47:43.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-18T14:47:43.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:47:43.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T14:47:43.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:47:43.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2024-09-18T14:47:43.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-18T14:47:43.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:47:43.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:47:43.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:47:43.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2024-09-18T14:47:43.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2024-09-18T14:47:43.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:47:43.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:47:43.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-18T14:47:43.812 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:47:43.812 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:47:43.812 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:47:43.812 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:47:43.812 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:47:43.812 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:47:43.812 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188945/$cluster-$name.asok' 2024-09-18T14:47:43.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' 2024-09-18T14:47:43.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:47:43.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:47:43.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:47:43.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-18T14:47:43.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-18T14:47:43.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:47:43.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:47:43.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:47:43.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:47:43.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:47:43.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:47:43.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2024-09-18T14:47:43.813 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:47:43.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=618bdaaa-e114-449c-9c2e-4fc25294323c 2024-09-18T14:47:43.813 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 618bdaaa-e114-449c-9c2e-4fc25294323c 2024-09-18T14:47:43.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 618bdaaa-e114-449c-9c2e-4fc25294323c' 2024-09-18T14:47:43.814 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:47:43.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAP6Opmqk4NMRAAOhPzpO0Q/u/qyQAyx6bzwA== 2024-09-18T14:47:43.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAP6Opmqk4NMRAAOhPzpO0Q/u/qyQAyx6bzwA=="}' 2024-09-18T14:47:43.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 618bdaaa-e114-449c-9c2e-4fc25294323c -i td/osd-markdown/1/new.json 2024-09-18T14:47:44.427 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:47:44.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2024-09-18T14:47:44.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAP6Opmqk4NMRAAOhPzpO0Q/u/qyQAyx6bzwA== --osd-uuid 618bdaaa-e114-449c-9c2e-4fc25294323c 2024-09-18T14:47:44.462 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:47:44.461+0000 7f35f143b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:47:44.486 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:47:44.486+0000 7f35f143b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:47:44.487 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:47:44.487+0000 7f35f143b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:47:44.488 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:47:44.487+0000 7f35f143b440 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-18T14:47:44.488 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:47:44.487+0000 7f35f143b440 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-18T14:47:44.488 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:47:44.487+0000 7f35f143b440 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-18T14:47:44.578 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:47:44.577+0000 7f35f143b440 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2024-09-18T14:47:50.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2024-09-18T14:47:50.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:47:50.253 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T14:47:50.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T14:47:50.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:47:50.763 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T14:47:50.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T14:47:50.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:47:50.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:47:50.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:47:50.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:47:50.784 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:47:50.783+0000 7f6857a1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:47:50.786 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:47:50.785+0000 7f6857a1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:47:50.787 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:47:50.786+0000 7f6857a1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:47:51.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T14:47:51.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:47:51.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:47:51.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:47:51.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:47:51.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:47:51.171 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:47:51.172 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:47:51.172 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:47:51.172 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:47:51.305 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:47:51.304+0000 7f6857a1d440 -1 Falling back to public interface 2024-09-18T14:47:51.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:47:52.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:47:52.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:47:52.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:47:52.582 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:47:52.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:47:52.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:47:52.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:47:53.821 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:47:53.820+0000 7f6857a1d440 -1 osd.1 0 log_to_monitors true 2024-09-18T14:47:53.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:47:53.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:47:53.983 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:47:53.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:47:53.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:47:53.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:47:54.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:47:55.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:47:55.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:47:55.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:47:55.574 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:47:55.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:47:55.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:47:55.948 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:47:56.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:47:56.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:47:56.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:47:56.950 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:47:56.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:47:56.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:47:57.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:47:58.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:47:58.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:47:58.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:47:58.359 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:47:58.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:47:58.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:47:58.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:47:59.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:47:59.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:47:59.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:47:59.792 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:47:59.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:47:59.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:48:00.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:48:01.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:48:01.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:48:01.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:48:01.170 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:48:01.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:48:01.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:48:01.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:48:02.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:48:02.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:48:02.598 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:48:02.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:48:02.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:48:02.599 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:48:03.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:48:04.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:48:04.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:48:04.038 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:48:04.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:48:04.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:48:04.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:48:04.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:48:05.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:48:05.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:48:05.416 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:48:05.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:48:05.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:48:05.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:48:06.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:48:07.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:48:07.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:48:07.099 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:48:07.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:48:07.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:48:07.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:48:07.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:48:08.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:48:08.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:48:08.790 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:48:08.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:48:08.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:48:08.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:48:09.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:48:10.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:48:10.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:48:10.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:48:10.680 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:48:10.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:48:10.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:48:11.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:48:12.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:48:12.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:48:12.239 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:48:12.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:48:12.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:48:12.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:48:12.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:48:13.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:48:13.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:48:13.783 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:48:13.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:48:13.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:48:13.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:48:14.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:48:15.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:48:15.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:48:15.738 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:48:15.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:48:15.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:48:15.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:48:16.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:48:17.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:48:17.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:48:17.216 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T14:48:17.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:48:17.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:48:17.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:48:17.598 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/209765,v1:127.0.0.1:6811/209765] [v2:127.0.0.1:6812/209765,v1:127.0.0.1:6813/209765] exists,up 618bdaaa-e114-449c-9c2e-4fc25294323c 2024-09-18T14:48:17.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:48:17.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:48:17.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:48:17.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:138: TEST_osd_stop: run_osd td/osd-markdown 2 2024-09-18T14:48:17.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-18T14:48:17.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:48:17.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T14:48:17.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:48:17.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2024-09-18T14:48:17.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-18T14:48:17.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:48:17.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:48:17.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:48:17.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2024-09-18T14:48:17.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2024-09-18T14:48:17.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:48:17.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:48:17.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-18T14:48:17.603 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:48:17.603 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:48:17.603 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:48:17.603 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:48:17.603 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:48:17.603 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:48:17.603 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.188945/$cluster-$name.asok' 2024-09-18T14:48:17.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' 2024-09-18T14:48:17.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:48:17.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:48:17.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:48:17.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-18T14:48:17.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-18T14:48:17.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:48:17.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:48:17.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:48:17.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:48:17.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:48:17.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:48:17.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2024-09-18T14:48:17.604 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:48:17.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f4566012-ad2e-4eb3-8680-63c96c7dce8f 2024-09-18T14:48:17.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f4566012-ad2e-4eb3-8680-63c96c7dce8f' 2024-09-18T14:48:17.605 INFO:tasks.workunit.client.0.smithi047.stdout:add osd2 f4566012-ad2e-4eb3-8680-63c96c7dce8f 2024-09-18T14:48:17.605 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:48:17.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAx6OpmCe+WJBAA9qi+6IezUq9G1GS5dfnidQ== 2024-09-18T14:48:17.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAx6OpmCe+WJBAA9qi+6IezUq9G1GS5dfnidQ=="}' 2024-09-18T14:48:17.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f4566012-ad2e-4eb3-8680-63c96c7dce8f -i td/osd-markdown/2/new.json 2024-09-18T14:48:18.105 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:48:18.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2024-09-18T14:48:18.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAx6OpmCe+WJBAA9qi+6IezUq9G1GS5dfnidQ== --osd-uuid f4566012-ad2e-4eb3-8680-63c96c7dce8f 2024-09-18T14:48:18.137 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:48:18.136+0000 7f62e2929440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:48:18.139 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:48:18.138+0000 7f62e2929440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:48:18.140 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:48:18.139+0000 7f62e2929440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:48:18.140 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:48:18.139+0000 7f62e2929440 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-18T14:48:18.140 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:48:18.139+0000 7f62e2929440 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-18T14:48:18.140 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:48:18.139+0000 7f62e2929440 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-18T14:48:18.193 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:48:18.192+0000 7f62e2929440 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2024-09-18T14:48:24.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2024-09-18T14:48:24.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:48:24.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T14:48:24.095 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd2 key to auth repository 2024-09-18T14:48:24.095 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:48:24.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T14:48:24.607 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T14:48:24.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0ebe85ed-ad49-46af-8a61-928928b0081b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.188945/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:48:24.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:48:24.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:48:24.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:48:24.627 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:48:24.626+0000 7f6159e12440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:48:24.650 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:48:24.649+0000 7f6159e12440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:48:24.652 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:48:24.650+0000 7f6159e12440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:48:25.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T14:48:25.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:48:25.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T14:48:25.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:48:25.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:48:25.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:48:25.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:48:25.032 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:48:25.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:48:25.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:48:25.183 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:48:25.182+0000 7f6159e12440 -1 Falling back to public interface 2024-09-18T14:48:25.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:48:26.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:48:26.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:48:26.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:48:26.442 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:48:26.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:48:26.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:48:26.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:48:27.706 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:48:27.705+0000 7f6159e12440 -1 osd.2 0 log_to_monitors true 2024-09-18T14:48:27.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:48:27.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:48:27.835 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:48:27.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:48:27.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:48:27.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:48:28.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:48:29.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:48:29.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:48:29.364 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:48:29.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:48:29.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:48:29.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:48:29.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:48:30.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:48:30.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:48:30.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:48:30.771 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:48:30.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:48:30.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:48:31.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:48:32.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:48:32.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:48:32.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:48:32.297 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:48:32.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:48:32.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:48:32.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:48:33.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:48:33.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:48:33.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:48:33.786 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:48:33.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:48:33.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:48:34.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:48:35.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:48:35.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:48:35.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:48:35.192 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:48:35.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:48:35.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:48:35.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:48:36.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:48:36.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:48:36.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:48:36.604 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:48:36.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:48:36.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:48:37.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:48:38.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:48:38.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:48:38.147 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:48:38.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:48:38.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:48:38.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:48:38.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:48:39.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:48:39.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:48:39.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:48:39.579 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:48:39.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:48:39.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:48:39.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:48:40.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:48:41.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:48:41.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:48:41.000 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:48:41.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:48:41.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:48:42.021 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:48:43.022 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:48:43.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:48:43.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:48:43.023 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:48:43.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:48:43.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:48:43.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:48:44.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:48:44.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:48:44.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:48:44.692 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:48:44.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:48:44.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:48:45.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:48:46.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:48:46.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:48:46.515 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:48:46.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:48:46.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:48:46.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:48:47.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:48:48.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:48:48.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:48:48.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:48:48.504 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:48:48.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:48:48.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:48:48.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:48:49.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:48:49.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:48:49.918 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:48:49.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:48:49.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:48:49.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:48:50.294 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/211519,v1:127.0.0.1:6819/211519] [v2:127.0.0.1:6820/211519,v1:127.0.0.1:6821/211519] exists,up f4566012-ad2e-4eb3-8680-63c96c7dce8f 2024-09-18T14:48:50.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:48:50.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:48:50.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:48:50.295 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:139: TEST_osd_stop: cat td/osd-markdown/osd.0.pid 2024-09-18T14:48:50.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:139: TEST_osd_stop: osd_0_pid=208036 2024-09-18T14:48:50.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:140: TEST_osd_stop: ps -p 208036 2024-09-18T14:48:50.302 INFO:tasks.workunit.client.0.smithi047.stdout: PID TTY TIME CMD 2024-09-18T14:48:50.303 INFO:tasks.workunit.client.0.smithi047.stdout: 208036 ? 00:00:00 ceph-osd 2024-09-18T14:48:50.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: ceph osd tree 2024-09-18T14:48:50.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: grep osd.0 2024-09-18T14:48:50.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: grep up 2024-09-18T14:48:50.771 INFO:tasks.workunit.client.0.smithi047.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-18T14:48:50.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:143: TEST_osd_stop: ceph osd stop osd.0 2024-09-18T14:48:51.323 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already stopped. 2024-09-18T14:48:51.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:144: TEST_osd_stop: sleep 15 2024-09-18T14:49:06.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: ceph osd tree 2024-09-18T14:49:06.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: grep down 2024-09-18T14:49:06.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: grep osd.0 2024-09-18T14:49:06.766 INFO:tasks.workunit.client.0.smithi047.stdout: 0 hdd 0.09769 osd.0 down 1.00000 1.00000 2024-09-18T14:49:06.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:146: TEST_osd_stop: ps -p 208036 2024-09-18T14:49:06.772 INFO:tasks.workunit.client.0.smithi047.stdout: PID TTY TIME CMD 2024-09-18T14:49:06.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2024-09-18T14:49:06.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-09-18T14:49:06.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:49:06.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-09-18T14:49:06.773 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:49:06.773 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:49:06.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:49:06.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:49:06.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:49:06.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:49:06.881 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:49:06.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:49:06.882 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:49:06.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:49:06.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:49:06.883 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:49:06.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:49:06.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:49:06.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:49:06.884 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:49:06.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:49:06.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:49:06.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-09-18T14:49:06.898 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:49:06.898 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:06.898 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:49:06.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.188945 2024-09-18T14:49:06.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:49:06.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:49:06.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T14:49:06.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/osd-markdown 0 2024-09-18T14:49:06.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-09-18T14:49:06.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T14:49:06.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-09-18T14:49:06.901 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:49:06.901 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:49:06.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:49:06.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:49:06.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:49:06.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:49:06.902 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:49:06.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:49:06.903 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:49:06.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:49:06.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:49:06.904 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:49:06.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:49:06.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:49:06.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:49:06.905 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:49:06.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:49:06.907 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T14:49:06.907 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-09-18T14:49:06.908 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:49:06.908 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:06.908 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.188945 2024-09-18T14:49:06.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.188945 2024-09-18T14:49:06.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:49:06.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:49:06.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T14:49:06.911 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T14:49:06.911 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T14:49:06.936 INFO:tasks.workunit:Running workunit osd/osd-reactivate.sh... 2024-09-18T14:49:06.936 DEBUG:teuthology.orchestra.run.smithi047:workunit test osd/osd-reactivate.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh 2024-09-18T14:49:06.998 INFO:tasks.workunit.client.0.smithi047.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T14:49:07.003 INFO:tasks.workunit.client.0.smithi047.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T14:49:07.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T14:49:07.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T14:49:07.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T14:49:07.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T14:49:07.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T14:49:07.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T14:49:07.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T14:49:07.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T14:49:07.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/osd-reactivate 2024-09-18T14:49:07.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:19: run: local dir=td/osd-reactivate 2024-09-18T14:49:07.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:20: run: shift 2024-09-18T14:49:07.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:22: run: export CEPH_MON=127.0.0.1:7122 2024-09-18T14:49:07.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:22: run: CEPH_MON=127.0.0.1:7122 2024-09-18T14:49:07.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:23: run: export CEPH_ARGS 2024-09-18T14:49:07.004 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:24: run: uuidgen 2024-09-18T14:49:07.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:24: run: CEPH_ARGS+='--fsid=1e98c68e-e7dd-400f-ac08-fcb8701ccef0 --auth-supported=none ' 2024-09-18T14:49:07.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:25: run: CEPH_ARGS+='--mon-host=127.0.0.1:7122 ' 2024-09-18T14:49:07.005 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: set 2024-09-18T14:49:07.005 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-18T14:49:07.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: local funcs=TEST_reactivate 2024-09-18T14:49:07.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:28: run: for func in $funcs 2024-09-18T14:49:07.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:29: run: setup td/osd-reactivate 2024-09-18T14:49:07.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-reactivate 2024-09-18T14:49:07.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-reactivate 2024-09-18T14:49:07.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2024-09-18T14:49:07.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:49:07.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2024-09-18T14:49:07.007 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:49:07.007 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:49:07.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:49:07.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:49:07.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:49:07.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:49:07.008 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:49:07.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:49:07.008 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:49:07.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:49:07.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:49:07.010 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:49:07.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:49:07.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:49:07.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:49:07.011 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:49:07.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:49:07.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:49:07.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2024-09-18T14:49:07.014 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:49:07.014 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:07.014 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.212569 2024-09-18T14:49:07.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.212569 2024-09-18T14:49:07.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:49:07.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:49:07.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-reactivate 2024-09-18T14:49:07.016 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T14:49:07.016 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:07.016 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.212569 2024-09-18T14:49:07.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.212569 2024-09-18T14:49:07.018 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T14:49:07.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T14:49:07.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T14:49:07.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T14:49:07.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-reactivate 1' TERM HUP INT 2024-09-18T14:49:07.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:30: run: TEST_reactivate td/osd-reactivate 2024-09-18T14:49:07.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:36: TEST_reactivate: local dir=td/osd-reactivate 2024-09-18T14:49:07.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:38: TEST_reactivate: run_mon td/osd-reactivate a 2024-09-18T14:49:07.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-reactivate 2024-09-18T14:49:07.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T14:49:07.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T14:49:07.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T14:49:07.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-reactivate/a 2024-09-18T14:49:07.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-reactivate/a --run-dir=td/osd-reactivate 2024-09-18T14:49:07.378 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T14:49:07.378 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:49:07.378 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:49:07.378 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:49:07.378 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:07.378 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.212569 2024-09-18T14:49:07.378 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.212569/$cluster-$name.asok' 2024-09-18T14:49:07.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-reactivate/a '--log-file=td/osd-reactivate/$name.log' '--admin-socket=/tmp/ceph-asok.212569/$cluster-$name.asok' --mon-cluster-log-file=td/osd-reactivate/log --run-dir=td/osd-reactivate '--pid-file=td/osd-reactivate/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T14:49:07.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T14:49:07.614 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T14:49:07.614 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:49:07.614 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:49:07.614 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T14:49:07.614 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T14:49:07.615 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:49:07.615 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:49:07.615 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:49:07.615 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:07.615 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:07.615 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.212569 2024-09-18T14:49:07.615 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.212569/ceph-mon.a.asok 2024-09-18T14:49:07.615 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:49:07.616 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.212569/ceph-mon.a.asok config get fsid 2024-09-18T14:49:07.695 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T14:49:07.695 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:49:07.695 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:49:07.695 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T14:49:07.695 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T14:49:07.695 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:49:07.696 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:49:07.696 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:49:07.696 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:07.696 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:07.696 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.212569 2024-09-18T14:49:07.696 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.212569/ceph-mon.a.asok 2024-09-18T14:49:07.696 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:49:07.696 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.212569/ceph-mon.a.asok config get mon_host 2024-09-18T14:49:07.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:39: TEST_reactivate: run_mgr td/osd-reactivate x 2024-09-18T14:49:07.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-reactivate 2024-09-18T14:49:07.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T14:49:07.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T14:49:07.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T14:49:07.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-reactivate/x 2024-09-18T14:49:07.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T14:49:08.618 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T14:49:08.618 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:49:08.618 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:49:08.618 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:49:08.619 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:08.619 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.212569 2024-09-18T14:49:08.619 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.212569/$cluster-$name.asok' 2024-09-18T14:49:08.619 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:49:08.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-reactivate/x '--log-file=td/osd-reactivate/$name.log' '--admin-socket=/tmp/ceph-asok.212569/$cluster-$name.asok' --run-dir=td/osd-reactivate '--pid-file=td/osd-reactivate/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:49:08.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:40: TEST_reactivate: run_osd td/osd-reactivate 0 2024-09-18T14:49:08.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-reactivate 2024-09-18T14:49:08.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:49:08.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T14:49:08.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:49:08.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-reactivate/0 2024-09-18T14:49:08.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1e98c68e-e7dd-400f-ac08-fcb8701ccef0 --auth-supported=none --mon-host=127.0.0.1:7122 ' 2024-09-18T14:49:08.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:49:08.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:49:08.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:49:08.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-reactivate/0' 2024-09-18T14:49:08.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-reactivate/0/journal' 2024-09-18T14:49:08.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:49:08.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:49:08.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-reactivate' 2024-09-18T14:49:08.644 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:49:08.644 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:49:08.644 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:49:08.644 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:49:08.644 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:08.644 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.212569 2024-09-18T14:49:08.644 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.212569/$cluster-$name.asok' 2024-09-18T14:49:08.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.212569/$cluster-$name.asok' 2024-09-18T14:49:08.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:49:08.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:49:08.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:49:08.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-reactivate/$name.log' 2024-09-18T14:49:08.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-reactivate/$name.pid' 2024-09-18T14:49:08.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:49:08.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:49:08.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:49:08.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:49:08.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:49:08.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:49:08.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-reactivate/0 2024-09-18T14:49:08.646 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:49:08.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e20ec933-70cc-4f94-9289-7862d1ea29a3 2024-09-18T14:49:08.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e20ec933-70cc-4f94-9289-7862d1ea29a3' 2024-09-18T14:49:08.646 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 e20ec933-70cc-4f94-9289-7862d1ea29a3 2024-09-18T14:49:08.646 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:49:08.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBk6OpmLfcnJxAA/CwCJd16lCu77jDlS7crkg== 2024-09-18T14:49:08.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBk6OpmLfcnJxAA/CwCJd16lCu77jDlS7crkg=="}' 2024-09-18T14:49:08.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e20ec933-70cc-4f94-9289-7862d1ea29a3 -i td/osd-reactivate/0/new.json 2024-09-18T14:49:08.969 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:49:08.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-reactivate/0/new.json 2024-09-18T14:49:08.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1e98c68e-e7dd-400f-ac08-fcb8701ccef0 --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.212569/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBk6OpmLfcnJxAA/CwCJd16lCu77jDlS7crkg== --osd-uuid e20ec933-70cc-4f94-9289-7862d1ea29a3 2024-09-18T14:49:09.002 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:49:09.001+0000 7fe0a0f2a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:49:09.013 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:49:09.012+0000 7fe0a0f2a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:49:09.014 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:49:09.014+0000 7fe0a0f2a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:49:09.015 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:49:09.014+0000 7fe0a0f2a440 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2024-09-18T14:49:09.015 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:49:09.014+0000 7fe0a0f2a440 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2024-09-18T14:49:09.015 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:49:09.014+0000 7fe0a0f2a440 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2024-09-18T14:49:09.054 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:49:09.053+0000 7fe0a0f2a440 -1 bluestore(td/osd-reactivate/0) _read_fsid unparsable uuid 2024-09-18T14:49:14.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-reactivate/0/keyring 2024-09-18T14:49:14.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:49:14.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T14:49:14.410 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T14:49:14.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-reactivate/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:49:14.936 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T14:49:14.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T14:49:14.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1e98c68e-e7dd-400f-ac08-fcb8701ccef0 --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.212569/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:49:14.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:49:14.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:49:14.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:49:14.957 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:49:14.956+0000 7fabef123440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:49:14.959 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:49:14.958+0000 7fabef123440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:49:14.960 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:49:14.959+0000 7fabef123440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:49:15.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T14:49:15.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:49:15.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:49:15.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:49:15.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:49:15.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:49:15.375 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:49:15.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:49:15.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:49:15.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:49:15.502 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:49:15.501+0000 7fabef123440 -1 Falling back to public interface 2024-09-18T14:49:15.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:49:16.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:49:16.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:49:16.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:49:16.764 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:49:16.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:49:16.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:49:17.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:49:17.998 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:49:17.997+0000 7fabef123440 -1 osd.0 0 log_to_monitors true 2024-09-18T14:49:18.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:49:18.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:49:18.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:49:18.179 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:49:18.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:49:18.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:49:18.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:49:19.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:49:19.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:49:19.692 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:49:19.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:49:19.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:49:19.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:49:20.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:49:21.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:49:21.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:49:21.165 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:49:21.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T14:49:21.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:49:21.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:49:21.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:49:22.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:49:22.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:49:22.664 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T14:49:22.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T14:49:22.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:49:22.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:49:23.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:49:24.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:49:24.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:49:24.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T14:49:24.068 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T14:49:24.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:49:24.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:49:24.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:49:25.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:49:25.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:49:25.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T14:49:25.453 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T14:49:25.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:49:25.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:49:25.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:49:26.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:49:26.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:49:26.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T14:49:26.867 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T14:49:26.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:49:26.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:49:27.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:49:28.242 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:49:28.242 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:49:28.242 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T14:49:28.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T14:49:28.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:49:28.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:49:28.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:49:29.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:49:29.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:49:29.679 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T14:49:29.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T14:49:29.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:49:29.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:49:30.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:49:31.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:49:31.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:49:31.481 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T14:49:31.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T14:49:31.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:49:31.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:49:32.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:49:33.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:49:33.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:49:33.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T14:49:33.051 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T14:49:33.052 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:49:33.052 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:49:33.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:49:34.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:49:34.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:49:34.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T14:49:34.791 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T14:49:34.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:49:34.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:49:35.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:49:36.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:49:36.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:49:36.458 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T14:49:36.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T14:49:36.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:49:36.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:49:37.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:49:38.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:49:38.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:49:38.186 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T14:49:38.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T14:49:38.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:49:38.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:49:38.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:49:39.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:49:39.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:49:39.640 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T14:49:39.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T14:49:39.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:49:39.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:49:40.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:49:41.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:49:41.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:49:41.073 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T14:49:41.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T14:49:41.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:49:41.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:49:41.477 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/213660,v1:127.0.0.1:6803/213660] [v2:127.0.0.1:6804/213660,v1:127.0.0.1:6805/213660] exists,up e20ec933-70cc-4f94-9289-7862d1ea29a3 2024-09-18T14:49:41.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:49:41.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:49:41.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:49:41.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:42: TEST_reactivate: kill_daemons td/osd-reactivate TERM osd 2024-09-18T14:49:41.478 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:49:41.478 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:49:41.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:49:41.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:49:41.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:49:41.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:49:41.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:44: TEST_reactivate: ready_path=td/osd-reactivate/0/ready 2024-09-18T14:49:41.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:45: TEST_reactivate: activate_path=td/osd-reactivate/0/active 2024-09-18T14:49:41.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:47: TEST_reactivate: rm -rf td/osd-reactivate/0/ready td/osd-reactivate/0/active 2024-09-18T14:49:41.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:48: TEST_reactivate: activate_osd td/osd-reactivate 0 2024-09-18T14:49:41.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-reactivate 2024-09-18T14:49:41.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T14:49:41.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T14:49:41.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T14:49:41.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-reactivate/0 2024-09-18T14:49:41.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1e98c68e-e7dd-400f-ac08-fcb8701ccef0 --auth-supported=none --mon-host=127.0.0.1:7122 ' 2024-09-18T14:49:41.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:49:41.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:49:41.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:49:41.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-reactivate/0' 2024-09-18T14:49:41.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-reactivate/0/journal' 2024-09-18T14:49:41.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T14:49:41.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T14:49:41.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-reactivate' 2024-09-18T14:49:41.789 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T14:49:41.789 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:49:41.789 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:49:41.789 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:49:41.789 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:41.789 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.212569 2024-09-18T14:49:41.789 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.212569/$cluster-$name.asok' 2024-09-18T14:49:41.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.212569/$cluster-$name.asok' 2024-09-18T14:49:41.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:49:41.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-reactivate/$name.log' 2024-09-18T14:49:41.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-reactivate/$name.pid' 2024-09-18T14:49:41.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:49:41.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:49:41.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:49:41.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:49:41.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T14:49:41.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T14:49:41.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-reactivate/0 2024-09-18T14:49:41.790 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T14:49:41.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T14:49:41.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1e98c68e-e7dd-400f-ac08-fcb8701ccef0 --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.212569/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:49:41.791 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-reactivate/0/whoami 2024-09-18T14:49:41.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T14:49:41.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T14:49:41.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T14:49:41.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T14:49:41.809 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:49:41.808+0000 7f42c668d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:49:41.811 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:49:41.810+0000 7f42c668d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:49:41.812 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:49:41.811+0000 7f42c668d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:49:42.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T14:49:42.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:49:42.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:49:42.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:49:42.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:49:42.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:49:42.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:49:42.215 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:49:42.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:49:42.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:49:42.329 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:49:42.328+0000 7f42c668d440 -1 Falling back to public interface 2024-09-18T14:49:42.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:49:43.599 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:49:43.599 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:49:43.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:49:43.600 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:49:43.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:49:43.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:49:43.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:49:44.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:49:44.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:49:44.980 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:49:44.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:49:44.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:49:44.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:49:45.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:49:45.460 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:49:45.459+0000 7f42c668d440 -1 osd.0 6 log_to_monitors true 2024-09-18T14:49:46.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:49:46.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:49:46.380 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:49:46.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:49:46.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:49:46.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:49:46.792 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/214599,v1:127.0.0.1:6803/214599] [v2:127.0.0.1:6804/214599,v1:127.0.0.1:6805/214599] exists,up e20ec933-70cc-4f94-9289-7862d1ea29a3 2024-09-18T14:49:46.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:49:46.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:49:46.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:49:46.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:31: run: teardown td/osd-reactivate 2024-09-18T14:49:46.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2024-09-18T14:49:46.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:49:46.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2024-09-18T14:49:46.794 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:49:46.794 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:49:46.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:49:46.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:49:46.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:49:46.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:49:46.899 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:49:46.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:49:46.900 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:49:46.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:49:46.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:49:46.901 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:49:46.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:49:46.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:49:46.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:49:46.903 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:49:46.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:49:46.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:49:46.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2024-09-18T14:49:46.910 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:49:46.910 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:46.910 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.212569 2024-09-18T14:49:46.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.212569 2024-09-18T14:49:46.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:49:46.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:49:46.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T14:49:46.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/osd-reactivate 0 2024-09-18T14:49:46.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2024-09-18T14:49:46.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T14:49:46.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2024-09-18T14:49:46.912 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:49:46.913 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:49:46.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:49:46.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:49:46.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:49:46.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:49:46.913 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:49:46.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:49:46.915 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:49:46.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:49:46.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:49:46.916 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:49:46.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:49:46.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:49:46.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:49:46.917 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:49:46.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:49:46.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T14:49:46.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2024-09-18T14:49:46.920 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:49:46.920 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:46.920 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.212569 2024-09-18T14:49:46.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.212569 2024-09-18T14:49:46.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:49:46.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:49:46.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T14:49:46.922 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T14:49:46.922 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T14:49:46.987 INFO:tasks.workunit:Running workunit osd/osd-recovery-prio.sh... 2024-09-18T14:49:46.988 DEBUG:teuthology.orchestra.run.smithi047:workunit test osd/osd-recovery-prio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh 2024-09-18T14:49:47.050 INFO:tasks.workunit.client.0.smithi047.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T14:49:47.058 INFO:tasks.workunit.client.0.smithi047.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T14:49:47.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T14:49:47.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T14:49:47.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T14:49:47.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T14:49:47.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T14:49:47.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T14:49:47.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T14:49:47.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T14:49:47.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/osd-recovery-prio 2024-09-18T14:49:47.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:21: run: local dir=td/osd-recovery-prio 2024-09-18T14:49:47.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:22: run: shift 2024-09-18T14:49:47.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:25: run: export CEPH_MON=127.0.0.1:7114 2024-09-18T14:49:47.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:25: run: CEPH_MON=127.0.0.1:7114 2024-09-18T14:49:47.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:26: run: export CEPH_ARGS 2024-09-18T14:49:47.059 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:27: run: uuidgen 2024-09-18T14:49:47.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:27: run: CEPH_ARGS+='--fsid=5c1cb17f-d525-402f-8061-e33fb6e2528e --auth-supported=none ' 2024-09-18T14:49:47.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 ' 2024-09-18T14:49:47.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:31: run: CEPH_ARGS+='--osd-op-queue=wpq ' 2024-09-18T14:49:47.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:32: run: export objects=200 2024-09-18T14:49:47.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:32: run: objects=200 2024-09-18T14:49:47.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:33: run: export poolprefix=test 2024-09-18T14:49:47.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:33: run: poolprefix=test 2024-09-18T14:49:47.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:34: run: export FORCE_PRIO=255 2024-09-18T14:49:47.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:34: run: FORCE_PRIO=255 2024-09-18T14:49:47.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:35: run: export NORMAL_PRIO=190 2024-09-18T14:49:47.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:35: run: NORMAL_PRIO=190 2024-09-18T14:49:47.060 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:37: run: set 2024-09-18T14:49:47.060 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:37: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-18T14:49:47.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:37: run: local 'funcs=TEST_recovery_pool_priority 2024-09-18T14:49:47.062 INFO:tasks.workunit.client.0.smithi047.stderr:TEST_recovery_priority' 2024-09-18T14:49:47.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:38: run: for func in $funcs 2024-09-18T14:49:47.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:39: run: setup td/osd-recovery-prio 2024-09-18T14:49:47.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-prio 2024-09-18T14:49:47.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-prio 2024-09-18T14:49:47.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2024-09-18T14:49:47.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:49:47.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2024-09-18T14:49:47.063 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:49:47.063 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:49:47.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:49:47.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:49:47.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:49:47.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:49:47.064 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:49:47.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:49:47.064 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:49:47.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:49:47.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:49:47.065 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:49:47.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:49:47.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:49:47.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:49:47.066 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:49:47.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:49:47.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:49:47.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2024-09-18T14:49:47.068 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:49:47.068 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:47.068 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215238 2024-09-18T14:49:47.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.215238 2024-09-18T14:49:47.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:49:47.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:49:47.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-prio 2024-09-18T14:49:47.070 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T14:49:47.070 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:47.070 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215238 2024-09-18T14:49:47.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.215238 2024-09-18T14:49:47.073 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T14:49:47.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T14:49:47.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T14:49:47.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T14:49:47.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-prio 1' TERM HUP INT 2024-09-18T14:49:47.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:40: run: TEST_recovery_pool_priority td/osd-recovery-prio 2024-09-18T14:49:47.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:326: TEST_recovery_pool_priority: local dir=td/osd-recovery-prio 2024-09-18T14:49:47.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:327: TEST_recovery_pool_priority: local pools=3 2024-09-18T14:49:47.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:328: TEST_recovery_pool_priority: local OSDS=2 2024-09-18T14:49:47.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:330: TEST_recovery_pool_priority: run_mon td/osd-recovery-prio a 2024-09-18T14:49:47.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-prio 2024-09-18T14:49:47.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T14:49:47.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T14:49:47.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T14:49:47.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-prio/a 2024-09-18T14:49:47.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-prio/a --run-dir=td/osd-recovery-prio 2024-09-18T14:49:47.421 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T14:49:47.421 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:49:47.421 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:49:47.421 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:49:47.421 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:47.421 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215238 2024-09-18T14:49:47.421 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.215238/$cluster-$name.asok' 2024-09-18T14:49:47.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-prio/a '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.215238/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-prio/log --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T14:49:47.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T14:49:47.665 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T14:49:47.665 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:49:47.665 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:49:47.665 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T14:49:47.665 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T14:49:47.665 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:49:47.665 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:49:47.665 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:49:47.666 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:47.666 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:47.666 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215238 2024-09-18T14:49:47.666 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.215238/ceph-mon.a.asok 2024-09-18T14:49:47.666 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:49:47.666 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.215238/ceph-mon.a.asok config get fsid 2024-09-18T14:49:47.746 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T14:49:47.746 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:49:47.746 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:49:47.746 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T14:49:47.746 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T14:49:47.746 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:49:47.746 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:49:47.746 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:49:47.746 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:49:47.747 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:47.747 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215238 2024-09-18T14:49:47.747 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.215238/ceph-mon.a.asok 2024-09-18T14:49:47.747 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:49:47.747 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.215238/ceph-mon.a.asok config get mon_host 2024-09-18T14:49:47.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:331: TEST_recovery_pool_priority: run_mgr td/osd-recovery-prio x 2024-09-18T14:49:47.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-prio 2024-09-18T14:49:47.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T14:49:47.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T14:49:47.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T14:49:47.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-prio/x 2024-09-18T14:49:47.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T14:49:48.669 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T14:49:48.669 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:49:48.669 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:49:48.669 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:49:48.669 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:48.670 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215238 2024-09-18T14:49:48.670 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.215238/$cluster-$name.asok' 2024-09-18T14:49:48.670 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:49:48.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-prio/x '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.215238/$cluster-$name.asok' --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:49:48.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:332: TEST_recovery_pool_priority: export CEPH_ARGS 2024-09-18T14:49:48.697 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:334: TEST_recovery_pool_priority: expr 2 - 1 2024-09-18T14:49:48.698 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:334: TEST_recovery_pool_priority: seq 0 1 2024-09-18T14:49:48.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:334: TEST_recovery_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:49:48.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: run_osd td/osd-recovery-prio 0 2024-09-18T14:49:48.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-09-18T14:49:48.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:49:48.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T14:49:48.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:49:48.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/0 2024-09-18T14:49:48.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5c1cb17f-d525-402f-8061-e33fb6e2528e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-09-18T14:49:48.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:49:48.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:49:48.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:49:48.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/0' 2024-09-18T14:49:48.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/0/journal' 2024-09-18T14:49:48.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:49:48.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:49:48.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-09-18T14:49:48.703 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:49:48.703 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:49:48.703 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:49:48.703 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:49:48.703 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:49:48.703 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215238 2024-09-18T14:49:48.703 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.215238/$cluster-$name.asok' 2024-09-18T14:49:48.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.215238/$cluster-$name.asok' 2024-09-18T14:49:48.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:49:48.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:49:48.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:49:48.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-09-18T14:49:48.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-09-18T14:49:48.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:49:48.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:49:48.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:49:48.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:49:48.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:49:48.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:49:48.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/0 2024-09-18T14:49:48.704 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:49:48.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f8e4ca89-0135-44de-84b4-2444c268218c 2024-09-18T14:49:48.705 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 f8e4ca89-0135-44de-84b4-2444c268218c 2024-09-18T14:49:48.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f8e4ca89-0135-44de-84b4-2444c268218c' 2024-09-18T14:49:48.706 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:49:48.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCM6OpmLpHNKhAA8KL9TYSzn5HGMmguQtoK4g== 2024-09-18T14:49:48.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCM6OpmLpHNKhAA8KL9TYSzn5HGMmguQtoK4g=="}' 2024-09-18T14:49:48.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f8e4ca89-0135-44de-84b4-2444c268218c -i td/osd-recovery-prio/0/new.json 2024-09-18T14:49:48.994 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:49:49.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/0/new.json 2024-09-18T14:49:49.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=5c1cb17f-d525-402f-8061-e33fb6e2528e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.215238/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCM6OpmLpHNKhAA8KL9TYSzn5HGMmguQtoK4g== --osd-uuid f8e4ca89-0135-44de-84b4-2444c268218c 2024-09-18T14:49:49.032 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:49:49.031+0000 7f13f3819440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:49:49.039 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:49:49.038+0000 7f13f3819440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:49:49.040 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:49:49.039+0000 7f13f3819440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:49:49.041 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:49:49.040+0000 7f13f3819440 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-09-18T14:49:49.041 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:49:49.040+0000 7f13f3819440 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-09-18T14:49:49.041 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:49:49.040+0000 7f13f3819440 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-09-18T14:49:49.088 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:49:49.087+0000 7f13f3819440 -1 bluestore(td/osd-recovery-prio/0) _read_fsid unparsable uuid 2024-09-18T14:49:54.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/0/keyring 2024-09-18T14:49:54.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:49:54.646 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T14:49:54.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T14:49:54.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:49:55.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T14:49:55.154 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T14:49:55.155 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=5c1cb17f-d525-402f-8061-e33fb6e2528e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.215238/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:49:55.155 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:49:55.155 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:49:55.155 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:49:55.179 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:49:55.178+0000 7f14c4689440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:49:55.204 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:49:55.203+0000 7f14c4689440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:49:55.205 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:49:55.204+0000 7f14c4689440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:49:55.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T14:49:55.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:49:55.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:49:55.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:49:55.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:49:55.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:49:55.610 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:49:55.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:49:55.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:49:55.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:49:55.742 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:49:55.741+0000 7f14c4689440 -1 Falling back to public interface 2024-09-18T14:49:55.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:49:56.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:49:56.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:49:56.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:49:56.974 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:49:56.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:49:56.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:49:57.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:49:58.233 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:49:58.232+0000 7f14c4689440 -1 osd.0 0 log_to_monitors true 2024-09-18T14:49:58.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:49:58.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:49:58.352 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:49:58.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:49:58.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:49:58.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:49:58.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:49:59.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:49:59.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:49:59.813 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:49:59.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:49:59.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:49:59.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:50:00.222 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/216331,v1:127.0.0.1:6803/216331] [v2:127.0.0.1:6804/216331,v1:127.0.0.1:6805/216331] exists,up f8e4ca89-0135-44de-84b4-2444c268218c 2024-09-18T14:50:00.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:50:00.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:50:00.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:50:00.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:334: TEST_recovery_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:50:00.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: run_osd td/osd-recovery-prio 1 2024-09-18T14:50:00.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-09-18T14:50:00.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:50:00.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T14:50:00.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:50:00.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/1 2024-09-18T14:50:00.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5c1cb17f-d525-402f-8061-e33fb6e2528e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-09-18T14:50:00.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:50:00.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:50:00.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:50:00.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/1' 2024-09-18T14:50:00.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/1/journal' 2024-09-18T14:50:00.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:50:00.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:50:00.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-09-18T14:50:00.227 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:50:00.227 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:50:00.227 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:50:00.228 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:50:00.228 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:50:00.228 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215238 2024-09-18T14:50:00.228 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.215238/$cluster-$name.asok' 2024-09-18T14:50:00.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.215238/$cluster-$name.asok' 2024-09-18T14:50:00.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:50:00.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:50:00.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:50:00.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-09-18T14:50:00.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-09-18T14:50:00.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:50:00.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:50:00.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:50:00.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:50:00.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:50:00.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:50:00.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/1 2024-09-18T14:50:00.229 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:50:00.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e769b2be-3e97-4a2f-9ade-8e3bfb4cb3a1 2024-09-18T14:50:00.229 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 e769b2be-3e97-4a2f-9ade-8e3bfb4cb3a1 2024-09-18T14:50:00.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 e769b2be-3e97-4a2f-9ade-8e3bfb4cb3a1' 2024-09-18T14:50:00.230 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:50:00.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCY6Opm0OFdDhAAmvR36vnTWf9FBV3akd9QYQ== 2024-09-18T14:50:00.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCY6Opm0OFdDhAAmvR36vnTWf9FBV3akd9QYQ=="}' 2024-09-18T14:50:00.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e769b2be-3e97-4a2f-9ade-8e3bfb4cb3a1 -i td/osd-recovery-prio/1/new.json 2024-09-18T14:50:00.730 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:50:00.740 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/1/new.json 2024-09-18T14:50:00.741 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=5c1cb17f-d525-402f-8061-e33fb6e2528e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.215238/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCY6Opm0OFdDhAAmvR36vnTWf9FBV3akd9QYQ== --osd-uuid e769b2be-3e97-4a2f-9ade-8e3bfb4cb3a1 2024-09-18T14:50:00.768 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:50:00.767+0000 7f0979039440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:50:00.770 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:50:00.769+0000 7f0979039440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:50:00.771 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:50:00.770+0000 7f0979039440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:50:00.772 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:50:00.771+0000 7f0979039440 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-09-18T14:50:00.772 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:50:00.771+0000 7f0979039440 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-09-18T14:50:00.772 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:50:00.771+0000 7f0979039440 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-09-18T14:50:00.855 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:50:00.853+0000 7f0979039440 -1 bluestore(td/osd-recovery-prio/1) _read_fsid unparsable uuid 2024-09-18T14:50:06.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/1/keyring 2024-09-18T14:50:06.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:50:06.460 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T14:50:06.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T14:50:06.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:50:06.997 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T14:50:06.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T14:50:06.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=5c1cb17f-d525-402f-8061-e33fb6e2528e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.215238/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:50:06.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:50:06.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:50:06.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:50:07.023 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:50:07.022+0000 7fe76d00d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:50:07.025 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:50:07.024+0000 7fe76d00d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:50:07.026 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:50:07.025+0000 7fe76d00d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:50:07.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T14:50:07.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:50:07.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:50:07.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:50:07.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:50:07.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:50:07.408 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:50:07.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:50:07.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:50:07.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:50:07.549 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:50:07.548+0000 7fe76d00d440 -1 Falling back to public interface 2024-09-18T14:50:07.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:50:08.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:50:08.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:50:08.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:50:08.818 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:50:08.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:50:08.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:50:09.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:50:10.041 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:50:10.039+0000 7fe76d00d440 -1 osd.1 0 log_to_monitors true 2024-09-18T14:50:10.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:50:10.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:50:10.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:50:10.245 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:50:10.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:50:10.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:50:10.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:50:11.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:50:11.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:50:11.711 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:50:11.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:50:11.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:50:11.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:50:12.132 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/217730,v1:127.0.0.1:6811/217730] [v2:127.0.0.1:6812/217730,v1:127.0.0.1:6813/217730] exists,up e769b2be-3e97-4a2f-9ade-8e3bfb4cb3a1 2024-09-18T14:50:12.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:50:12.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:50:12.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:50:12.134 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: seq 1 3 2024-09-18T14:50:12.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-09-18T14:50:12.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:341: TEST_recovery_pool_priority: create_pool test1 1 1 2024-09-18T14:50:12.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2024-09-18T14:50:12.654 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test1' already exists 2024-09-18T14:50:12.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T14:50:13.666 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:342: TEST_recovery_pool_priority: ceph osd pool set test1 size 2 2024-09-18T14:50:14.478 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 1 size to 2 2024-09-18T14:50:14.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-09-18T14:50:14.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:341: TEST_recovery_pool_priority: create_pool test2 1 1 2024-09-18T14:50:14.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test2 1 1 2024-09-18T14:50:15.096 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test2' already exists 2024-09-18T14:50:15.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T14:50:16.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:342: TEST_recovery_pool_priority: ceph osd pool set test2 size 2 2024-09-18T14:50:17.283 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 2 size to 2 2024-09-18T14:50:17.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-09-18T14:50:17.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:341: TEST_recovery_pool_priority: create_pool test3 1 1 2024-09-18T14:50:17.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test3 1 1 2024-09-18T14:50:17.895 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test3' already exists 2024-09-18T14:50:17.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T14:50:18.907 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:342: TEST_recovery_pool_priority: ceph osd pool set test3 size 2 2024-09-18T14:50:19.635 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 3 size to 2 2024-09-18T14:50:19.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:344: TEST_recovery_pool_priority: sleep 5 2024-09-18T14:50:24.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:346: TEST_recovery_pool_priority: wait_for_clean 2024-09-18T14:50:24.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:50:24.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:50:24.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:50:24.649 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:50:24.650 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:50:24.650 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:50:24.650 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:50:24.650 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:50:24.650 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:50:24.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T14:50:24.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:50:24.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:50:24.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:50:24.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:50:24.750 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:50:25.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:50:25.179 INFO:tasks.workunit.client.0.smithi047.stderr:1' 2024-09-18T14:50:25.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:50:25.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:50:25.179 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:50:25.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836488 2024-09-18T14:50:25.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836488 2024-09-18T14:50:25.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836488' 2024-09-18T14:50:25.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:50:25.289 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:50:25.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672965 2024-09-18T14:50:25.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672965 2024-09-18T14:50:25.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836488 1-42949672965' 2024-09-18T14:50:25.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:50:25.400 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836488 2024-09-18T14:50:25.400 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:50:25.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:50:25.402 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836488 2024-09-18T14:50:25.402 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:50:25.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836488 2024-09-18T14:50:25.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2024-09-18T14:50:25.403 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836488 2024-09-18T14:50:25.404 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:50:25.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836485 -lt 21474836488 2024-09-18T14:50:25.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:50:26.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:50:26.774 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:50:27.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836488 -lt 21474836488 2024-09-18T14:50:27.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:50:27.168 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672965 2024-09-18T14:50:27.168 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:50:27.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:50:27.169 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672965 2024-09-18T14:50:27.169 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:50:27.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672965 2024-09-18T14:50:27.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-09-18T14:50:27.171 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672965 2024-09-18T14:50:27.171 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:50:27.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672965 -lt 42949672965 2024-09-18T14:50:27.540 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:50:27.541 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:50:27.541 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:50:28.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 == 0 2024-09-18T14:50:28.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:50:28.037 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:50:28.037 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:50:28.037 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:50:28.037 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:50:28.037 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:50:28.037 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:50:28.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-09-18T14:50:28.401 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:50:28.401 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:50:28.401 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:50:28.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 3 2024-09-18T14:50:28.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:50:28.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:50:28.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:348: TEST_recovery_pool_priority: ceph pg dump pgs 2024-09-18T14:50:29.272 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:50:29.272 INFO:tasks.workunit.client.0.smithi047.stdout:3.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:50:20.513206+0000 0'0 28:19 [1,0] 1 [1,0] 1 0'0 2024-09-18T14:50:17.678530+0000 0'0 2024-09-18T14:50:17.678530+0000 0 0 periodic scrub scheduled @ 2024-09-19T18:43:15.280956+0000 0 0 2024-09-18T14:50:29.272 INFO:tasks.workunit.client.0.smithi047.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:50:17.145213+0000 0'0 27:23 [0,1] 0 [0,1] 0 0'0 2024-09-18T14:50:14.761688+0000 0'0 2024-09-18T14:50:14.761688+0000 0 0 periodic scrub scheduled @ 2024-09-19T15:35:28.228424+0000 0 0 2024-09-18T14:50:29.272 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:50:15.074136+0000 0'0 27:30 [1,0] 1 [1,0] 1 0'0 2024-09-18T14:50:12.444513+0000 0'0 2024-09-18T14:50:12.444513+0000 0 0 periodic scrub scheduled @ 2024-09-19T22:12:25.125495+0000 0 0 2024-09-18T14:50:29.272 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:50:29.272 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:50:29.272 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:50:29.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:352: TEST_recovery_pool_priority: local PG1 2024-09-18T14:50:29.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:353: TEST_recovery_pool_priority: local POOLNUM1 2024-09-18T14:50:29.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:354: TEST_recovery_pool_priority: local pool1 2024-09-18T14:50:29.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:355: TEST_recovery_pool_priority: local chk_osd1_1 2024-09-18T14:50:29.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:356: TEST_recovery_pool_priority: local chk_osd1_2 2024-09-18T14:50:29.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:358: TEST_recovery_pool_priority: local PG2 2024-09-18T14:50:29.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:359: TEST_recovery_pool_priority: local POOLNUM2 2024-09-18T14:50:29.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:360: TEST_recovery_pool_priority: local pool2 2024-09-18T14:50:29.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:361: TEST_recovery_pool_priority: local chk_osd2_1 2024-09-18T14:50:29.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:362: TEST_recovery_pool_priority: local chk_osd2_2 2024-09-18T14:50:29.283 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: seq 1 3 2024-09-18T14:50:29.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-09-18T14:50:29.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: ceph pg map 1.0 --format=json 2024-09-18T14:50:29.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: jq '.acting[]' 2024-09-18T14:50:29.686 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:367: TEST_recovery_pool_priority: head -1 td/osd-recovery-prio/acting 2024-09-18T14:50:29.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:367: TEST_recovery_pool_priority: local test_osd1=1 2024-09-18T14:50:29.687 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: tail -1 td/osd-recovery-prio/acting 2024-09-18T14:50:29.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: local test_osd2=0 2024-09-18T14:50:29.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:369: TEST_recovery_pool_priority: '[' -z '' ']' 2024-09-18T14:50:29.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:371: TEST_recovery_pool_priority: PG1=1.0 2024-09-18T14:50:29.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:372: TEST_recovery_pool_priority: POOLNUM1=1 2024-09-18T14:50:29.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:373: TEST_recovery_pool_priority: pool1=test1 2024-09-18T14:50:29.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:374: TEST_recovery_pool_priority: chk_osd1_1=1 2024-09-18T14:50:29.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:375: TEST_recovery_pool_priority: chk_osd1_2=0 2024-09-18T14:50:29.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-09-18T14:50:29.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: ceph pg map 2.0 --format=json 2024-09-18T14:50:29.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: jq '.acting[]' 2024-09-18T14:50:30.051 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:367: TEST_recovery_pool_priority: head -1 td/osd-recovery-prio/acting 2024-09-18T14:50:30.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:367: TEST_recovery_pool_priority: local test_osd1=0 2024-09-18T14:50:30.052 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: tail -1 td/osd-recovery-prio/acting 2024-09-18T14:50:30.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: local test_osd2=1 2024-09-18T14:50:30.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:369: TEST_recovery_pool_priority: '[' -z 1.0 ']' 2024-09-18T14:50:30.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:376: TEST_recovery_pool_priority: '[' 1 '!=' 0 ']' 2024-09-18T14:50:30.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:378: TEST_recovery_pool_priority: PG2=2.0 2024-09-18T14:50:30.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:379: TEST_recovery_pool_priority: POOLNUM2=2 2024-09-18T14:50:30.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:380: TEST_recovery_pool_priority: pool2=test2 2024-09-18T14:50:30.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:381: TEST_recovery_pool_priority: chk_osd2_1=0 2024-09-18T14:50:30.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:382: TEST_recovery_pool_priority: chk_osd2_2=1 2024-09-18T14:50:30.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:383: TEST_recovery_pool_priority: break 2024-09-18T14:50:30.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:386: TEST_recovery_pool_priority: rm -f td/osd-recovery-prio/acting 2024-09-18T14:50:30.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:388: TEST_recovery_pool_priority: '[' test2 = '' ']' 2024-09-18T14:50:30.068 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:394: TEST_recovery_pool_priority: seq 1 3 2024-09-18T14:50:30.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:394: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-09-18T14:50:30.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:396: TEST_recovery_pool_priority: '[' 1 '!=' 1 -a 1 '!=' 2 ']' 2024-09-18T14:50:30.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:394: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-09-18T14:50:30.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:396: TEST_recovery_pool_priority: '[' 2 '!=' 1 -a 2 '!=' 2 ']' 2024-09-18T14:50:30.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:394: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-09-18T14:50:30.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:396: TEST_recovery_pool_priority: '[' 3 '!=' 1 -a 3 '!=' 2 ']' 2024-09-18T14:50:30.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:398: TEST_recovery_pool_priority: delete_pool test3 2024-09-18T14:50:30.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test3 2024-09-18T14:50:30.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2024-09-18T14:50:30.701 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test3' does not exist 2024-09-18T14:50:30.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:402: TEST_recovery_pool_priority: pool1_extra_prio=1 2024-09-18T14:50:30.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:403: TEST_recovery_pool_priority: pool2_extra_prio=2 2024-09-18T14:50:30.713 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:404: TEST_recovery_pool_priority: expr 190 + 1 2024-09-18T14:50:30.714 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:404: TEST_recovery_pool_priority: pool1_prio=191 2024-09-18T14:50:30.714 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:405: TEST_recovery_pool_priority: expr 190 + 2 2024-09-18T14:50:30.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:405: TEST_recovery_pool_priority: pool2_prio=192 2024-09-18T14:50:30.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:407: TEST_recovery_pool_priority: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2024-09-18T14:50:31.507 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 1 size to 1 2024-09-18T14:50:31.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:408: TEST_recovery_pool_priority: ceph osd pool set test1 recovery_priority 1 2024-09-18T14:50:32.353 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 1 recovery_priority to 1 2024-09-18T14:50:32.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:409: TEST_recovery_pool_priority: ceph osd pool set test2 size 1 --yes-i-really-mean-it 2024-09-18T14:50:33.158 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 2 size to 1 2024-09-18T14:50:33.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:410: TEST_recovery_pool_priority: ceph osd pool set test2 recovery_priority 2 2024-09-18T14:50:33.987 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 2 recovery_priority to 2 2024-09-18T14:50:33.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:411: TEST_recovery_pool_priority: wait_for_clean 2024-09-18T14:50:33.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:50:33.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:50:33.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:50:33.999 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:50:33.999 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:50:33.999 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:50:34.000 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:50:34.000 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:50:34.000 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:50:34.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T14:50:34.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:50:34.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:50:34.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:50:34.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:50:34.089 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:50:34.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:50:34.464 INFO:tasks.workunit.client.0.smithi047.stderr:1' 2024-09-18T14:50:34.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:50:34.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:50:34.464 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:50:34.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836491 2024-09-18T14:50:34.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836491 2024-09-18T14:50:34.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836491' 2024-09-18T14:50:34.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:50:34.575 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:50:34.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672969 2024-09-18T14:50:34.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672969 2024-09-18T14:50:34.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836491 1-42949672969' 2024-09-18T14:50:34.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:50:34.686 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836491 2024-09-18T14:50:34.686 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:50:34.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:50:34.687 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836491 2024-09-18T14:50:34.688 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:50:34.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836491 2024-09-18T14:50:34.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2024-09-18T14:50:34.689 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836491 2024-09-18T14:50:34.689 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:50:35.119 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836489 -lt 21474836491 2024-09-18T14:50:35.119 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:50:36.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:50:36.120 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:50:36.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836492 -lt 21474836491 2024-09-18T14:50:36.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:50:36.506 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672969 2024-09-18T14:50:36.506 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:50:36.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:50:36.508 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672969 2024-09-18T14:50:36.508 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:50:36.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672969 2024-09-18T14:50:36.509 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672969 2024-09-18T14:50:36.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-09-18T14:50:36.509 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:50:36.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672969 -lt 42949672969 2024-09-18T14:50:36.888 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:50:36.888 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:50:36.888 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:50:37.420 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-09-18T14:50:37.420 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:50:37.421 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:50:37.421 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:50:37.421 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:50:37.421 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:50:37.421 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:50:37.421 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:50:37.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T14:50:37.776 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:50:37.776 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:50:37.776 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:50:38.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-09-18T14:50:38.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:50:38.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:50:38.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:413: TEST_recovery_pool_priority: dd if=/dev/urandom of=td/osd-recovery-prio/data bs=1M count=10 2024-09-18T14:50:38.305 INFO:tasks.workunit.client.0.smithi047.stderr:10+0 records in 2024-09-18T14:50:38.305 INFO:tasks.workunit.client.0.smithi047.stderr:10+0 records out 2024-09-18T14:50:38.305 INFO:tasks.workunit.client.0.smithi047.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0279734 s, 375 MB/s 2024-09-18T14:50:38.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: p=1 2024-09-18T14:50:38.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:415: TEST_recovery_pool_priority: for pname in $pool1 $pool2 2024-09-18T14:50:38.306 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: seq 1 200 2024-09-18T14:50:38.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:38.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj1-p1 td/osd-recovery-prio/data 2024-09-18T14:50:38.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:38.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj2-p1 td/osd-recovery-prio/data 2024-09-18T14:50:39.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:39.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj3-p1 td/osd-recovery-prio/data 2024-09-18T14:50:39.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:39.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj4-p1 td/osd-recovery-prio/data 2024-09-18T14:50:39.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:39.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj5-p1 td/osd-recovery-prio/data 2024-09-18T14:50:40.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:40.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj6-p1 td/osd-recovery-prio/data 2024-09-18T14:50:40.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:40.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj7-p1 td/osd-recovery-prio/data 2024-09-18T14:50:41.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:41.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj8-p1 td/osd-recovery-prio/data 2024-09-18T14:50:41.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:41.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj9-p1 td/osd-recovery-prio/data 2024-09-18T14:50:41.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:41.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj10-p1 td/osd-recovery-prio/data 2024-09-18T14:50:42.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:42.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj11-p1 td/osd-recovery-prio/data 2024-09-18T14:50:42.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:42.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj12-p1 td/osd-recovery-prio/data 2024-09-18T14:50:42.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:42.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj13-p1 td/osd-recovery-prio/data 2024-09-18T14:50:43.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:43.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj14-p1 td/osd-recovery-prio/data 2024-09-18T14:50:43.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:43.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj15-p1 td/osd-recovery-prio/data 2024-09-18T14:50:43.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:43.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj16-p1 td/osd-recovery-prio/data 2024-09-18T14:50:44.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:44.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj17-p1 td/osd-recovery-prio/data 2024-09-18T14:50:44.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:44.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj18-p1 td/osd-recovery-prio/data 2024-09-18T14:50:44.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:44.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj19-p1 td/osd-recovery-prio/data 2024-09-18T14:50:45.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:45.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj20-p1 td/osd-recovery-prio/data 2024-09-18T14:50:45.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:45.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj21-p1 td/osd-recovery-prio/data 2024-09-18T14:50:45.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:45.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj22-p1 td/osd-recovery-prio/data 2024-09-18T14:50:46.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:46.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj23-p1 td/osd-recovery-prio/data 2024-09-18T14:50:46.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:46.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj24-p1 td/osd-recovery-prio/data 2024-09-18T14:50:47.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:47.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj25-p1 td/osd-recovery-prio/data 2024-09-18T14:50:47.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:47.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj26-p1 td/osd-recovery-prio/data 2024-09-18T14:50:47.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:47.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj27-p1 td/osd-recovery-prio/data 2024-09-18T14:50:48.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:48.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj28-p1 td/osd-recovery-prio/data 2024-09-18T14:50:48.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:48.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj29-p1 td/osd-recovery-prio/data 2024-09-18T14:50:48.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:48.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj30-p1 td/osd-recovery-prio/data 2024-09-18T14:50:49.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:49.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj31-p1 td/osd-recovery-prio/data 2024-09-18T14:50:49.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:49.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj32-p1 td/osd-recovery-prio/data 2024-09-18T14:50:49.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:49.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj33-p1 td/osd-recovery-prio/data 2024-09-18T14:50:50.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:50.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj34-p1 td/osd-recovery-prio/data 2024-09-18T14:50:50.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:50.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj35-p1 td/osd-recovery-prio/data 2024-09-18T14:50:51.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:51.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj36-p1 td/osd-recovery-prio/data 2024-09-18T14:50:51.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:51.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj37-p1 td/osd-recovery-prio/data 2024-09-18T14:50:51.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:51.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj38-p1 td/osd-recovery-prio/data 2024-09-18T14:50:52.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:52.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj39-p1 td/osd-recovery-prio/data 2024-09-18T14:50:52.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:52.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj40-p1 td/osd-recovery-prio/data 2024-09-18T14:50:52.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:52.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj41-p1 td/osd-recovery-prio/data 2024-09-18T14:50:53.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:53.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj42-p1 td/osd-recovery-prio/data 2024-09-18T14:50:53.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:53.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj43-p1 td/osd-recovery-prio/data 2024-09-18T14:50:53.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:53.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj44-p1 td/osd-recovery-prio/data 2024-09-18T14:50:54.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:54.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj45-p1 td/osd-recovery-prio/data 2024-09-18T14:50:54.648 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:54.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj46-p1 td/osd-recovery-prio/data 2024-09-18T14:50:55.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:55.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj47-p1 td/osd-recovery-prio/data 2024-09-18T14:50:55.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:55.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj48-p1 td/osd-recovery-prio/data 2024-09-18T14:50:55.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:55.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj49-p1 td/osd-recovery-prio/data 2024-09-18T14:50:56.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:56.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj50-p1 td/osd-recovery-prio/data 2024-09-18T14:50:56.389 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:56.389 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj51-p1 td/osd-recovery-prio/data 2024-09-18T14:50:56.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:56.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj52-p1 td/osd-recovery-prio/data 2024-09-18T14:50:57.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:57.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj53-p1 td/osd-recovery-prio/data 2024-09-18T14:50:57.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:57.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj54-p1 td/osd-recovery-prio/data 2024-09-18T14:50:57.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:57.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj55-p1 td/osd-recovery-prio/data 2024-09-18T14:50:58.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:58.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj56-p1 td/osd-recovery-prio/data 2024-09-18T14:50:58.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:58.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj57-p1 td/osd-recovery-prio/data 2024-09-18T14:50:59.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:59.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj58-p1 td/osd-recovery-prio/data 2024-09-18T14:50:59.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:59.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj59-p1 td/osd-recovery-prio/data 2024-09-18T14:50:59.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:50:59.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj60-p1 td/osd-recovery-prio/data 2024-09-18T14:51:00.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:00.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj61-p1 td/osd-recovery-prio/data 2024-09-18T14:51:00.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:00.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj62-p1 td/osd-recovery-prio/data 2024-09-18T14:51:00.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:00.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj63-p1 td/osd-recovery-prio/data 2024-09-18T14:51:01.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:01.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj64-p1 td/osd-recovery-prio/data 2024-09-18T14:51:01.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:01.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj65-p1 td/osd-recovery-prio/data 2024-09-18T14:51:02.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:02.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj66-p1 td/osd-recovery-prio/data 2024-09-18T14:51:02.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:02.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj67-p1 td/osd-recovery-prio/data 2024-09-18T14:51:02.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:02.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj68-p1 td/osd-recovery-prio/data 2024-09-18T14:51:03.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:03.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj69-p1 td/osd-recovery-prio/data 2024-09-18T14:51:03.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:03.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj70-p1 td/osd-recovery-prio/data 2024-09-18T14:51:03.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:03.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj71-p1 td/osd-recovery-prio/data 2024-09-18T14:51:04.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:04.194 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj72-p1 td/osd-recovery-prio/data 2024-09-18T14:51:04.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:04.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj73-p1 td/osd-recovery-prio/data 2024-09-18T14:51:05.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:05.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj74-p1 td/osd-recovery-prio/data 2024-09-18T14:51:05.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:05.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj75-p1 td/osd-recovery-prio/data 2024-09-18T14:51:05.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:05.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj76-p1 td/osd-recovery-prio/data 2024-09-18T14:51:06.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:06.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj77-p1 td/osd-recovery-prio/data 2024-09-18T14:51:06.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:06.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj78-p1 td/osd-recovery-prio/data 2024-09-18T14:51:06.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:06.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj79-p1 td/osd-recovery-prio/data 2024-09-18T14:51:07.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:07.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj80-p1 td/osd-recovery-prio/data 2024-09-18T14:51:07.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:07.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj81-p1 td/osd-recovery-prio/data 2024-09-18T14:51:08.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:08.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj82-p1 td/osd-recovery-prio/data 2024-09-18T14:51:08.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:08.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj83-p1 td/osd-recovery-prio/data 2024-09-18T14:51:08.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:08.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj84-p1 td/osd-recovery-prio/data 2024-09-18T14:51:09.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:09.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj85-p1 td/osd-recovery-prio/data 2024-09-18T14:51:09.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:09.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj86-p1 td/osd-recovery-prio/data 2024-09-18T14:51:09.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:09.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj87-p1 td/osd-recovery-prio/data 2024-09-18T14:51:10.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:10.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj88-p1 td/osd-recovery-prio/data 2024-09-18T14:51:10.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:10.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj89-p1 td/osd-recovery-prio/data 2024-09-18T14:51:11.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:11.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj90-p1 td/osd-recovery-prio/data 2024-09-18T14:51:11.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:11.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj91-p1 td/osd-recovery-prio/data 2024-09-18T14:51:11.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:11.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj92-p1 td/osd-recovery-prio/data 2024-09-18T14:51:12.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:12.157 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj93-p1 td/osd-recovery-prio/data 2024-09-18T14:51:12.549 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:12.549 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj94-p1 td/osd-recovery-prio/data 2024-09-18T14:51:12.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:12.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj95-p1 td/osd-recovery-prio/data 2024-09-18T14:51:13.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:13.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj96-p1 td/osd-recovery-prio/data 2024-09-18T14:51:13.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:13.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj97-p1 td/osd-recovery-prio/data 2024-09-18T14:51:14.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:14.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj98-p1 td/osd-recovery-prio/data 2024-09-18T14:51:14.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:14.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj99-p1 td/osd-recovery-prio/data 2024-09-18T14:51:14.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:14.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj100-p1 td/osd-recovery-prio/data 2024-09-18T14:51:15.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:15.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj101-p1 td/osd-recovery-prio/data 2024-09-18T14:51:15.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:15.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj102-p1 td/osd-recovery-prio/data 2024-09-18T14:51:15.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:15.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj103-p1 td/osd-recovery-prio/data 2024-09-18T14:51:16.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:16.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj104-p1 td/osd-recovery-prio/data 2024-09-18T14:51:16.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:16.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj105-p1 td/osd-recovery-prio/data 2024-09-18T14:51:17.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:17.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj106-p1 td/osd-recovery-prio/data 2024-09-18T14:51:17.343 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:17.343 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj107-p1 td/osd-recovery-prio/data 2024-09-18T14:51:17.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:17.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj108-p1 td/osd-recovery-prio/data 2024-09-18T14:51:17.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:17.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj109-p1 td/osd-recovery-prio/data 2024-09-18T14:51:18.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:18.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj110-p1 td/osd-recovery-prio/data 2024-09-18T14:51:18.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:18.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj111-p1 td/osd-recovery-prio/data 2024-09-18T14:51:19.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:19.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj112-p1 td/osd-recovery-prio/data 2024-09-18T14:51:19.476 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:19.476 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj113-p1 td/osd-recovery-prio/data 2024-09-18T14:51:19.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:19.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj114-p1 td/osd-recovery-prio/data 2024-09-18T14:51:20.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:20.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj115-p1 td/osd-recovery-prio/data 2024-09-18T14:51:20.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:20.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj116-p1 td/osd-recovery-prio/data 2024-09-18T14:51:20.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:20.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj117-p1 td/osd-recovery-prio/data 2024-09-18T14:51:21.242 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:21.242 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj118-p1 td/osd-recovery-prio/data 2024-09-18T14:51:21.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:21.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj119-p1 td/osd-recovery-prio/data 2024-09-18T14:51:21.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:21.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj120-p1 td/osd-recovery-prio/data 2024-09-18T14:51:22.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:22.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj121-p1 td/osd-recovery-prio/data 2024-09-18T14:51:22.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:22.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj122-p1 td/osd-recovery-prio/data 2024-09-18T14:51:23.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:23.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj123-p1 td/osd-recovery-prio/data 2024-09-18T14:51:23.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:23.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj124-p1 td/osd-recovery-prio/data 2024-09-18T14:51:23.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:23.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj125-p1 td/osd-recovery-prio/data 2024-09-18T14:51:24.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:24.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj126-p1 td/osd-recovery-prio/data 2024-09-18T14:51:24.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:24.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj127-p1 td/osd-recovery-prio/data 2024-09-18T14:51:25.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:25.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj128-p1 td/osd-recovery-prio/data 2024-09-18T14:51:25.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:25.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj129-p1 td/osd-recovery-prio/data 2024-09-18T14:51:25.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:25.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj130-p1 td/osd-recovery-prio/data 2024-09-18T14:51:26.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:26.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj131-p1 td/osd-recovery-prio/data 2024-09-18T14:51:26.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:26.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj132-p1 td/osd-recovery-prio/data 2024-09-18T14:51:26.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:26.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj133-p1 td/osd-recovery-prio/data 2024-09-18T14:51:27.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:27.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj134-p1 td/osd-recovery-prio/data 2024-09-18T14:51:27.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:27.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj135-p1 td/osd-recovery-prio/data 2024-09-18T14:51:27.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:27.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj136-p1 td/osd-recovery-prio/data 2024-09-18T14:51:28.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:28.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj137-p1 td/osd-recovery-prio/data 2024-09-18T14:51:28.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:28.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj138-p1 td/osd-recovery-prio/data 2024-09-18T14:51:28.946 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:28.946 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj139-p1 td/osd-recovery-prio/data 2024-09-18T14:51:29.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:29.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj140-p1 td/osd-recovery-prio/data 2024-09-18T14:51:29.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:29.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj141-p1 td/osd-recovery-prio/data 2024-09-18T14:51:29.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:29.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj142-p1 td/osd-recovery-prio/data 2024-09-18T14:51:30.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:30.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj143-p1 td/osd-recovery-prio/data 2024-09-18T14:51:30.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:30.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj144-p1 td/osd-recovery-prio/data 2024-09-18T14:51:31.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:31.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj145-p1 td/osd-recovery-prio/data 2024-09-18T14:51:31.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:31.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj146-p1 td/osd-recovery-prio/data 2024-09-18T14:51:31.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:31.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj147-p1 td/osd-recovery-prio/data 2024-09-18T14:51:32.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:32.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj148-p1 td/osd-recovery-prio/data 2024-09-18T14:51:32.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:32.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj149-p1 td/osd-recovery-prio/data 2024-09-18T14:51:32.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:32.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj150-p1 td/osd-recovery-prio/data 2024-09-18T14:51:33.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:33.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj151-p1 td/osd-recovery-prio/data 2024-09-18T14:51:33.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:33.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj152-p1 td/osd-recovery-prio/data 2024-09-18T14:51:33.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:33.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj153-p1 td/osd-recovery-prio/data 2024-09-18T14:51:34.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:34.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj154-p1 td/osd-recovery-prio/data 2024-09-18T14:51:34.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:34.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj155-p1 td/osd-recovery-prio/data 2024-09-18T14:51:35.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:35.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj156-p1 td/osd-recovery-prio/data 2024-09-18T14:51:35.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:35.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj157-p1 td/osd-recovery-prio/data 2024-09-18T14:51:35.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:35.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj158-p1 td/osd-recovery-prio/data 2024-09-18T14:51:36.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:36.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj159-p1 td/osd-recovery-prio/data 2024-09-18T14:51:36.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:36.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj160-p1 td/osd-recovery-prio/data 2024-09-18T14:51:36.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:36.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj161-p1 td/osd-recovery-prio/data 2024-09-18T14:51:37.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:37.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj162-p1 td/osd-recovery-prio/data 2024-09-18T14:51:37.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:37.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj163-p1 td/osd-recovery-prio/data 2024-09-18T14:51:37.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:37.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj164-p1 td/osd-recovery-prio/data 2024-09-18T14:51:38.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:38.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj165-p1 td/osd-recovery-prio/data 2024-09-18T14:51:38.599 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:38.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj166-p1 td/osd-recovery-prio/data 2024-09-18T14:51:39.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:39.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj167-p1 td/osd-recovery-prio/data 2024-09-18T14:51:39.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:39.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj168-p1 td/osd-recovery-prio/data 2024-09-18T14:51:39.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:39.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj169-p1 td/osd-recovery-prio/data 2024-09-18T14:51:39.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:39.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj170-p1 td/osd-recovery-prio/data 2024-09-18T14:51:40.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:40.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj171-p1 td/osd-recovery-prio/data 2024-09-18T14:51:40.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:40.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj172-p1 td/osd-recovery-prio/data 2024-09-18T14:51:41.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:41.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj173-p1 td/osd-recovery-prio/data 2024-09-18T14:51:41.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:41.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj174-p1 td/osd-recovery-prio/data 2024-09-18T14:51:41.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:41.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj175-p1 td/osd-recovery-prio/data 2024-09-18T14:51:42.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:42.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj176-p1 td/osd-recovery-prio/data 2024-09-18T14:51:42.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:42.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj177-p1 td/osd-recovery-prio/data 2024-09-18T14:51:42.991 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:42.991 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj178-p1 td/osd-recovery-prio/data 2024-09-18T14:51:43.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:43.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj179-p1 td/osd-recovery-prio/data 2024-09-18T14:51:43.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:43.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj180-p1 td/osd-recovery-prio/data 2024-09-18T14:51:44.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:44.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj181-p1 td/osd-recovery-prio/data 2024-09-18T14:51:44.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:44.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj182-p1 td/osd-recovery-prio/data 2024-09-18T14:51:44.714 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:44.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj183-p1 td/osd-recovery-prio/data 2024-09-18T14:51:45.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:45.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj184-p1 td/osd-recovery-prio/data 2024-09-18T14:51:45.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:45.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj185-p1 td/osd-recovery-prio/data 2024-09-18T14:51:45.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:45.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj186-p1 td/osd-recovery-prio/data 2024-09-18T14:51:46.095 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:46.095 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj187-p1 td/osd-recovery-prio/data 2024-09-18T14:51:46.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:46.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj188-p1 td/osd-recovery-prio/data 2024-09-18T14:51:46.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:46.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj189-p1 td/osd-recovery-prio/data 2024-09-18T14:51:47.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:47.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj190-p1 td/osd-recovery-prio/data 2024-09-18T14:51:47.551 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:47.551 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj191-p1 td/osd-recovery-prio/data 2024-09-18T14:51:47.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:47.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj192-p1 td/osd-recovery-prio/data 2024-09-18T14:51:48.212 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:48.212 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj193-p1 td/osd-recovery-prio/data 2024-09-18T14:51:48.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:48.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj194-p1 td/osd-recovery-prio/data 2024-09-18T14:51:48.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:48.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj195-p1 td/osd-recovery-prio/data 2024-09-18T14:51:49.343 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:49.343 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj196-p1 td/osd-recovery-prio/data 2024-09-18T14:51:49.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:49.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj197-p1 td/osd-recovery-prio/data 2024-09-18T14:51:50.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:50.021 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj198-p1 td/osd-recovery-prio/data 2024-09-18T14:51:50.355 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:50.355 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj199-p1 td/osd-recovery-prio/data 2024-09-18T14:51:50.740 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:50.740 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj200-p1 td/osd-recovery-prio/data 2024-09-18T14:51:51.159 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: expr 1 + 1 2024-09-18T14:51:51.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: p=2 2024-09-18T14:51:51.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:415: TEST_recovery_pool_priority: for pname in $pool1 $pool2 2024-09-18T14:51:51.161 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: seq 1 200 2024-09-18T14:51:51.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:51.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj1-p2 td/osd-recovery-prio/data 2024-09-18T14:51:51.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:51.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj2-p2 td/osd-recovery-prio/data 2024-09-18T14:51:51.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:51.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj3-p2 td/osd-recovery-prio/data 2024-09-18T14:51:52.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:52.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj4-p2 td/osd-recovery-prio/data 2024-09-18T14:51:52.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:52.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj5-p2 td/osd-recovery-prio/data 2024-09-18T14:51:53.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:53.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj6-p2 td/osd-recovery-prio/data 2024-09-18T14:51:53.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:53.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj7-p2 td/osd-recovery-prio/data 2024-09-18T14:51:53.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:53.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj8-p2 td/osd-recovery-prio/data 2024-09-18T14:51:54.119 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:54.119 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj9-p2 td/osd-recovery-prio/data 2024-09-18T14:51:54.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:54.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj10-p2 td/osd-recovery-prio/data 2024-09-18T14:51:55.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:55.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj11-p2 td/osd-recovery-prio/data 2024-09-18T14:51:55.483 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:55.483 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj12-p2 td/osd-recovery-prio/data 2024-09-18T14:51:55.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:55.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj13-p2 td/osd-recovery-prio/data 2024-09-18T14:51:56.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:56.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj14-p2 td/osd-recovery-prio/data 2024-09-18T14:51:56.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:56.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj15-p2 td/osd-recovery-prio/data 2024-09-18T14:51:57.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:57.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj16-p2 td/osd-recovery-prio/data 2024-09-18T14:51:57.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:57.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj17-p2 td/osd-recovery-prio/data 2024-09-18T14:51:57.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:57.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj18-p2 td/osd-recovery-prio/data 2024-09-18T14:51:58.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:58.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj19-p2 td/osd-recovery-prio/data 2024-09-18T14:51:58.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:58.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj20-p2 td/osd-recovery-prio/data 2024-09-18T14:51:58.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:58.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj21-p2 td/osd-recovery-prio/data 2024-09-18T14:51:59.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:59.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj22-p2 td/osd-recovery-prio/data 2024-09-18T14:51:59.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:59.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj23-p2 td/osd-recovery-prio/data 2024-09-18T14:51:59.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:51:59.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj24-p2 td/osd-recovery-prio/data 2024-09-18T14:52:00.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:00.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj25-p2 td/osd-recovery-prio/data 2024-09-18T14:52:00.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:00.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj26-p2 td/osd-recovery-prio/data 2024-09-18T14:52:01.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:01.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj27-p2 td/osd-recovery-prio/data 2024-09-18T14:52:01.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:01.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj28-p2 td/osd-recovery-prio/data 2024-09-18T14:52:01.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:01.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj29-p2 td/osd-recovery-prio/data 2024-09-18T14:52:02.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:02.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj30-p2 td/osd-recovery-prio/data 2024-09-18T14:52:02.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:02.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj31-p2 td/osd-recovery-prio/data 2024-09-18T14:52:02.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:02.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj32-p2 td/osd-recovery-prio/data 2024-09-18T14:52:03.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:03.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj33-p2 td/osd-recovery-prio/data 2024-09-18T14:52:03.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:03.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj34-p2 td/osd-recovery-prio/data 2024-09-18T14:52:03.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:03.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj35-p2 td/osd-recovery-prio/data 2024-09-18T14:52:04.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:04.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj36-p2 td/osd-recovery-prio/data 2024-09-18T14:52:04.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:04.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj37-p2 td/osd-recovery-prio/data 2024-09-18T14:52:05.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:05.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj38-p2 td/osd-recovery-prio/data 2024-09-18T14:52:05.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:05.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj39-p2 td/osd-recovery-prio/data 2024-09-18T14:52:05.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:05.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj40-p2 td/osd-recovery-prio/data 2024-09-18T14:52:06.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:06.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj41-p2 td/osd-recovery-prio/data 2024-09-18T14:52:06.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:06.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj42-p2 td/osd-recovery-prio/data 2024-09-18T14:52:06.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:06.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj43-p2 td/osd-recovery-prio/data 2024-09-18T14:52:07.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:07.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj44-p2 td/osd-recovery-prio/data 2024-09-18T14:52:07.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:07.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj45-p2 td/osd-recovery-prio/data 2024-09-18T14:52:07.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:07.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj46-p2 td/osd-recovery-prio/data 2024-09-18T14:52:08.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:08.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj47-p2 td/osd-recovery-prio/data 2024-09-18T14:52:08.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:08.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj48-p2 td/osd-recovery-prio/data 2024-09-18T14:52:09.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:09.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj49-p2 td/osd-recovery-prio/data 2024-09-18T14:52:09.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:09.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj50-p2 td/osd-recovery-prio/data 2024-09-18T14:52:09.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:09.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj51-p2 td/osd-recovery-prio/data 2024-09-18T14:52:10.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:10.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj52-p2 td/osd-recovery-prio/data 2024-09-18T14:52:10.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:10.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj53-p2 td/osd-recovery-prio/data 2024-09-18T14:52:10.740 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:10.741 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj54-p2 td/osd-recovery-prio/data 2024-09-18T14:52:11.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:11.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj55-p2 td/osd-recovery-prio/data 2024-09-18T14:52:11.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:11.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj56-p2 td/osd-recovery-prio/data 2024-09-18T14:52:11.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:11.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj57-p2 td/osd-recovery-prio/data 2024-09-18T14:52:12.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:12.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj58-p2 td/osd-recovery-prio/data 2024-09-18T14:52:12.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:12.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj59-p2 td/osd-recovery-prio/data 2024-09-18T14:52:12.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:12.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj60-p2 td/osd-recovery-prio/data 2024-09-18T14:52:13.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:13.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj61-p2 td/osd-recovery-prio/data 2024-09-18T14:52:13.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:13.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj62-p2 td/osd-recovery-prio/data 2024-09-18T14:52:13.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:13.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj63-p2 td/osd-recovery-prio/data 2024-09-18T14:52:14.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:14.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj64-p2 td/osd-recovery-prio/data 2024-09-18T14:52:14.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:14.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj65-p2 td/osd-recovery-prio/data 2024-09-18T14:52:15.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:15.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj66-p2 td/osd-recovery-prio/data 2024-09-18T14:52:15.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:15.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj67-p2 td/osd-recovery-prio/data 2024-09-18T14:52:15.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:15.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj68-p2 td/osd-recovery-prio/data 2024-09-18T14:52:16.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:16.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj69-p2 td/osd-recovery-prio/data 2024-09-18T14:52:16.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:16.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj70-p2 td/osd-recovery-prio/data 2024-09-18T14:52:16.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:16.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj71-p2 td/osd-recovery-prio/data 2024-09-18T14:52:17.157 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:17.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj72-p2 td/osd-recovery-prio/data 2024-09-18T14:52:17.483 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:17.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj73-p2 td/osd-recovery-prio/data 2024-09-18T14:52:17.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:17.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj74-p2 td/osd-recovery-prio/data 2024-09-18T14:52:18.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:18.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj75-p2 td/osd-recovery-prio/data 2024-09-18T14:52:18.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:18.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj76-p2 td/osd-recovery-prio/data 2024-09-18T14:52:18.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:18.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj77-p2 td/osd-recovery-prio/data 2024-09-18T14:52:19.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:19.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj78-p2 td/osd-recovery-prio/data 2024-09-18T14:52:19.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:19.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj79-p2 td/osd-recovery-prio/data 2024-09-18T14:52:19.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:19.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj80-p2 td/osd-recovery-prio/data 2024-09-18T14:52:20.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:20.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj81-p2 td/osd-recovery-prio/data 2024-09-18T14:52:20.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:20.699 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj82-p2 td/osd-recovery-prio/data 2024-09-18T14:52:21.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:21.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj83-p2 td/osd-recovery-prio/data 2024-09-18T14:52:21.432 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:21.432 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj84-p2 td/osd-recovery-prio/data 2024-09-18T14:52:21.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:21.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj85-p2 td/osd-recovery-prio/data 2024-09-18T14:52:22.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:22.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj86-p2 td/osd-recovery-prio/data 2024-09-18T14:52:22.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:22.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj87-p2 td/osd-recovery-prio/data 2024-09-18T14:52:22.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:22.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj88-p2 td/osd-recovery-prio/data 2024-09-18T14:52:23.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:23.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj89-p2 td/osd-recovery-prio/data 2024-09-18T14:52:23.491 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:23.491 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj90-p2 td/osd-recovery-prio/data 2024-09-18T14:52:23.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:23.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj91-p2 td/osd-recovery-prio/data 2024-09-18T14:52:24.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:24.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj92-p2 td/osd-recovery-prio/data 2024-09-18T14:52:24.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:24.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj93-p2 td/osd-recovery-prio/data 2024-09-18T14:52:25.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:25.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj94-p2 td/osd-recovery-prio/data 2024-09-18T14:52:25.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:25.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj95-p2 td/osd-recovery-prio/data 2024-09-18T14:52:25.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:25.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj96-p2 td/osd-recovery-prio/data 2024-09-18T14:52:26.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:26.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj97-p2 td/osd-recovery-prio/data 2024-09-18T14:52:26.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:26.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj98-p2 td/osd-recovery-prio/data 2024-09-18T14:52:26.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:26.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj99-p2 td/osd-recovery-prio/data 2024-09-18T14:52:27.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:27.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj100-p2 td/osd-recovery-prio/data 2024-09-18T14:52:27.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:27.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj101-p2 td/osd-recovery-prio/data 2024-09-18T14:52:27.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:27.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj102-p2 td/osd-recovery-prio/data 2024-09-18T14:52:28.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:28.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj103-p2 td/osd-recovery-prio/data 2024-09-18T14:52:28.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:28.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj104-p2 td/osd-recovery-prio/data 2024-09-18T14:52:29.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:29.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj105-p2 td/osd-recovery-prio/data 2024-09-18T14:52:29.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:29.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj106-p2 td/osd-recovery-prio/data 2024-09-18T14:52:29.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:29.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj107-p2 td/osd-recovery-prio/data 2024-09-18T14:52:30.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:30.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj108-p2 td/osd-recovery-prio/data 2024-09-18T14:52:30.483 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:30.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj109-p2 td/osd-recovery-prio/data 2024-09-18T14:52:30.860 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:30.860 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj110-p2 td/osd-recovery-prio/data 2024-09-18T14:52:31.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:31.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj111-p2 td/osd-recovery-prio/data 2024-09-18T14:52:31.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:31.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj112-p2 td/osd-recovery-prio/data 2024-09-18T14:52:31.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:31.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj113-p2 td/osd-recovery-prio/data 2024-09-18T14:52:32.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:32.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj114-p2 td/osd-recovery-prio/data 2024-09-18T14:52:32.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:32.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj115-p2 td/osd-recovery-prio/data 2024-09-18T14:52:33.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:33.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj116-p2 td/osd-recovery-prio/data 2024-09-18T14:52:33.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:33.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj117-p2 td/osd-recovery-prio/data 2024-09-18T14:52:33.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:33.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj118-p2 td/osd-recovery-prio/data 2024-09-18T14:52:34.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:34.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj119-p2 td/osd-recovery-prio/data 2024-09-18T14:52:34.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:34.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj120-p2 td/osd-recovery-prio/data 2024-09-18T14:52:34.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:34.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj121-p2 td/osd-recovery-prio/data 2024-09-18T14:52:35.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:35.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj122-p2 td/osd-recovery-prio/data 2024-09-18T14:52:35.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:35.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj123-p2 td/osd-recovery-prio/data 2024-09-18T14:52:35.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:35.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj124-p2 td/osd-recovery-prio/data 2024-09-18T14:52:36.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:36.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj125-p2 td/osd-recovery-prio/data 2024-09-18T14:52:36.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:36.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj126-p2 td/osd-recovery-prio/data 2024-09-18T14:52:37.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:37.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj127-p2 td/osd-recovery-prio/data 2024-09-18T14:52:37.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:37.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj128-p2 td/osd-recovery-prio/data 2024-09-18T14:52:37.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:37.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj129-p2 td/osd-recovery-prio/data 2024-09-18T14:52:38.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:38.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj130-p2 td/osd-recovery-prio/data 2024-09-18T14:52:38.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:38.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj131-p2 td/osd-recovery-prio/data 2024-09-18T14:52:38.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:38.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj132-p2 td/osd-recovery-prio/data 2024-09-18T14:52:39.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:39.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj133-p2 td/osd-recovery-prio/data 2024-09-18T14:52:39.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:39.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj134-p2 td/osd-recovery-prio/data 2024-09-18T14:52:39.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:39.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj135-p2 td/osd-recovery-prio/data 2024-09-18T14:52:40.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:40.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj136-p2 td/osd-recovery-prio/data 2024-09-18T14:52:40.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:40.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj137-p2 td/osd-recovery-prio/data 2024-09-18T14:52:40.993 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:40.993 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj138-p2 td/osd-recovery-prio/data 2024-09-18T14:52:41.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:41.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj139-p2 td/osd-recovery-prio/data 2024-09-18T14:52:41.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:41.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj140-p2 td/osd-recovery-prio/data 2024-09-18T14:52:41.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:41.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj141-p2 td/osd-recovery-prio/data 2024-09-18T14:52:42.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:42.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj142-p2 td/osd-recovery-prio/data 2024-09-18T14:52:42.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:42.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj143-p2 td/osd-recovery-prio/data 2024-09-18T14:52:43.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:43.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj144-p2 td/osd-recovery-prio/data 2024-09-18T14:52:43.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:43.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj145-p2 td/osd-recovery-prio/data 2024-09-18T14:52:43.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:43.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj146-p2 td/osd-recovery-prio/data 2024-09-18T14:52:44.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:44.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj147-p2 td/osd-recovery-prio/data 2024-09-18T14:52:44.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:44.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj148-p2 td/osd-recovery-prio/data 2024-09-18T14:52:44.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:44.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj149-p2 td/osd-recovery-prio/data 2024-09-18T14:52:45.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:45.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj150-p2 td/osd-recovery-prio/data 2024-09-18T14:52:45.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:45.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj151-p2 td/osd-recovery-prio/data 2024-09-18T14:52:45.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:45.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj152-p2 td/osd-recovery-prio/data 2024-09-18T14:52:46.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:46.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj153-p2 td/osd-recovery-prio/data 2024-09-18T14:52:46.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:46.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj154-p2 td/osd-recovery-prio/data 2024-09-18T14:52:46.985 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:46.985 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj155-p2 td/osd-recovery-prio/data 2024-09-18T14:52:47.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:47.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj156-p2 td/osd-recovery-prio/data 2024-09-18T14:52:47.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:47.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj157-p2 td/osd-recovery-prio/data 2024-09-18T14:52:47.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:47.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj158-p2 td/osd-recovery-prio/data 2024-09-18T14:52:48.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:48.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj159-p2 td/osd-recovery-prio/data 2024-09-18T14:52:48.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:48.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj160-p2 td/osd-recovery-prio/data 2024-09-18T14:52:49.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:49.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj161-p2 td/osd-recovery-prio/data 2024-09-18T14:52:49.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:49.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj162-p2 td/osd-recovery-prio/data 2024-09-18T14:52:49.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:49.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj163-p2 td/osd-recovery-prio/data 2024-09-18T14:52:50.172 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:50.172 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj164-p2 td/osd-recovery-prio/data 2024-09-18T14:52:50.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:50.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj165-p2 td/osd-recovery-prio/data 2024-09-18T14:52:51.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:51.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj166-p2 td/osd-recovery-prio/data 2024-09-18T14:52:51.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:51.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj167-p2 td/osd-recovery-prio/data 2024-09-18T14:52:51.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:51.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj168-p2 td/osd-recovery-prio/data 2024-09-18T14:52:52.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:52.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj169-p2 td/osd-recovery-prio/data 2024-09-18T14:52:52.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:52.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj170-p2 td/osd-recovery-prio/data 2024-09-18T14:52:52.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:52.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj171-p2 td/osd-recovery-prio/data 2024-09-18T14:52:53.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:53.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj172-p2 td/osd-recovery-prio/data 2024-09-18T14:52:53.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:53.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj173-p2 td/osd-recovery-prio/data 2024-09-18T14:52:53.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:53.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj174-p2 td/osd-recovery-prio/data 2024-09-18T14:52:54.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:54.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj175-p2 td/osd-recovery-prio/data 2024-09-18T14:52:54.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:54.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj176-p2 td/osd-recovery-prio/data 2024-09-18T14:52:54.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:54.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj177-p2 td/osd-recovery-prio/data 2024-09-18T14:52:55.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:55.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj178-p2 td/osd-recovery-prio/data 2024-09-18T14:52:55.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:55.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj179-p2 td/osd-recovery-prio/data 2024-09-18T14:52:56.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:56.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj180-p2 td/osd-recovery-prio/data 2024-09-18T14:52:56.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:56.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj181-p2 td/osd-recovery-prio/data 2024-09-18T14:52:57.022 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:57.022 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj182-p2 td/osd-recovery-prio/data 2024-09-18T14:52:57.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:57.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj183-p2 td/osd-recovery-prio/data 2024-09-18T14:52:57.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:57.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj184-p2 td/osd-recovery-prio/data 2024-09-18T14:52:58.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:58.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj185-p2 td/osd-recovery-prio/data 2024-09-18T14:52:58.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:58.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj186-p2 td/osd-recovery-prio/data 2024-09-18T14:52:58.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:58.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj187-p2 td/osd-recovery-prio/data 2024-09-18T14:52:59.155 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:59.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj188-p2 td/osd-recovery-prio/data 2024-09-18T14:52:59.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:59.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj189-p2 td/osd-recovery-prio/data 2024-09-18T14:52:59.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:52:59.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj190-p2 td/osd-recovery-prio/data 2024-09-18T14:53:00.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:53:00.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj191-p2 td/osd-recovery-prio/data 2024-09-18T14:53:00.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:53:00.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj192-p2 td/osd-recovery-prio/data 2024-09-18T14:53:00.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:53:00.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj193-p2 td/osd-recovery-prio/data 2024-09-18T14:53:01.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:53:01.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj194-p2 td/osd-recovery-prio/data 2024-09-18T14:53:01.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:53:01.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj195-p2 td/osd-recovery-prio/data 2024-09-18T14:53:02.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:53:02.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj196-p2 td/osd-recovery-prio/data 2024-09-18T14:53:02.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:53:02.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj197-p2 td/osd-recovery-prio/data 2024-09-18T14:53:02.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:53:02.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj198-p2 td/osd-recovery-prio/data 2024-09-18T14:53:03.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:53:03.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj199-p2 td/osd-recovery-prio/data 2024-09-18T14:53:03.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-18T14:53:03.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj200-p2 td/osd-recovery-prio/data 2024-09-18T14:53:03.798 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: expr 2 + 1 2024-09-18T14:53:03.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: p=3 2024-09-18T14:53:03.801 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:424: TEST_recovery_pool_priority: get_not_primary test1 obj1-p1 2024-09-18T14:53:03.801 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test1 2024-09-18T14:53:03.801 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1-p1 2024-09-18T14:53:03.801 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test1 obj1-p1 2024-09-18T14:53:03.801 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test1 2024-09-18T14:53:03.801 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1-p1 2024-09-18T14:53:03.802 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test1 obj1-p1 2024-09-18T14:53:03.802 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T14:53:04.180 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-18T14:53:04.180 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test1 obj1-p1 2024-09-18T14:53:04.180 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-18T14:53:04.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:424: TEST_recovery_pool_priority: local otherosd=null 2024-09-18T14:53:04.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:426: TEST_recovery_pool_priority: ceph pg dump pgs 2024-09-18T14:53:04.950 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:53:04.950 INFO:tasks.workunit.client.0.smithi047.stdout:2.0 188 0 0 0 0 1971322880 0 0 564 0 564 active+clean 2024-09-18T14:50:33.516986+0000 39'564 39:604 [0] 0 [0] 0 0'0 2024-09-18T14:50:14.761688+0000 0'0 2024-09-18T14:50:14.761688+0000 0 0 periodic scrub scheduled @ 2024-09-19T15:42:56.420018+0000 0 0 2024-09-18T14:53:04.950 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-09-18T14:50:31.883482+0000 39'600 39:647 [1] 1 [1] 1 0'0 2024-09-18T14:50:12.444513+0000 0'0 2024-09-18T14:50:12.444513+0000 0 0 periodic scrub scheduled @ 2024-09-20T02:42:57.750471+0000 0 0 2024-09-18T14:53:04.950 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:53:04.950 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:53:04.951 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:53:04.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:427: TEST_recovery_pool_priority: ERRORS=0 2024-09-18T14:53:04.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:429: TEST_recovery_pool_priority: ceph osd pool set test1 size 2 2024-09-18T14:53:05.735 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 1 size to 2 2024-09-18T14:53:05.746 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:430: TEST_recovery_pool_priority: ceph osd pool set test2 size 2 2024-09-18T14:53:07.393 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 2 size to 2 2024-09-18T14:53:07.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:433: TEST_recovery_pool_priority: ceph pg dump pgs 2024-09-18T14:53:07.834 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:53:07.834 INFO:tasks.workunit.client.0.smithi047.stdout:2.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-09-18T14:50:33.516986+0000 39'600 39:640 [0] 0 [0] 0 0'0 2024-09-18T14:50:14.761688+0000 0'0 2024-09-18T14:50:14.761688+0000 0 0 periodic scrub scheduled @ 2024-09-19T15:42:56.420018+0000 0 0 2024-09-18T14:53:07.834 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovering+undersized+degraded+remapped 2024-09-18T14:53:06.388689+0000 39'600 42:660 [1,0] 1 [1] 1 0'0 2024-09-18T14:50:12.444513+0000 0'0 2024-09-18T14:50:12.444513+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:55:51.091174+0000 0 0 2024-09-18T14:53:07.834 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:53:07.834 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:53:07.834 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:53:07.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:436: TEST_recovery_pool_priority: set -o pipefail 2024-09-18T14:53:07.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:437: TEST_recovery_pool_priority: count=0 2024-09-18T14:53:07.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: true 2024-09-18T14:53:07.845 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2024-09-18T14:53:07.845 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2024-09-18T14:53:07.845 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: grep -c true 2024-09-18T14:53:08.439 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:53:08.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: test 1 == 2 2024-09-18T14:53:08.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:445: TEST_recovery_pool_priority: sleep 2 2024-09-18T14:53:10.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:446: TEST_recovery_pool_priority: test 0 -eq 10 2024-09-18T14:53:10.451 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:451: TEST_recovery_pool_priority: expr 0 + 1 2024-09-18T14:53:10.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:451: TEST_recovery_pool_priority: count=1 2024-09-18T14:53:10.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: true 2024-09-18T14:53:10.454 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2024-09-18T14:53:10.454 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2024-09-18T14:53:10.454 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: grep -c true 2024-09-18T14:53:11.013 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:53:11.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: test 2 == 2 2024-09-18T14:53:11.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:443: TEST_recovery_pool_priority: break 2024-09-18T14:53:11.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:453: TEST_recovery_pool_priority: set +o pipefail 2024-09-18T14:53:11.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:454: TEST_recovery_pool_priority: ceph pg dump pgs 2024-09-18T14:53:11.501 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:53:11.501 INFO:tasks.workunit.client.0.smithi047.stdout:2.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovering+undersized+degraded+remapped 2024-09-18T14:53:09.562250+0000 39'600 46:657 [0,1] 0 [0] 0 0'0 2024-09-18T14:50:14.761688+0000 0'0 2024-09-18T14:50:14.761688+0000 0 0 periodic scrub scheduled @ 2024-09-19T22:33:53.538471+0000 0 0 2024-09-18T14:53:11.501 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovering+undersized+degraded+remapped 2024-09-18T14:53:06.388689+0000 39'600 42:660 [1,0] 1 [1] 1 0'0 2024-09-18T14:50:12.444513+0000 0'0 2024-09-18T14:50:12.444513+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:55:51.091174+0000 0 0 2024-09-18T14:53:11.501 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:53:11.501 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:53:11.502 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:53:11.513 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: get_asok_path osd.1 2024-09-18T14:53:11.513 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T14:53:11.513 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T14:53:11.513 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:53:11.513 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:53:11.513 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215238 2024-09-18T14:53:11.513 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.215238/ceph-osd.1.asok 2024-09-18T14:53:11.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: CEPH_ARGS= 2024-09-18T14:53:11.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T14:53:11.597 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 2024-09-18T14:53:11.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:457: TEST_recovery_pool_priority: echo osd.1 2024-09-18T14:53:11.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:458: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2024-09-18T14:53:11.598 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T14:53:11.598 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T14:53:11.598 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T14:53:11.598 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T14:53:11.598 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T14:53:11.598 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T14:53:11.598 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:53:11.599 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T14:53:11.599 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 191, 2024-09-18T14:53:11.599 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T14:53:11.599 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:53:11.599 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:53:11.599 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:53:11.599 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T14:53:11.599 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T14:53:11.599 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T14:53:11.599 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T14:53:11.599 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T14:53:11.599 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:53:11.599 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "2.0", 2024-09-18T14:53:11.599 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 192, 2024-09-18T14:53:11.599 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T14:53:11.600 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:53:11.600 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:53:11.600 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:53:11.600 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T14:53:11.601 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:459: TEST_recovery_pool_priority: get_asok_path osd.0 2024-09-18T14:53:11.601 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T14:53:11.601 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T14:53:11.601 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:53:11.601 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:53:11.601 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215238 2024-09-18T14:53:11.601 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.215238/ceph-osd.0.asok 2024-09-18T14:53:11.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:459: TEST_recovery_pool_priority: CEPH_ARGS= 2024-09-18T14:53:11.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:459: TEST_recovery_pool_priority: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.0.asok dump_recovery_reservations 2024-09-18T14:53:11.682 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 2024-09-18T14:53:11.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:460: TEST_recovery_pool_priority: echo osd.0 2024-09-18T14:53:11.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:461: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2024-09-18T14:53:11.683 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T14:53:11.683 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T14:53:11.683 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T14:53:11.683 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T14:53:11.683 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T14:53:11.683 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T14:53:11.683 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:53:11.683 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "2.0", 2024-09-18T14:53:11.683 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 192, 2024-09-18T14:53:11.683 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T14:53:11.684 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:53:11.684 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:53:11.684 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T14:53:11.684 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T14:53:11.684 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T14:53:11.684 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T14:53:11.684 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T14:53:11.684 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T14:53:11.684 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T14:53:11.684 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T14:53:11.684 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 191, 2024-09-18T14:53:11.684 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T14:53:11.684 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:53:11.685 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T14:53:11.685 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T14:53:11.685 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T14:53:11.685 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:464: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2024-09-18T14:53:11.685 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:464: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].item' 2024-09-18T14:53:11.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:464: TEST_recovery_pool_priority: eval 'ITEM="1.0"' 2024-09-18T14:53:11.686 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:464: TEST_recovery_pool_priority: ITEM=1.0 2024-09-18T14:53:11.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:465: TEST_recovery_pool_priority: '[' 1.0 '!=' 1.0 ']' 2024-09-18T14:53:11.687 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:470: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2024-09-18T14:53:11.687 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:470: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].prio' 2024-09-18T14:53:11.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:470: TEST_recovery_pool_priority: PRIO=191 2024-09-18T14:53:11.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:471: TEST_recovery_pool_priority: '[' 191 '!=' 191 ']' 2024-09-18T14:53:11.690 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:479: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2024-09-18T14:53:11.690 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:479: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].item' 2024-09-18T14:53:11.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:479: TEST_recovery_pool_priority: eval 'ITEM="1.0"' 2024-09-18T14:53:11.693 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:479: TEST_recovery_pool_priority: ITEM=1.0 2024-09-18T14:53:11.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:480: TEST_recovery_pool_priority: '[' 1.0 '!=' 1.0 ']' 2024-09-18T14:53:11.694 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:485: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2024-09-18T14:53:11.694 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:485: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2024-09-18T14:53:11.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:485: TEST_recovery_pool_priority: PRIO=191 2024-09-18T14:53:11.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:486: TEST_recovery_pool_priority: '[' 191 '!=' 191 ']' 2024-09-18T14:53:11.697 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:494: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2024-09-18T14:53:11.697 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:494: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].item' 2024-09-18T14:53:11.699 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:494: TEST_recovery_pool_priority: eval 'ITEM="2.0"' 2024-09-18T14:53:11.699 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:494: TEST_recovery_pool_priority: ITEM=2.0 2024-09-18T14:53:11.699 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:495: TEST_recovery_pool_priority: '[' 2.0 '!=' 2.0 ']' 2024-09-18T14:53:11.700 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:500: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2024-09-18T14:53:11.700 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:500: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].prio' 2024-09-18T14:53:11.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:500: TEST_recovery_pool_priority: PRIO=192 2024-09-18T14:53:11.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:501: TEST_recovery_pool_priority: '[' 192 '!=' 192 ']' 2024-09-18T14:53:11.703 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:509: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2024-09-18T14:53:11.703 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:509: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].item' 2024-09-18T14:53:11.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:509: TEST_recovery_pool_priority: eval 'ITEM="2.0"' 2024-09-18T14:53:11.705 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:509: TEST_recovery_pool_priority: ITEM=2.0 2024-09-18T14:53:11.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:510: TEST_recovery_pool_priority: '[' 2.0 '!=' 2.0 ']' 2024-09-18T14:53:11.706 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:515: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2024-09-18T14:53:11.706 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:515: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2024-09-18T14:53:11.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:515: TEST_recovery_pool_priority: PRIO=192 2024-09-18T14:53:11.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:516: TEST_recovery_pool_priority: '[' 192 '!=' 192 ']' 2024-09-18T14:53:11.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:523: TEST_recovery_pool_priority: wait_for_clean 2024-09-18T14:53:11.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:53:11.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:53:11.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:53:11.710 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:53:11.710 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:53:11.710 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:53:11.710 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:53:11.710 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:53:11.711 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:53:11.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T14:53:11.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:53:11.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:53:11.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:53:11.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:53:11.801 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:53:12.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:53:12.271 INFO:tasks.workunit.client.0.smithi047.stderr:1' 2024-09-18T14:53:12.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:53:12.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:53:12.271 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:53:12.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836525 2024-09-18T14:53:12.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836525 2024-09-18T14:53:12.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836525' 2024-09-18T14:53:12.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:53:12.633 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:53:12.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673003 2024-09-18T14:53:12.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673003 2024-09-18T14:53:12.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836525 1-42949673003' 2024-09-18T14:53:12.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:53:12.748 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836525 2024-09-18T14:53:12.748 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:53:12.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:53:12.749 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836525 2024-09-18T14:53:12.750 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:53:12.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836525 2024-09-18T14:53:12.751 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836525 2024-09-18T14:53:12.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836525' 2024-09-18T14:53:12.751 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:53:13.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836523 -lt 21474836525 2024-09-18T14:53:13.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:53:14.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:53:14.450 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:53:15.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836525 -lt 21474836525 2024-09-18T14:53:15.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:53:15.041 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673003 2024-09-18T14:53:15.041 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:53:15.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:53:15.042 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673003 2024-09-18T14:53:15.043 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:53:15.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673003 2024-09-18T14:53:15.044 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949673003 2024-09-18T14:53:15.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673003' 2024-09-18T14:53:15.044 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:53:15.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673003 -lt 42949673003 2024-09-18T14:53:15.565 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:53:15.565 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:53:15.565 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:53:16.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-09-18T14:53:16.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:53:16.335 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:53:16.335 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:53:16.335 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:53:16.335 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:53:16.335 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:53:16.335 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:53:16.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:53:16.811 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:53:16.811 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:53:16.811 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:53:17.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:53:17.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-18T14:53:17.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T14:53:17.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-18T14:53:17.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:53:17.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:53:17.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:53:17.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:53:17.447 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:53:17.447 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:53:17.447 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:53:17.447 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:53:17.447 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:53:17.447 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:53:17.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:53:17.808 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:53:17.808 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:53:17.808 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:53:18.476 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:53:18.476 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:53:18.476 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:53:18.477 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:53:18.477 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:53:18.477 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:53:18.477 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:53:18.477 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:53:18.477 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:53:19.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27258406 2024-09-18T14:53:19.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27258406 '!=' null 2024-09-18T14:53:19.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:53:19.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:53:19.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:53:19.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:53:19.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:53:19.524 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:53:19.524 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:53:19.524 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:53:19.524 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:53:19.524 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:53:19.524 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:53:19.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:53:19.928 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:53:19.928 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:53:19.928 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:53:20.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:53:20.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:53:20.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:53:20.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:53:20.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:53:20.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:53:20.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:53:20.653 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:53:20.653 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:53:21.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29876642 2024-09-18T14:53:21.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29876642 '!=' null 2024-09-18T14:53:21.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:53:21.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:53:21.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:53:21.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:53:21.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:53:21.277 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:53:21.277 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:53:21.277 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:53:21.277 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:53:21.277 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:53:21.277 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:53:21.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:53:21.764 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:53:21.765 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:53:21.765 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:53:22.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:53:22.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:53:22.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:53:22.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:53:22.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:53:22.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:53:22.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:53:22.383 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:53:22.383 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:53:23.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=34946802 2024-09-18T14:53:23.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 34946802 '!=' null 2024-09-18T14:53:23.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:53:23.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:53:23.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:53:23.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:53:23.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:53:23.244 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:53:23.244 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:53:23.244 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:53:23.244 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:53:23.244 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:53:23.244 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:53:23.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:53:23.603 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:53:23.603 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:53:23.603 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:53:24.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:53:24.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:53:24.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:53:24.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:53:24.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:53:24.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:53:24.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:53:24.218 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:53:24.218 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:53:24.907 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462904 2024-09-18T14:53:24.907 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462904 '!=' null 2024-09-18T14:53:24.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:53:24.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:53:24.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:53:25.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:53:25.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:53:25.010 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:53:25.010 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:53:25.010 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:53:25.010 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:53:25.010 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:53:25.010 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:53:25.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:53:25.514 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:53:25.514 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:53:25.514 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:53:26.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:53:26.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:53:26.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:53:26.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:53:26.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:53:26.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:53:26.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:53:26.164 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:53:26.164 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:53:26.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24463064 2024-09-18T14:53:26.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24463064 '!=' null 2024-09-18T14:53:26.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:53:26.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:53:26.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:53:26.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:53:26.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:53:26.824 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:53:26.824 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:53:26.824 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:53:26.824 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:53:26.824 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:53:26.824 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:53:27.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:53:27.305 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:53:27.305 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:53:27.305 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:53:27.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:53:27.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:53:27.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:53:27.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:53:27.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:53:27.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:53:27.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:53:27.818 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:53:27.818 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:53:28.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24463064 2024-09-18T14:53:28.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24463064 '!=' null 2024-09-18T14:53:28.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:53:28.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:53:28.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:53:28.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:53:28.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:53:28.496 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:53:28.496 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:53:28.496 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:53:28.496 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:53:28.497 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:53:28.497 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:53:28.946 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:53:28.947 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:53:28.947 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:53:28.947 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:53:29.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:53:29.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:53:29.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:53:29.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:53:29.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:53:29.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:53:29.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:53:29.560 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:53:29.560 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:53:30.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27957908 2024-09-18T14:53:30.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27957908 '!=' null 2024-09-18T14:53:30.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:53:30.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:53:30.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:53:30.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:53:30.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:53:30.264 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:53:30.264 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:53:30.264 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:53:30.264 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:53:30.264 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:53:30.264 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:53:30.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:53:30.767 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:53:30.767 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:53:30.767 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:53:31.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:53:31.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:53:31.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:53:31.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:53:31.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:53:31.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:53:31.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:53:31.440 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:53:31.440 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:53:32.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29705345 2024-09-18T14:53:32.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29705345 '!=' null 2024-09-18T14:53:32.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:53:32.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:53:32.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:53:32.242 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:53:32.242 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:53:32.243 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:53:32.243 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:53:32.243 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:53:32.243 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:53:32.243 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:53:32.244 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:53:32.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:53:32.619 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:53:32.619 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:53:32.619 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:53:33.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:53:33.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:53:33.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:53:33.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:53:33.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:53:33.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:53:33.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:53:33.324 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:53:33.324 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:53:33.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28831713 2024-09-18T14:53:33.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28831713 '!=' null 2024-09-18T14:53:33.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:53:33.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:53:33.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:53:34.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:53:34.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:53:34.025 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:53:34.025 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:53:34.025 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:53:34.025 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:53:34.026 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:53:34.026 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:53:34.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:53:34.386 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:53:34.386 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:53:34.387 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:53:34.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:53:34.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:53:34.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:53:34.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:53:34.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:53:34.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:53:34.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:53:34.917 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:53:34.917 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:53:35.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22715872 2024-09-18T14:53:35.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22715872 '!=' null 2024-09-18T14:53:35.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:53:35.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:53:35.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:53:35.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:53:35.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:53:35.675 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:53:35.675 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:53:35.675 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:53:35.675 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:53:35.675 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:53:35.676 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:53:36.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:53:36.114 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:53:36.115 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:53:36.115 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:53:36.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:53:36.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:53:36.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:53:36.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:53:36.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:53:36.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:53:36.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:53:36.745 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:53:36.745 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:53:37.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27958043 2024-09-18T14:53:37.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27958043 '!=' null 2024-09-18T14:53:37.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:53:37.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:53:37.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:53:37.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:53:37.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:53:37.454 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:53:37.454 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:53:37.454 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:53:37.454 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:53:37.454 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:53:37.454 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:53:37.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:53:37.819 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:53:37.819 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:53:37.819 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:53:38.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:53:38.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:53:38.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:53:38.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:53:38.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:53:38.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:53:38.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:53:38.418 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:53:38.419 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:53:39.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28831634 2024-09-18T14:53:39.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28831634 '!=' null 2024-09-18T14:53:39.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:53:39.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:53:39.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:53:39.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:53:39.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:53:39.240 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:53:39.240 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:53:39.240 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:53:39.240 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:53:39.240 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:53:39.240 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:53:39.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:53:39.606 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:53:39.606 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:53:39.607 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:53:40.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:53:40.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:53:40.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:53:40.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:53:40.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:53:40.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:53:40.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:53:40.102 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:53:40.102 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:53:40.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28831566 2024-09-18T14:53:40.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28831566 '!=' null 2024-09-18T14:53:40.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:53:40.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:53:40.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:53:41.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:53:41.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:53:41.032 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:53:41.033 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:53:41.033 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:53:41.033 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:53:41.033 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:53:41.033 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:53:41.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:53:41.487 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:53:41.488 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:53:41.488 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:53:42.080 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:53:42.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:53:42.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:53:42.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:53:42.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:53:42.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:53:42.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:53:42.081 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:53:42.081 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:53:42.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28831566 2024-09-18T14:53:42.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28831566 '!=' null 2024-09-18T14:53:42.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:53:42.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:53:42.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:53:42.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:53:42.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:53:42.711 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:53:42.711 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:53:42.711 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:53:42.711 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:53:42.711 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:53:42.711 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:53:43.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:53:43.071 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:53:43.071 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:53:43.072 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:53:43.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:53:43.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:53:43.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:53:43.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:53:43.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:53:43.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:53:43.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:53:43.571 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:53:43.571 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:53:44.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28831637 2024-09-18T14:53:44.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28831637 '!=' null 2024-09-18T14:53:44.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:53:44.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:53:44.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:53:44.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:53:44.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:53:44.305 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:53:44.305 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:53:44.305 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:53:44.305 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:53:44.305 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:53:44.305 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:53:44.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:53:44.672 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:53:44.673 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:53:44.673 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:53:45.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:53:45.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:53:45.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:53:45.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:53:45.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:53:45.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:53:45.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:53:45.306 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:53:45.306 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:53:45.948 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23589483 2024-09-18T14:53:45.948 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23589483 '!=' null 2024-09-18T14:53:45.948 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:53:45.948 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:53:45.948 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:53:46.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:53:46.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:53:46.050 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:53:46.050 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:53:46.050 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:53:46.050 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:53:46.050 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:53:46.050 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:53:46.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:53:46.413 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:53:46.413 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:53:46.414 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:53:46.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:53:46.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:53:46.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:53:46.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:53:46.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:53:46.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:53:46.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:53:46.953 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:53:46.953 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:53:47.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29705294 2024-09-18T14:53:47.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29705294 '!=' null 2024-09-18T14:53:47.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:53:47.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:53:47.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:53:47.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:53:47.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:53:47.721 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:53:47.722 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:53:47.722 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:53:47.722 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:53:47.722 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:53:47.722 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:53:48.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:53:48.215 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:53:48.215 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:53:48.215 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:53:48.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:53:48.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:53:48.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:53:48.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:53:48.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:53:48.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:53:48.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:53:48.824 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:53:48.824 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:53:49.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29705161 2024-09-18T14:53:49.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29705161 '!=' null 2024-09-18T14:53:49.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:53:49.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:53:49.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:53:49.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:53:49.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:53:49.517 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:53:49.517 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:53:49.517 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:53:49.518 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:53:49.518 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:53:49.518 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:53:49.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:53:49.918 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:53:49.918 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:53:49.918 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:53:50.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:53:50.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:53:50.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:53:50.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:53:50.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:53:50.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:53:50.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:53:50.423 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:53:50.423 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:53:50.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28831458 2024-09-18T14:53:50.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28831458 '!=' null 2024-09-18T14:53:50.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:53:50.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:53:50.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:53:51.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:53:51.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:53:51.074 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:53:51.074 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:53:51.074 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:53:51.074 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:53:51.074 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:53:51.074 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:53:51.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:53:51.530 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:53:51.530 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:53:51.530 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:53:52.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:53:52.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:53:52.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:53:52.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:53:52.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:53:52.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:53:52.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:53:52.025 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:53:52.026 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:53:52.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28831467 2024-09-18T14:53:52.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28831467 '!=' null 2024-09-18T14:53:52.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:53:52.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:53:52.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:53:53.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:53:53.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:53:53.046 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:53:53.046 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:53:53.046 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:53:53.046 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:53:53.046 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:53:53.046 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:53:53.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:53:53.454 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:53:53.454 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:53:53.454 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:53:54.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:53:54.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:53:54.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:53:54.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:53:54.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:53:54.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:53:54.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:53:54.294 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:53:54.294 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:53:55.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22715273 2024-09-18T14:53:55.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22715273 '!=' null 2024-09-18T14:53:55.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:53:55.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:53:55.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:53:55.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:53:55.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:53:55.111 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:53:55.111 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:53:55.111 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:53:55.111 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:53:55.111 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:53:55.111 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:53:55.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:53:55.693 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:53:55.693 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:53:55.693 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:53:56.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:53:56.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:53:56.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:53:56.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:53:56.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:53:56.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:53:56.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:53:56.190 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:53:56.190 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:53:56.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28830941 2024-09-18T14:53:56.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28830941 '!=' null 2024-09-18T14:53:56.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:53:56.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:53:56.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:53:57.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:53:57.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:53:57.053 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:53:57.053 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:53:57.053 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:53:57.053 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:53:57.054 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:53:57.054 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:53:57.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:53:57.433 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:53:57.433 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:53:57.433 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:53:57.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:53:57.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:53:57.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:53:57.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:53:57.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:53:57.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:53:57.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:53:57.937 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:53:57.937 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:53:58.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28830941 2024-09-18T14:53:58.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28830941 '!=' null 2024-09-18T14:53:58.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:53:58.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:53:58.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:53:58.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:53:58.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:53:58.851 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:53:58.851 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:53:58.851 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:53:58.851 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:53:58.851 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:53:58.851 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:53:59.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:53:59.251 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:53:59.251 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:53:59.251 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:53:59.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:53:59.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:53:59.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:53:59.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:53:59.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:53:59.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:53:59.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:53:59.764 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:53:59.764 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:54:00.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27956734 2024-09-18T14:54:00.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27956734 '!=' null 2024-09-18T14:54:00.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:54:00.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:54:00.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:54:00.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:54:00.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:00.632 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:00.632 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:00.632 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:00.633 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:00.633 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:00.633 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:01.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:54:01.089 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:01.089 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:01.089 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:01.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:54:01.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:54:01.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:54:01.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:54:01.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:54:01.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:54:01.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:54:01.638 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:54:01.638 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:54:02.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28830460 2024-09-18T14:54:02.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28830460 '!=' null 2024-09-18T14:54:02.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:54:02.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:54:02.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:54:02.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:54:02.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:02.440 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:02.440 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:02.440 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:02.440 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:02.440 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:02.440 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:02.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:54:02.916 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:02.916 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:02.917 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:03.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:54:03.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:54:03.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:54:03.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:54:03.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:54:03.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:54:03.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:54:03.531 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:54:03.531 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:54:04.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29704082 2024-09-18T14:54:04.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29704082 '!=' null 2024-09-18T14:54:04.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:54:04.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:54:04.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:54:04.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:54:04.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:04.385 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:04.386 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:04.386 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:04.386 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:04.386 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:04.386 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:04.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:54:04.753 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:04.753 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:04.753 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:05.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:54:05.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:54:05.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:54:05.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:54:05.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:54:05.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:54:05.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:54:05.436 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:54:05.436 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:54:06.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23588520 2024-09-18T14:54:06.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23588520 '!=' null 2024-09-18T14:54:06.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:54:06.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:54:06.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:54:06.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:54:06.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:06.342 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:06.343 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:06.343 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:06.343 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:06.343 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:06.343 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:06.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:54:06.705 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:06.705 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:06.706 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:07.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:54:07.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:54:07.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:54:07.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:54:07.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:54:07.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:54:07.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:54:07.387 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:54:07.387 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:54:07.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28830606 2024-09-18T14:54:07.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28830606 '!=' null 2024-09-18T14:54:07.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:54:07.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:54:07.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:54:08.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:54:08.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:08.055 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:08.055 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:08.055 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:08.056 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:08.056 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:08.056 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:08.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:54:08.421 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:08.422 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:08.422 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:09.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:54:09.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:54:09.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:54:09.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:54:09.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:54:09.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:54:09.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:54:09.044 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:54:09.044 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:54:09.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27957059 2024-09-18T14:54:09.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27957059 '!=' null 2024-09-18T14:54:09.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:54:09.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:54:09.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:54:09.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:54:09.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:09.984 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:09.984 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:09.985 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:09.985 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:09.985 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:09.985 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:10.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:54:10.357 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:10.357 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:10.357 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:10.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:54:10.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:54:10.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:54:10.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:54:10.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:54:10.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:54:10.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:54:10.884 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:54:10.884 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:54:11.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28831201 2024-09-18T14:54:11.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28831201 '!=' null 2024-09-18T14:54:11.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:54:11.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:54:11.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:54:11.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:54:11.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:11.730 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:11.730 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:11.730 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:11.730 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:11.730 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:11.730 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:12.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:54:12.114 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:12.114 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:12.114 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:12.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:54:12.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:54:12.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:54:12.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:54:12.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:54:12.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:54:12.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:54:12.619 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:54:12.619 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:54:13.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28831189 2024-09-18T14:54:13.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28831189 '!=' null 2024-09-18T14:54:13.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:54:13.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:54:13.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:54:13.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:54:13.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:13.329 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:13.330 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:13.330 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:13.330 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:13.330 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:13.330 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:13.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:54:13.692 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:13.692 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:13.692 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:14.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:54:14.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:54:14.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:54:14.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:54:14.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:54:14.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:54:14.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:54:14.228 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:54:14.228 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:54:14.746 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28831189 2024-09-18T14:54:14.746 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28831189 '!=' null 2024-09-18T14:54:14.746 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:54:14.746 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:54:14.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:54:14.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:54:14.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:14.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:14.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:14.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:14.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:14.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:14.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:15.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:54:15.215 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:15.215 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:15.215 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:15.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:54:15.956 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:54:15.956 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:54:15.956 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:54:15.956 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:54:15.956 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:54:15.956 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:54:15.956 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:54:15.956 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:54:16.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27957640 2024-09-18T14:54:16.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27957640 '!=' null 2024-09-18T14:54:16.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:54:16.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:54:16.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:54:16.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:54:16.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:16.723 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:16.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:16.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:16.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:16.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:16.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:17.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:54:17.244 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:17.244 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:17.245 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:17.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:54:17.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:54:17.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:54:17.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:54:17.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:54:17.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:54:17.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:54:17.769 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:54:17.769 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:54:18.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27957640 2024-09-18T14:54:18.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27957640 '!=' null 2024-09-18T14:54:18.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:54:18.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:54:18.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:54:18.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:54:18.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:18.734 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:18.734 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:18.734 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:18.734 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:18.734 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:18.734 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:19.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:54:19.101 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:19.101 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:19.101 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:19.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:54:19.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:54:19.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:54:19.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:54:19.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:54:19.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:54:19.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:54:19.807 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:54:19.807 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:54:20.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27957566 2024-09-18T14:54:20.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27957566 '!=' null 2024-09-18T14:54:20.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:54:20.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:54:20.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:54:20.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:54:20.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:20.637 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:20.638 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:20.638 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:20.638 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:20.638 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:20.638 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:21.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:54:21.113 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:21.113 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:21.114 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:21.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:54:21.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:54:21.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:54:21.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:54:21.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:54:21.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:54:21.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:54:21.659 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:54:21.659 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:54:22.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28831177 2024-09-18T14:54:22.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28831177 '!=' null 2024-09-18T14:54:22.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:54:22.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:54:22.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:54:22.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:54:22.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:22.418 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:22.418 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:22.418 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:22.418 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:22.418 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:22.418 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:22.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:54:22.785 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:22.785 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:22.785 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:23.514 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:54:23.514 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:54:23.514 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:54:23.514 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:54:23.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:54:23.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:54:23.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:54:23.515 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:54:23.515 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:54:24.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28831184 2024-09-18T14:54:24.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28831184 '!=' null 2024-09-18T14:54:24.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:54:24.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:54:24.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:54:24.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:54:24.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:24.212 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:24.212 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:24.212 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:24.212 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:24.212 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:24.212 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:24.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:54:24.703 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:24.703 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:24.703 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:25.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:54:25.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:54:25.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:54:25.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:54:25.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:54:25.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:54:25.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:54:25.327 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:54:25.327 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:54:26.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22715291 2024-09-18T14:54:26.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22715291 '!=' null 2024-09-18T14:54:26.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:54:26.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:54:26.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:54:26.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:54:26.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:26.227 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:26.227 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:26.227 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:26.227 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:26.227 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:26.227 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:26.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:54:26.595 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:26.595 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:26.595 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:27.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:54:27.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:54:27.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:54:27.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:54:27.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:54:27.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:54:27.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:54:27.263 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:54:27.263 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:54:27.956 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27957395 2024-09-18T14:54:27.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27957395 '!=' null 2024-09-18T14:54:27.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:54:27.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:54:27.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:54:28.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:54:28.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:28.058 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:28.058 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:28.058 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:28.058 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:28.058 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:28.059 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:28.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:54:28.542 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:28.542 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:28.542 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:29.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:54:29.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:54:29.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:54:29.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:54:29.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:54:29.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:54:29.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:54:29.454 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:54:29.454 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:54:30.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28831044 2024-09-18T14:54:30.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28831044 '!=' null 2024-09-18T14:54:30.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:54:30.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:54:30.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:54:30.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:54:30.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:30.121 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:30.121 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:30.121 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:30.121 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:30.121 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:30.121 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:30.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:54:30.487 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:30.487 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:30.487 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:31.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:54:31.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:54:31.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:54:31.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:54:31.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:54:31.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:54:31.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:54:31.127 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:54:31.127 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:54:31.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29705065 2024-09-18T14:54:31.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29705065 '!=' null 2024-09-18T14:54:31.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:54:31.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:54:31.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:54:31.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:54:31.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:31.737 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:31.737 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:31.737 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:31.737 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:31.737 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:31.737 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:32.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:54:32.100 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:32.100 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:32.100 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:32.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:54:32.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:54:32.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:54:32.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:54:32.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:54:32.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:54:32.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:54:32.609 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:54:32.610 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:54:33.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28831304 2024-09-18T14:54:33.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28831304 '!=' null 2024-09-18T14:54:33.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:54:33.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:54:33.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:54:33.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:54:33.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:33.418 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:33.418 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:33.419 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:33.419 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:33.419 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:33.419 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:33.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:54:33.906 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:33.907 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:33.907 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:34.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:54:34.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:54:34.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:54:34.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:54:34.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:54:34.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:54:34.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:54:34.362 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:54:34.362 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:54:35.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22715577 2024-09-18T14:54:35.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22715577 '!=' null 2024-09-18T14:54:35.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:54:35.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:54:35.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:54:35.514 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:54:35.514 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:35.515 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:35.515 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:35.515 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:35.516 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:35.516 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:35.516 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:36.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:54:36.030 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:36.030 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:36.030 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:36.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:54:36.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:54:36.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:54:36.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:54:36.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:54:36.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:54:36.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:54:36.562 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:54:36.562 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:54:37.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27957842 2024-09-18T14:54:37.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27957842 '!=' null 2024-09-18T14:54:37.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:54:37.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:54:37.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:54:37.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:54:37.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:37.224 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:37.224 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:37.224 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:37.224 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:37.224 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:37.224 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:37.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:54:37.621 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:37.621 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:37.622 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:38.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:54:38.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:54:38.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:54:38.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:54:38.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:54:38.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:54:38.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:54:38.131 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:54:38.131 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:54:38.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27957842 2024-09-18T14:54:38.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27957842 '!=' null 2024-09-18T14:54:38.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:54:38.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:54:38.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:54:38.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:54:38.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:38.853 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:38.853 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:38.853 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:38.854 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:38.854 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:38.854 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:39.248 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:54:39.249 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:39.249 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:39.249 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:39.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:54:39.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:54:39.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:54:39.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:54:39.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:54:39.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:54:39.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:54:39.998 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:54:39.998 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:54:40.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28831562 2024-09-18T14:54:40.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28831562 '!=' null 2024-09-18T14:54:40.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:54:40.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:54:40.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:54:40.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:54:40.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:40.914 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:40.914 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:40.914 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:40.914 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:40.914 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:40.914 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:41.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:54:41.288 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:41.288 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:41.288 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:41.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:54:41.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:54:41.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:54:41.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:54:41.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:54:41.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:54:41.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:54:41.789 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:54:41.789 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:54:42.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28831562 2024-09-18T14:54:42.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28831562 '!=' null 2024-09-18T14:54:42.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:54:42.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:54:42.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:54:42.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:54:42.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:42.835 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:42.835 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:42.835 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:42.835 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:42.835 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:42.835 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:43.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:54:43.213 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:43.214 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:43.214 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:43.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:54:43.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:54:43.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:54:43.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:54:43.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:54:43.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:54:43.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:54:43.794 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:54:43.794 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:54:44.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27957878 2024-09-18T14:54:44.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27957878 '!=' null 2024-09-18T14:54:44.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:54:44.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:54:44.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:54:44.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:54:44.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:44.743 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:44.743 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:44.743 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:44.743 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:44.743 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:44.743 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:45.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:54:45.107 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:45.107 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:45.107 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:45.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:54:45.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:54:45.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:54:45.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:54:45.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:54:45.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:54:45.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:54:45.607 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:54:45.607 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:54:46.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22715886 2024-09-18T14:54:46.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22715886 '!=' null 2024-09-18T14:54:46.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:54:46.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:54:46.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:54:46.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:54:46.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:46.373 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:46.373 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:46.373 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:46.374 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:46.374 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:46.374 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:46.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:54:46.753 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:46.754 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:46.754 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:47.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:54:47.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:54:47.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:54:47.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:54:47.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:54:47.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:54:47.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:54:47.561 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:54:47.561 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:54:48.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27958042 2024-09-18T14:54:48.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27958042 '!=' null 2024-09-18T14:54:48.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:54:48.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:54:48.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:54:48.390 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:54:48.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:48.391 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:48.392 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:48.392 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:48.392 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:48.392 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:48.392 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:48.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:54:48.754 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:48.754 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:48.754 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:49.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:54:49.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:54:49.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:54:49.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:54:49.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:54:49.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:54:49.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:54:49.417 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:54:49.417 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:54:49.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28831721 2024-09-18T14:54:49.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28831721 '!=' null 2024-09-18T14:54:50.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:54:50.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:54:50.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:54:50.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:54:50.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:50.101 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:50.101 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:50.101 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:50.102 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:50.102 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:50.102 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:50.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:54:50.471 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:50.472 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:50.472 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:51.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:54:51.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:54:51.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:54:51.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:54:51.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:54:51.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:54:51.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:54:51.274 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:54:51.274 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:54:52.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28831757 2024-09-18T14:54:52.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28831757 '!=' null 2024-09-18T14:54:52.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:54:52.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:54:52.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:54:52.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:54:52.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:52.212 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:52.212 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:52.212 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:52.212 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:52.212 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:52.212 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:52.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:54:52.580 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:52.580 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:52.580 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:53.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:54:53.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:54:53.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:54:53.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:54:53.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:54:53.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:54:53.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:54:53.113 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:54:53.113 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:54:53.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27957931 2024-09-18T14:54:53.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27957931 '!=' null 2024-09-18T14:54:53.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:54:53.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:54:53.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:54:53.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:54:53.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:53.769 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:53.770 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:53.770 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:53.770 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:53.770 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:53.770 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:54.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:54:54.144 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:54.144 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:54.144 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:54.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:54:54.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:54:54.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:54:54.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:54:54.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:54:54.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:54:54.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:54:54.702 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:54:54.702 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:54:55.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22715808 2024-09-18T14:54:55.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22715808 '!=' null 2024-09-18T14:54:55.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:54:55.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:54:55.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:54:55.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:54:55.355 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:55.355 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:55.355 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:55.355 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:55.355 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:55.356 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:55.356 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:55.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:54:55.792 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:55.792 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:55.793 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:56.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:54:56.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:54:56.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:54:56.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:54:56.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:54:56.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:54:56.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:54:56.375 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:54:56.375 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:54:57.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28831642 2024-09-18T14:54:57.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28831642 '!=' null 2024-09-18T14:54:57.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:54:57.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:54:57.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:54:57.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:54:57.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:57.312 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:57.312 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:57.313 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:57.313 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:57.313 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:57.313 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:57.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:54:57.699 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:57.699 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:57.699 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:58.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:54:58.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:54:58.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:54:58.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:54:58.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:54:58.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:54:58.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:54:58.221 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:54:58.221 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:54:58.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29705282 2024-09-18T14:54:58.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29705282 '!=' null 2024-09-18T14:54:58.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:54:58.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:54:58.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:54:59.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:54:59.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:54:59.064 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:54:59.064 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:54:59.064 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:54:59.065 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:54:59.065 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:54:59.065 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:54:59.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:54:59.432 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:54:59.432 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:54:59.432 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:54:59.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:54:59.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:54:59.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:54:59.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:54:59.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:54:59.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:54:59.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:54:59.941 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:54:59.941 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:55:00.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29705282 2024-09-18T14:55:00.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29705282 '!=' null 2024-09-18T14:55:00.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:55:00.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:55:00.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:55:00.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:55:00.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:55:00.857 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:55:00.857 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:55:00.858 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:55:00.858 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:55:00.858 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:55:00.858 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:55:01.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:55:01.221 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:55:01.222 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:55:01.222 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:55:01.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:55:01.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:55:01.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:55:01.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:55:01.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:55:01.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:55:01.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:55:01.909 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:55:01.910 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:55:02.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28831615 2024-09-18T14:55:02.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28831615 '!=' null 2024-09-18T14:55:02.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:55:02.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:55:02.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:55:02.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:55:02.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:55:02.634 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:55:02.634 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:55:02.634 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:55:02.634 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:55:02.634 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:55:02.634 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:55:03.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:55:03.190 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:55:03.190 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:55:03.190 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:55:03.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:55:03.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:55:03.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:55:03.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:55:03.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:55:03.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:55:03.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:55:03.885 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:55:03.885 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:55:04.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28831618 2024-09-18T14:55:04.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28831618 '!=' null 2024-09-18T14:55:04.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:55:04.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:55:04.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:55:04.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:55:04.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:55:04.730 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:55:04.730 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:55:04.730 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:55:04.730 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:55:04.731 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:55:04.731 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:55:05.095 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:55:05.096 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:55:05.096 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:55:05.096 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:55:05.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:55:05.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:55:05.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:55:05.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:55:05.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:55:05.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:55:05.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:55:05.665 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:55:05.665 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:55:06.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23589591 2024-09-18T14:55:06.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23589591 '!=' null 2024-09-18T14:55:06.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:55:06.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:55:06.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:55:06.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:55:06.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:55:06.485 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:55:06.485 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:55:06.486 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:55:06.486 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:55:06.486 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:55:06.486 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:55:06.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:55:06.951 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:55:06.951 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:55:06.951 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:55:07.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:55:07.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:55:07.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:55:07.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:55:07.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:55:07.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:55:07.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:55:07.592 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:55:07.592 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:55:08.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29705184 2024-09-18T14:55:08.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29705184 '!=' null 2024-09-18T14:55:08.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:55:08.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:55:08.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:55:08.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:55:08.476 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:55:08.476 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:55:08.477 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:55:08.477 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:55:08.477 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:55:08.477 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:55:08.477 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:55:09.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:55:09.102 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:55:09.102 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:55:09.103 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:55:09.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:55:09.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:55:09.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:55:09.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:55:09.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:55:09.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:55:09.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:55:09.656 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:55:09.656 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:55:10.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28831370 2024-09-18T14:55:10.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28831370 '!=' null 2024-09-18T14:55:10.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:55:10.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:55:10.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:55:10.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:55:10.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:55:10.383 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:55:10.383 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:55:10.383 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:55:10.383 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:55:10.383 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:55:10.383 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:55:10.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:55:10.763 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:55:10.763 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:55:10.763 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:55:11.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:55:11.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:55:11.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:55:11.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:55:11.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:55:11.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:55:11.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:55:11.261 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:55:11.261 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:55:11.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28831352 2024-09-18T14:55:11.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28831352 '!=' null 2024-09-18T14:55:11.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:55:11.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:55:11.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:55:11.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:55:11.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:55:11.905 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:55:11.905 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:55:11.905 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:55:11.905 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:55:11.905 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:55:11.905 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:55:12.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:55:12.284 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:55:12.284 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:55:12.284 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:55:12.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:55:12.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:55:12.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:55:12.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:55:12.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:55:12.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:55:12.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:55:12.790 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:55:12.790 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:55:13.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29705021 2024-09-18T14:55:13.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29705021 '!=' null 2024-09-18T14:55:13.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:55:13.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:55:13.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:55:13.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:55:13.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:55:13.698 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:55:13.698 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:55:13.698 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:55:13.698 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:55:13.698 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:55:13.698 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:55:14.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:55:14.060 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:55:14.060 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:55:14.060 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:55:14.860 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:55:14.860 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:55:14.860 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:55:14.860 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:55:14.860 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:55:14.860 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:55:14.860 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:55:14.861 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:55:14.861 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:55:15.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23589053 2024-09-18T14:55:15.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23589053 '!=' null 2024-09-18T14:55:15.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:55:15.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:55:15.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:55:15.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:55:15.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:55:15.758 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:55:15.758 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:55:15.758 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:55:15.758 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:55:15.758 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:55:15.758 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:55:16.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:55:16.125 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:55:16.125 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:55:16.125 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:55:16.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:55:16.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:55:16.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:55:16.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:55:16.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:55:16.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:55:16.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:55:16.627 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:55:16.627 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:55:17.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28831099 2024-09-18T14:55:17.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28831099 '!=' null 2024-09-18T14:55:17.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:55:17.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:55:17.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:55:17.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:55:17.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:55:17.346 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:55:17.346 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:55:17.346 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:55:17.346 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:55:17.346 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:55:17.346 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:55:17.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:55:17.775 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:55:17.775 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:55:17.775 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:55:18.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:55:18.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:55:18.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:55:18.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:55:18.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:55:18.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:55:18.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:55:18.298 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:55:18.298 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:55:19.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27957441 2024-09-18T14:55:19.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27957441 '!=' null 2024-09-18T14:55:19.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:55:19.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:55:19.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:55:19.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:55:19.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:55:19.177 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:55:19.177 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:55:19.177 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:55:19.177 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:55:19.177 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:55:19.177 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:55:19.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:55:19.635 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:55:19.635 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:55:19.636 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:55:20.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:55:20.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:55:20.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:55:20.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:55:20.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:55:20.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:55:20.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:55:20.328 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:55:20.328 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:55:21.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28831151 2024-09-18T14:55:21.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28831151 '!=' null 2024-09-18T14:55:21.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:55:21.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:55:21.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:55:21.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:55:21.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:55:21.204 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:55:21.205 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:55:21.205 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:55:21.205 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:55:21.205 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:55:21.205 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:55:21.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:55:21.571 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:55:21.571 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:55:21.571 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:55:22.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:55:22.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:55:22.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:55:22.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:55:22.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:55:22.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:55:22.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:55:22.269 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:55:22.269 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:55:22.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28831151 2024-09-18T14:55:22.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28831151 '!=' null 2024-09-18T14:55:22.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:55:22.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:55:22.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:55:22.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:55:22.938 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:55:22.939 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:55:22.939 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:55:22.939 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:55:22.939 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:55:22.939 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:55:22.939 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:55:23.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:55:23.512 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:55:23.512 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:55:23.512 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:55:24.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:55:24.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:55:24.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:55:24.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:55:24.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:55:24.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:55:24.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:55:24.186 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:55:24.186 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:55:24.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28830963 2024-09-18T14:55:24.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28830963 '!=' null 2024-09-18T14:55:24.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:55:24.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:55:24.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:55:24.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:55:24.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:55:24.822 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:55:24.823 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:55:24.823 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:55:24.823 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:55:24.823 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:55:24.823 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:55:25.299 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:55:25.300 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:55:25.300 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:55:25.300 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:55:25.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:55:25.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:55:25.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:55:25.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:55:25.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:55:25.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:55:25.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:55:25.796 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:55:25.796 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:55:26.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22715356 2024-09-18T14:55:26.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22715356 '!=' null 2024-09-18T14:55:26.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:55:26.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:55:26.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:55:26.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:55:26.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:55:26.564 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:55:26.564 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:55:26.564 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:55:26.564 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:55:26.564 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:55:26.564 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:55:26.992 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:55:26.992 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:55:26.992 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:55:26.992 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:55:27.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:55:27.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:55:27.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:55:27.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:55:27.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:55:27.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:55:27.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:55:27.674 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:55:27.674 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:55:28.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27957654 2024-09-18T14:55:28.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27957654 '!=' null 2024-09-18T14:55:28.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:55:28.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:55:28.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:55:28.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:55:28.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:55:28.345 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:55:28.345 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:55:28.345 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:55:28.345 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:55:28.345 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:55:28.345 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:55:28.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:55:28.710 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:55:28.710 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:55:28.710 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:55:29.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:55:29.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:55:29.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:55:29.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:55:29.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:55:29.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:55:29.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:55:29.358 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:55:29.358 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:55:29.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27957550 2024-09-18T14:55:29.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27957550 '!=' null 2024-09-18T14:55:29.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:55:29.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:55:29.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:55:30.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:55:30.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:55:30.060 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:55:30.060 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:55:30.061 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:55:30.061 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:55:30.061 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:55:30.061 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:55:30.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:55:30.508 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:55:30.508 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:55:30.508 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:55:31.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:55:31.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:55:31.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:55:31.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:55:31.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:55:31.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:55:31.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:55:31.153 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:55:31.153 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:55:31.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28831063 2024-09-18T14:55:31.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28831063 '!=' null 2024-09-18T14:55:31.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:55:31.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:55:31.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:55:31.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:55:31.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:55:31.929 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:55:31.930 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:55:31.930 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:55:31.930 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:55:31.930 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:55:31.930 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:55:32.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:55:32.304 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:55:32.304 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:55:32.304 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:55:32.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:55:32.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:55:32.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:55:32.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:55:32.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:55:32.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:55:32.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:55:32.806 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:55:32.806 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:55:33.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28831210 2024-09-18T14:55:33.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28831210 '!=' null 2024-09-18T14:55:33.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:55:33.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:55:33.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:55:33.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:55:33.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:55:33.649 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:55:33.649 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:55:33.649 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:55:33.649 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:55:33.649 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:55:33.649 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:55:34.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:55:34.086 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:55:34.086 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:55:34.086 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:55:34.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:55:34.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:55:34.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:55:34.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:55:34.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:55:34.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:55:34.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:55:34.841 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:55:34.841 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:55:35.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26210226 2024-09-18T14:55:35.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26210226 '!=' null 2024-09-18T14:55:35.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:55:35.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:55:35.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:55:35.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:55:35.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:55:35.663 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:55:35.663 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:55:35.664 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:55:35.664 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:55:35.664 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:55:35.664 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:55:36.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:55:36.103 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:55:36.103 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:55:36.103 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:55:36.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:55:36.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:55:36.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:55:36.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:55:36.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:55:36.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:55:36.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:55:36.608 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:55:36.608 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:55:37.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27520919 2024-09-18T14:55:37.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27520919 '!=' null 2024-09-18T14:55:37.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:55:37.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:55:37.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:55:37.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:55:37.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:55:37.251 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:55:37.251 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:55:37.251 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:55:37.251 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:55:37.252 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:55:37.252 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:55:37.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T14:55:37.657 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:55:37.657 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:55:37.657 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:55:38.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-18T14:55:38.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T14:55:38.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:55:38.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:55:38.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:55:38.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:55:38.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:55:38.255 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:55:38.255 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:55:38.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20968463 2024-09-18T14:55:38.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20968463 '!=' null 2024-09-18T14:55:38.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:55:38.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:55:38.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:55:38.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:55:38.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:55:38.880 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:55:38.880 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:55:38.880 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:55:38.880 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:55:38.880 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:55:38.880 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:55:39.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T14:55:39.274 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:55:39.274 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:55:39.274 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:55:39.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-09-18T14:55:39.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 0 2024-09-18T14:55:39.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T14:55:39.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=1 2024-09-18T14:55:39.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:55:39.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:55:39.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:55:39.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:55:39.970 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:55:39.970 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:55:39.970 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:55:39.970 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:55:39.970 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:55:39.970 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:55:40.355 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T14:55:40.356 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:55:40.356 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:55:40.356 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:55:40.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-09-18T14:55:40.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T14:55:40.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T14:55:40.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T14:55:40.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T14:55:40.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T14:55:40.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T14:55:40.880 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T14:55:40.880 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T14:55:41.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=31151264 2024-09-18T14:55:41.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 31151264 '!=' null 2024-09-18T14:55:41.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T14:55:41.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T14:55:41.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T14:55:41.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T14:55:41.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:55:41.488 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:55:41.489 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:55:41.489 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:55:41.489 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:55:41.489 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:55:41.489 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:55:41.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T14:55:41.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:55:41.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:55:41.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:55:42.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-09-18T14:55:42.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:55:42.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:55:42.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:525: TEST_recovery_pool_priority: '[' 0 '!=' 0 ']' 2024-09-18T14:55:42.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:529: TEST_recovery_pool_priority: echo TEST PASSED 2024-09-18T14:55:42.345 INFO:tasks.workunit.client.0.smithi047.stdout:TEST PASSED 2024-09-18T14:55:42.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:532: TEST_recovery_pool_priority: delete_pool test1 2024-09-18T14:55:42.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test1 2024-09-18T14:55:42.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2024-09-18T14:55:42.896 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test1' does not exist 2024-09-18T14:55:42.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:533: TEST_recovery_pool_priority: delete_pool test2 2024-09-18T14:55:42.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test2 2024-09-18T14:55:42.907 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2024-09-18T14:55:43.501 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test2' does not exist 2024-09-18T14:55:43.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:534: TEST_recovery_pool_priority: kill_daemons td/osd-recovery-prio 2024-09-18T14:55:43.512 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:55:43.512 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:55:43.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:55:43.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:55:43.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:55:48.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:55:48.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:535: TEST_recovery_pool_priority: return 0 2024-09-18T14:55:48.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:41: run: teardown td/osd-recovery-prio 2024-09-18T14:55:48.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2024-09-18T14:55:48.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:55:48.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2024-09-18T14:55:48.827 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:55:48.828 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:55:48.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:55:48.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:55:48.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:55:48.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:55:48.833 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:55:48.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:55:48.834 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:55:48.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:55:48.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:55:48.836 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:55:48.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:55:48.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:55:48.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:55:48.837 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:55:48.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:55:48.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:55:48.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2024-09-18T14:55:48.879 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:55:48.879 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:55:48.879 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215238 2024-09-18T14:55:48.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.215238 2024-09-18T14:55:48.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:55:48.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:55:48.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:38: run: for func in $funcs 2024-09-18T14:55:48.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:39: run: setup td/osd-recovery-prio 2024-09-18T14:55:48.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-prio 2024-09-18T14:55:48.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-prio 2024-09-18T14:55:48.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2024-09-18T14:55:48.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T14:55:48.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2024-09-18T14:55:48.882 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T14:55:48.882 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T14:55:48.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T14:55:48.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T14:55:48.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T14:55:48.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T14:55:48.883 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T14:55:48.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T14:55:48.884 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T14:55:48.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T14:55:48.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T14:55:48.885 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T14:55:48.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:55:48.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T14:55:48.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T14:55:48.887 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T14:55:48.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:55:48.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T14:55:48.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2024-09-18T14:55:48.890 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T14:55:48.890 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:55:48.890 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215238 2024-09-18T14:55:48.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.215238 2024-09-18T14:55:48.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T14:55:48.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T14:55:48.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-prio 2024-09-18T14:55:48.893 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T14:55:48.893 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:55:48.893 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215238 2024-09-18T14:55:48.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.215238 2024-09-18T14:55:48.896 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T14:55:48.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T14:55:48.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T14:55:48.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-prio 1' TERM HUP INT 2024-09-18T14:55:48.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:40: run: TEST_recovery_priority td/osd-recovery-prio 2024-09-18T14:55:48.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:47: TEST_recovery_priority: local dir=td/osd-recovery-prio 2024-09-18T14:55:48.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:48: TEST_recovery_priority: local pools=10 2024-09-18T14:55:48.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:49: TEST_recovery_priority: local OSDS=5 2024-09-18T14:55:48.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:50: TEST_recovery_priority: local max_tries=10 2024-09-18T14:55:48.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:52: TEST_recovery_priority: run_mon td/osd-recovery-prio a 2024-09-18T14:55:48.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-prio 2024-09-18T14:55:48.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T14:55:48.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T14:55:48.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T14:55:48.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-prio/a 2024-09-18T14:55:48.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-prio/a --run-dir=td/osd-recovery-prio 2024-09-18T14:55:49.254 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T14:55:49.254 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:55:49.254 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:55:49.254 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:55:49.254 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:55:49.254 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215238 2024-09-18T14:55:49.254 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.215238/$cluster-$name.asok' 2024-09-18T14:55:49.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-prio/a '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.215238/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-prio/log --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T14:55:49.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T14:55:49.506 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T14:55:49.506 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:55:49.506 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:55:49.506 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T14:55:49.507 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T14:55:49.507 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:55:49.508 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:55:49.508 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:55:49.508 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:55:49.508 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:55:49.508 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215238 2024-09-18T14:55:49.508 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.215238/ceph-mon.a.asok 2024-09-18T14:55:49.508 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:55:49.508 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.215238/ceph-mon.a.asok config get fsid 2024-09-18T14:55:49.595 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T14:55:49.595 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T14:55:49.595 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T14:55:49.595 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T14:55:49.595 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T14:55:49.595 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T14:55:49.596 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T14:55:49.596 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T14:55:49.596 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T14:55:49.596 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:55:49.596 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215238 2024-09-18T14:55:49.596 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.215238/ceph-mon.a.asok 2024-09-18T14:55:49.596 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T14:55:49.596 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.215238/ceph-mon.a.asok config get mon_host 2024-09-18T14:55:49.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: run_mgr td/osd-recovery-prio x 2024-09-18T14:55:49.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-prio 2024-09-18T14:55:49.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T14:55:49.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T14:55:49.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T14:55:49.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-prio/x 2024-09-18T14:55:49.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T14:55:50.527 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T14:55:50.527 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:55:50.527 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:55:50.527 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:55:50.528 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:55:50.528 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215238 2024-09-18T14:55:50.528 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.215238/$cluster-$name.asok' 2024-09-18T14:55:50.528 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:55:50.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-prio/x '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.215238/$cluster-$name.asok' --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T14:55:50.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:54: TEST_recovery_priority: export CEPH_ARGS 2024-09-18T14:55:50.554 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: expr 5 - 1 2024-09-18T14:55:50.555 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: seq 0 4 2024-09-18T14:55:50.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:55:50.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 0 2024-09-18T14:55:50.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-09-18T14:55:50.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:55:50.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T14:55:50.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:55:50.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/0 2024-09-18T14:55:50.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5c1cb17f-d525-402f-8061-e33fb6e2528e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-09-18T14:55:50.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:55:50.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:55:50.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:55:50.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/0' 2024-09-18T14:55:50.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/0/journal' 2024-09-18T14:55:50.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:55:50.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:55:50.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-09-18T14:55:50.560 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:55:50.561 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:55:50.561 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:55:50.561 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:55:50.561 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:55:50.561 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215238 2024-09-18T14:55:50.561 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.215238/$cluster-$name.asok' 2024-09-18T14:55:50.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.215238/$cluster-$name.asok' 2024-09-18T14:55:50.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:55:50.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:55:50.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:55:50.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-09-18T14:55:50.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-09-18T14:55:50.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:55:50.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:55:50.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:55:50.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:55:50.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:55:50.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:55:50.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/0 2024-09-18T14:55:50.562 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:55:50.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b08dc7da-830d-460b-96c9-b626df884e65 2024-09-18T14:55:50.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b08dc7da-830d-460b-96c9-b626df884e65' 2024-09-18T14:55:50.563 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 b08dc7da-830d-460b-96c9-b626df884e65 2024-09-18T14:55:50.563 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:55:50.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD26epm8XBLIhAALWyR3hC4PW4A1kAtWn11TQ== 2024-09-18T14:55:50.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD26epm8XBLIhAALWyR3hC4PW4A1kAtWn11TQ=="}' 2024-09-18T14:55:50.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b08dc7da-830d-460b-96c9-b626df884e65 -i td/osd-recovery-prio/0/new.json 2024-09-18T14:55:50.895 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:55:50.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/0/new.json 2024-09-18T14:55:50.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=5c1cb17f-d525-402f-8061-e33fb6e2528e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.215238/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD26epm8XBLIhAALWyR3hC4PW4A1kAtWn11TQ== --osd-uuid b08dc7da-830d-460b-96c9-b626df884e65 2024-09-18T14:55:50.932 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:55:50.931+0000 7f3640a8d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:55:50.956 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:55:50.955+0000 7f3640a8d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:55:50.957 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:55:50.956+0000 7f3640a8d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:55:50.957 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:55:50.956+0000 7f3640a8d440 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-09-18T14:55:50.957 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:55:50.956+0000 7f3640a8d440 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-09-18T14:55:50.957 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:55:50.956+0000 7f3640a8d440 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-09-18T14:55:50.996 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:55:50.995+0000 7f3640a8d440 -1 bluestore(td/osd-recovery-prio/0) _read_fsid unparsable uuid 2024-09-18T14:55:56.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/0/keyring 2024-09-18T14:55:56.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:55:56.343 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T14:55:56.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T14:55:56.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:55:56.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T14:55:56.857 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T14:55:56.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=5c1cb17f-d525-402f-8061-e33fb6e2528e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.215238/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:55:56.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:55:56.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:55:56.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:55:56.882 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:55:56.881+0000 7fd5d0e08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:55:56.884 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:55:56.883+0000 7fd5d0e08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:55:56.885 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:55:56.884+0000 7fd5d0e08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:55:57.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T14:55:57.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:55:57.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T14:55:57.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:55:57.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:55:57.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:55:57.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:55:57.325 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:55:57.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:55:57.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:55:57.421 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:55:57.420+0000 7fd5d0e08440 -1 Falling back to public interface 2024-09-18T14:55:57.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:55:58.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:55:58.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:55:58.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:55:58.726 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:55:58.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:55:58.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:55:59.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:55:59.965 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:55:59.964+0000 7fd5d0e08440 -1 osd.0 0 log_to_monitors true 2024-09-18T14:56:00.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:00.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:00.110 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:56:00.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:56:00.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:00.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:56:00.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:01.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:01.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:01.584 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:56:01.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:56:01.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:01.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T14:56:01.997 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/235236,v1:127.0.0.1:6803/235236] [v2:127.0.0.1:6804/235236,v1:127.0.0.1:6805/235236] exists,up b08dc7da-830d-460b-96c9-b626df884e65 2024-09-18T14:56:02.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:56:02.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:56:02.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:56:02.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:56:02.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 1 2024-09-18T14:56:02.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-09-18T14:56:02.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:56:02.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T14:56:02.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:56:02.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/1 2024-09-18T14:56:02.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5c1cb17f-d525-402f-8061-e33fb6e2528e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-09-18T14:56:02.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:56:02.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:56:02.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:56:02.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/1' 2024-09-18T14:56:02.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/1/journal' 2024-09-18T14:56:02.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:56:02.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:56:02.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-09-18T14:56:02.002 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:56:02.002 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:56:02.002 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:56:02.002 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:56:02.002 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:56:02.002 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215238 2024-09-18T14:56:02.002 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.215238/$cluster-$name.asok' 2024-09-18T14:56:02.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.215238/$cluster-$name.asok' 2024-09-18T14:56:02.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:56:02.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:56:02.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:56:02.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-09-18T14:56:02.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-09-18T14:56:02.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:56:02.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:56:02.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:56:02.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:56:02.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:56:02.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:56:02.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/1 2024-09-18T14:56:02.003 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:56:02.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=45dbe055-29f2-4ece-b2f0-2ab35aae2799 2024-09-18T14:56:02.004 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 45dbe055-29f2-4ece-b2f0-2ab35aae2799 2024-09-18T14:56:02.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 45dbe055-29f2-4ece-b2f0-2ab35aae2799' 2024-09-18T14:56:02.004 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:56:02.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAC6upmezP2ABAANzItQDsezpBYqtN8IKMDnQ== 2024-09-18T14:56:02.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAC6upmezP2ABAANzItQDsezpBYqtN8IKMDnQ=="}' 2024-09-18T14:56:02.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 45dbe055-29f2-4ece-b2f0-2ab35aae2799 -i td/osd-recovery-prio/1/new.json 2024-09-18T14:56:02.519 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:56:02.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/1/new.json 2024-09-18T14:56:02.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=5c1cb17f-d525-402f-8061-e33fb6e2528e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.215238/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAC6upmezP2ABAANzItQDsezpBYqtN8IKMDnQ== --osd-uuid 45dbe055-29f2-4ece-b2f0-2ab35aae2799 2024-09-18T14:56:02.558 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:02.557+0000 7f2acdc0e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:56:02.560 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:02.559+0000 7f2acdc0e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:56:02.561 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:02.560+0000 7f2acdc0e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:56:02.562 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:02.560+0000 7f2acdc0e440 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-09-18T14:56:02.562 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:02.560+0000 7f2acdc0e440 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-09-18T14:56:02.562 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:02.560+0000 7f2acdc0e440 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-09-18T14:56:02.610 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:02.608+0000 7f2acdc0e440 -1 bluestore(td/osd-recovery-prio/1) _read_fsid unparsable uuid 2024-09-18T14:56:08.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/1/keyring 2024-09-18T14:56:08.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:56:08.257 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T14:56:08.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T14:56:08.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:56:08.781 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T14:56:08.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T14:56:08.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=5c1cb17f-d525-402f-8061-e33fb6e2528e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.215238/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:56:08.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:56:08.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:56:08.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:56:08.806 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:08.805+0000 7fbf8140d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:56:08.832 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:08.831+0000 7fbf8140d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:56:08.833 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:08.832+0000 7fbf8140d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:56:09.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T14:56:09.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:56:09.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T14:56:09.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:56:09.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:56:09.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:09.192 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:56:09.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:56:09.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:09.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:56:09.365 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:09.364+0000 7fbf8140d440 -1 Falling back to public interface 2024-09-18T14:56:09.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:10.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:10.563 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:56:10.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:10.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:56:10.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:10.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:56:10.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:11.870 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:11.869+0000 7fbf8140d440 -1 osd.1 0 log_to_monitors true 2024-09-18T14:56:11.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:11.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:11.942 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:56:11.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:56:11.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:11.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:56:12.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:13.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:13.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:13.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:56:13.497 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:56:13.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:13.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T14:56:13.876 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/236635,v1:127.0.0.1:6811/236635] [v2:127.0.0.1:6812/236635,v1:127.0.0.1:6813/236635] exists,up 45dbe055-29f2-4ece-b2f0-2ab35aae2799 2024-09-18T14:56:13.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:56:13.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:56:13.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:56:13.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:56:13.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 2 2024-09-18T14:56:13.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-09-18T14:56:13.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:56:13.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T14:56:13.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:56:13.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/2 2024-09-18T14:56:13.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5c1cb17f-d525-402f-8061-e33fb6e2528e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-09-18T14:56:13.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:56:13.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:56:13.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:56:13.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/2' 2024-09-18T14:56:13.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/2/journal' 2024-09-18T14:56:13.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:56:13.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:56:13.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-09-18T14:56:13.882 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:56:13.882 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:56:13.882 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:56:13.882 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:56:13.882 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:56:13.882 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215238 2024-09-18T14:56:13.882 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.215238/$cluster-$name.asok' 2024-09-18T14:56:13.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.215238/$cluster-$name.asok' 2024-09-18T14:56:13.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:56:13.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:56:13.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:56:13.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-09-18T14:56:13.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-09-18T14:56:13.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:56:13.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:56:13.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:56:13.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:56:13.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:56:13.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:56:13.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/2 2024-09-18T14:56:13.883 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:56:13.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7b2224b4-74ef-4aa0-ae89-f72e80e00a39 2024-09-18T14:56:13.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 7b2224b4-74ef-4aa0-ae89-f72e80e00a39' 2024-09-18T14:56:13.883 INFO:tasks.workunit.client.0.smithi047.stdout:add osd2 7b2224b4-74ef-4aa0-ae89-f72e80e00a39 2024-09-18T14:56:13.884 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:56:13.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAN6upm3PNiNRAAajv6BAQgbECCvAUOFejj5Q== 2024-09-18T14:56:13.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAN6upm3PNiNRAAajv6BAQgbECCvAUOFejj5Q=="}' 2024-09-18T14:56:13.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7b2224b4-74ef-4aa0-ae89-f72e80e00a39 -i td/osd-recovery-prio/2/new.json 2024-09-18T14:56:14.416 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:56:14.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/2/new.json 2024-09-18T14:56:14.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=5c1cb17f-d525-402f-8061-e33fb6e2528e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/2 --osd-journal=td/osd-recovery-prio/2/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.215238/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAN6upm3PNiNRAAajv6BAQgbECCvAUOFejj5Q== --osd-uuid 7b2224b4-74ef-4aa0-ae89-f72e80e00a39 2024-09-18T14:56:14.454 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:14.453+0000 7f3b5be16440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:56:14.474 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:14.473+0000 7f3b5be16440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:56:14.475 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:14.474+0000 7f3b5be16440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:56:14.476 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:14.474+0000 7f3b5be16440 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2024-09-18T14:56:14.476 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:14.474+0000 7f3b5be16440 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2024-09-18T14:56:14.476 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:14.475+0000 7f3b5be16440 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2024-09-18T14:56:14.566 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:14.565+0000 7f3b5be16440 -1 bluestore(td/osd-recovery-prio/2) _read_fsid unparsable uuid 2024-09-18T14:56:20.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/2/keyring 2024-09-18T14:56:20.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:56:20.452 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd2 key to auth repository 2024-09-18T14:56:20.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T14:56:20.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:56:20.970 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T14:56:20.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T14:56:20.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=5c1cb17f-d525-402f-8061-e33fb6e2528e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/2 --osd-journal=td/osd-recovery-prio/2/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.215238/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:56:20.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:56:20.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:56:20.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:56:20.995 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:20.994+0000 7f5662216440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:56:21.014 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:21.013+0000 7f5662216440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:56:21.015 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:21.014+0000 7f5662216440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:56:21.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T14:56:21.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:56:21.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T14:56:21.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:56:21.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:56:21.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:21.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:56:21.412 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:56:21.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:21.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:56:21.550 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:21.549+0000 7f5662216440 -1 Falling back to public interface 2024-09-18T14:56:21.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:22.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:22.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:22.785 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:56:22.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:56:22.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:22.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:56:23.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:24.045 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:24.044+0000 7f5662216440 -1 osd.2 0 log_to_monitors true 2024-09-18T14:56:24.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:24.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:24.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:56:24.215 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:56:24.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:24.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:56:24.761 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:25.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:25.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:25.763 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:56:25.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:56:25.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:25.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T14:56:26.138 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/238034,v1:127.0.0.1:6819/238034] [v2:127.0.0.1:6820/238034,v1:127.0.0.1:6821/238034] exists,up 7b2224b4-74ef-4aa0-ae89-f72e80e00a39 2024-09-18T14:56:26.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:56:26.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:56:26.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:56:26.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:56:26.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 3 2024-09-18T14:56:26.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-09-18T14:56:26.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:56:26.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-18T14:56:26.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:56:26.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/3 2024-09-18T14:56:26.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5c1cb17f-d525-402f-8061-e33fb6e2528e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-09-18T14:56:26.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:56:26.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:56:26.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:56:26.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/3' 2024-09-18T14:56:26.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/3/journal' 2024-09-18T14:56:26.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:56:26.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:56:26.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-09-18T14:56:26.144 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:56:26.144 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:56:26.144 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:56:26.144 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:56:26.145 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:56:26.145 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215238 2024-09-18T14:56:26.145 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.215238/$cluster-$name.asok' 2024-09-18T14:56:26.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.215238/$cluster-$name.asok' 2024-09-18T14:56:26.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:56:26.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:56:26.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:56:26.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-09-18T14:56:26.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-09-18T14:56:26.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:56:26.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:56:26.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:56:26.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:56:26.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:56:26.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:56:26.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/3 2024-09-18T14:56:26.146 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:56:26.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f8d137a0-a3a2-419c-b84d-24e390b7d2fd 2024-09-18T14:56:26.146 INFO:tasks.workunit.client.0.smithi047.stdout:add osd3 f8d137a0-a3a2-419c-b84d-24e390b7d2fd 2024-09-18T14:56:26.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 f8d137a0-a3a2-419c-b84d-24e390b7d2fd' 2024-09-18T14:56:26.147 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:56:26.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAa6upmrcRnCRAASQEgs+eITz0sapBKpREFCw== 2024-09-18T14:56:26.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAa6upmrcRnCRAASQEgs+eITz0sapBKpREFCw=="}' 2024-09-18T14:56:26.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f8d137a0-a3a2-419c-b84d-24e390b7d2fd -i td/osd-recovery-prio/3/new.json 2024-09-18T14:56:26.658 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:56:26.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/3/new.json 2024-09-18T14:56:26.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=5c1cb17f-d525-402f-8061-e33fb6e2528e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/3 --osd-journal=td/osd-recovery-prio/3/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.215238/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAa6upmrcRnCRAASQEgs+eITz0sapBKpREFCw== --osd-uuid f8d137a0-a3a2-419c-b84d-24e390b7d2fd 2024-09-18T14:56:26.697 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:26.696+0000 7f424d809440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:56:26.699 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:26.698+0000 7f424d809440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:56:26.700 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:26.699+0000 7f424d809440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:56:26.700 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:26.699+0000 7f424d809440 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2024-09-18T14:56:26.700 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:26.699+0000 7f424d809440 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2024-09-18T14:56:26.700 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:26.699+0000 7f424d809440 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2024-09-18T14:56:26.748 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:26.747+0000 7f424d809440 -1 bluestore(td/osd-recovery-prio/3) _read_fsid unparsable uuid 2024-09-18T14:56:32.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/3/keyring 2024-09-18T14:56:32.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:56:32.895 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd3 key to auth repository 2024-09-18T14:56:32.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-18T14:56:32.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:56:33.408 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.3 2024-09-18T14:56:33.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-18T14:56:33.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=5c1cb17f-d525-402f-8061-e33fb6e2528e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/3 --osd-journal=td/osd-recovery-prio/3/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.215238/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:56:33.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:56:33.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:56:33.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:56:33.433 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:33.432+0000 7f4302529440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:56:33.435 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:33.434+0000 7f4302529440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:56:33.436 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:33.435+0000 7f4302529440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:56:33.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-18T14:56:33.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:56:33.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T14:56:33.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:56:33.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:56:33.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:33.777 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:56:33.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:56:33.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:33.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:56:33.958 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:33.957+0000 7f4302529440 -1 Falling back to public interface 2024-09-18T14:56:34.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:35.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:35.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:35.141 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:56:35.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:56:35.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:35.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:56:35.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:36.481 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:36.479+0000 7f4302529440 -1 osd.3 0 log_to_monitors true 2024-09-18T14:56:36.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:36.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:36.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:56:36.559 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:56:36.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:36.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:56:37.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:38.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:38.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:38.102 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:56:38.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:56:38.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:38.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T14:56:38.503 INFO:tasks.workunit.client.0.smithi047.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/239433,v1:127.0.0.1:6827/239433] [v2:127.0.0.1:6828/239433,v1:127.0.0.1:6829/239433] exists,up f8d137a0-a3a2-419c-b84d-24e390b7d2fd 2024-09-18T14:56:38.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:56:38.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:56:38.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:56:38.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T14:56:38.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 4 2024-09-18T14:56:38.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-09-18T14:56:38.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T14:56:38.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-18T14:56:38.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T14:56:38.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/4 2024-09-18T14:56:38.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5c1cb17f-d525-402f-8061-e33fb6e2528e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-09-18T14:56:38.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T14:56:38.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T14:56:38.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T14:56:38.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/4' 2024-09-18T14:56:38.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/4/journal' 2024-09-18T14:56:38.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T14:56:38.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T14:56:38.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-09-18T14:56:38.508 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T14:56:38.508 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T14:56:38.508 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T14:56:38.508 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T14:56:38.508 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T14:56:38.509 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215238 2024-09-18T14:56:38.509 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.215238/$cluster-$name.asok' 2024-09-18T14:56:38.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.215238/$cluster-$name.asok' 2024-09-18T14:56:38.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T14:56:38.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T14:56:38.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T14:56:38.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-09-18T14:56:38.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-09-18T14:56:38.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T14:56:38.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T14:56:38.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T14:56:38.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T14:56:38.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T14:56:38.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T14:56:38.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/4 2024-09-18T14:56:38.510 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T14:56:38.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=78f92760-2ba7-412d-9fa2-7f8cd1374d33 2024-09-18T14:56:38.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 78f92760-2ba7-412d-9fa2-7f8cd1374d33' 2024-09-18T14:56:38.510 INFO:tasks.workunit.client.0.smithi047.stdout:add osd4 78f92760-2ba7-412d-9fa2-7f8cd1374d33 2024-09-18T14:56:38.510 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T14:56:38.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAm6upmMqccHxAAL8CkjwZKYL17Z2IUUKv14g== 2024-09-18T14:56:38.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAm6upmMqccHxAAL8CkjwZKYL17Z2IUUKv14g=="}' 2024-09-18T14:56:38.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 78f92760-2ba7-412d-9fa2-7f8cd1374d33 -i td/osd-recovery-prio/4/new.json 2024-09-18T14:56:39.032 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T14:56:39.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/4/new.json 2024-09-18T14:56:39.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=5c1cb17f-d525-402f-8061-e33fb6e2528e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/4 --osd-journal=td/osd-recovery-prio/4/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.215238/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAm6upmMqccHxAAL8CkjwZKYL17Z2IUUKv14g== --osd-uuid 78f92760-2ba7-412d-9fa2-7f8cd1374d33 2024-09-18T14:56:39.072 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:39.070+0000 7f3093e13440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:56:39.104 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:39.103+0000 7f3093e13440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:56:39.105 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:39.104+0000 7f3093e13440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:56:39.105 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:39.104+0000 7f3093e13440 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2024-09-18T14:56:39.105 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:39.104+0000 7f3093e13440 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2024-09-18T14:56:39.105 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:39.104+0000 7f3093e13440 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2024-09-18T14:56:39.152 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:39.151+0000 7f3093e13440 -1 bluestore(td/osd-recovery-prio/4) _read_fsid unparsable uuid 2024-09-18T14:56:45.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/4/keyring 2024-09-18T14:56:45.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T14:56:45.009 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd4 key to auth repository 2024-09-18T14:56:45.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-18T14:56:45.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T14:56:45.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-18T14:56:45.518 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.4 2024-09-18T14:56:45.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=5c1cb17f-d525-402f-8061-e33fb6e2528e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/4 --osd-journal=td/osd-recovery-prio/4/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.215238/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T14:56:45.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T14:56:45.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T14:56:45.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T14:56:45.544 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:45.543+0000 7fbeb5010440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:56:45.546 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:45.545+0000 7fbeb5010440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:56:45.547 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:45.546+0000 7fbeb5010440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T14:56:45.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-18T14:56:45.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T14:56:45.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-18T14:56:45.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T14:56:45.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T14:56:45.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:45.933 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T14:56:45.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T14:56:45.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:45.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T14:56:46.064 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:46.063+0000 7fbeb5010440 -1 Falling back to public interface 2024-09-18T14:56:46.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:47.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:47.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:47.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T14:56:47.333 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T14:56:47.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:47.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T14:56:47.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:48.616 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T14:56:48.615+0000 7fbeb5010440 -1 osd.4 0 log_to_monitors true 2024-09-18T14:56:48.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:48.753 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T14:56:48.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:48.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T14:56:48.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:48.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T14:56:49.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T14:56:50.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T14:56:50.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T14:56:50.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T14:56:50.262 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T14:56:50.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T14:56:50.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T14:56:50.638 INFO:tasks.workunit.client.0.smithi047.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/240832,v1:127.0.0.1:6835/240832] [v2:127.0.0.1:6836/240832,v1:127.0.0.1:6837/240832] exists,up 78f92760-2ba7-412d-9fa2-7f8cd1374d33 2024-09-18T14:56:50.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T14:56:50.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T14:56:50.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T14:56:50.639 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: seq 1 10 2024-09-18T14:56:50.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-18T14:56:50.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test1 1 1 2024-09-18T14:56:50.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2024-09-18T14:56:51.198 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test1' already exists 2024-09-18T14:56:51.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T14:56:52.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test1 size 2 2024-09-18T14:56:53.283 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 1 size to 2 2024-09-18T14:56:53.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-18T14:56:53.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test2 1 1 2024-09-18T14:56:53.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test2 1 1 2024-09-18T14:56:54.015 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test2' already exists 2024-09-18T14:56:54.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T14:56:55.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test2 size 2 2024-09-18T14:56:55.662 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 2 size to 2 2024-09-18T14:56:55.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-18T14:56:55.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test3 1 1 2024-09-18T14:56:55.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test3 1 1 2024-09-18T14:56:56.553 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test3' already exists 2024-09-18T14:56:56.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T14:56:57.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test3 size 2 2024-09-18T14:56:58.663 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 3 size to 2 2024-09-18T14:56:58.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-18T14:56:58.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test4 1 1 2024-09-18T14:56:58.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test4 1 1 2024-09-18T14:56:59.315 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test4' already exists 2024-09-18T14:56:59.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T14:57:00.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test4 size 2 2024-09-18T14:57:01.411 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 4 size to 2 2024-09-18T14:57:01.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-18T14:57:01.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test5 1 1 2024-09-18T14:57:01.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test5 1 1 2024-09-18T14:57:02.359 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test5' already exists 2024-09-18T14:57:02.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T14:57:03.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test5 size 2 2024-09-18T14:57:04.225 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 5 size to 2 2024-09-18T14:57:04.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-18T14:57:04.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test6 1 1 2024-09-18T14:57:04.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test6 1 1 2024-09-18T14:57:05.006 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test6' already exists 2024-09-18T14:57:05.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T14:57:06.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test6 size 2 2024-09-18T14:57:07.008 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 6 size to 2 2024-09-18T14:57:07.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-18T14:57:07.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test7 1 1 2024-09-18T14:57:07.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test7 1 1 2024-09-18T14:57:07.638 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test7' already exists 2024-09-18T14:57:07.648 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T14:57:08.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test7 size 2 2024-09-18T14:57:09.757 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 7 size to 2 2024-09-18T14:57:09.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-18T14:57:09.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test8 1 1 2024-09-18T14:57:09.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test8 1 1 2024-09-18T14:57:10.493 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test8' already exists 2024-09-18T14:57:10.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T14:57:11.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test8 size 2 2024-09-18T14:57:12.741 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 8 size to 2 2024-09-18T14:57:12.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-18T14:57:12.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test9 1 1 2024-09-18T14:57:12.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test9 1 1 2024-09-18T14:57:13.764 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test9' already exists 2024-09-18T14:57:13.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T14:57:14.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test9 size 2 2024-09-18T14:57:15.560 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 9 size to 2 2024-09-18T14:57:15.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-18T14:57:15.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test10 1 1 2024-09-18T14:57:15.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test10 1 1 2024-09-18T14:57:16.400 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test10' already exists 2024-09-18T14:57:16.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T14:57:17.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test10 size 2 2024-09-18T14:57:18.213 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 10 size to 2 2024-09-18T14:57:18.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:66: TEST_recovery_priority: sleep 5 2024-09-18T14:57:23.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:68: TEST_recovery_priority: wait_for_clean 2024-09-18T14:57:23.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:57:23.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:57:23.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:57:23.227 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:57:23.227 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:57:23.227 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:57:23.227 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:57:23.228 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:57:23.228 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:57:23.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T14:57:23.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:57:23.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:57:23.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:57:23.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:57:23.330 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:57:23.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:57:23.697 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T14:57:23.697 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T14:57:23.697 INFO:tasks.workunit.client.0.smithi047.stderr:3 2024-09-18T14:57:23.697 INFO:tasks.workunit.client.0.smithi047.stderr:4' 2024-09-18T14:57:23.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:57:23.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:57:23.698 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:57:23.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836500 2024-09-18T14:57:23.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836500 2024-09-18T14:57:23.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500' 2024-09-18T14:57:23.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:57:23.808 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:57:23.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672977 2024-09-18T14:57:23.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672977 2024-09-18T14:57:23.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500 1-42949672977' 2024-09-18T14:57:23.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:57:23.919 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:57:24.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509455 2024-09-18T14:57:24.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509455 2024-09-18T14:57:24.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500 1-42949672977 2-64424509455' 2024-09-18T14:57:24.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:57:24.030 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T14:57:24.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345932 2024-09-18T14:57:24.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345932 2024-09-18T14:57:24.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500 1-42949672977 2-64424509455 3-85899345932' 2024-09-18T14:57:24.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:57:24.140 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T14:57:24.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182409 2024-09-18T14:57:24.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182409 2024-09-18T14:57:24.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500 1-42949672977 2-64424509455 3-85899345932 4-107374182409' 2024-09-18T14:57:24.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:57:24.252 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836500 2024-09-18T14:57:24.252 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:57:24.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:57:24.253 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836500 2024-09-18T14:57:24.253 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:57:24.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836500 2024-09-18T14:57:24.254 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836500 2024-09-18T14:57:24.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-09-18T14:57:24.255 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:57:24.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836500 -lt 21474836500 2024-09-18T14:57:24.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:57:24.623 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672977 2024-09-18T14:57:24.623 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:57:24.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:57:24.624 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672977 2024-09-18T14:57:24.624 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:57:24.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672977 2024-09-18T14:57:24.625 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672977 2024-09-18T14:57:24.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-09-18T14:57:24.626 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:57:25.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672977 -lt 42949672977 2024-09-18T14:57:25.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:57:25.027 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509455 2024-09-18T14:57:25.027 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:57:25.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:57:25.028 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509455 2024-09-18T14:57:25.029 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:57:25.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509455 2024-09-18T14:57:25.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-09-18T14:57:25.030 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509455 2024-09-18T14:57:25.030 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:57:25.477 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509455 -lt 64424509455 2024-09-18T14:57:25.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:57:25.478 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345932 2024-09-18T14:57:25.478 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:57:25.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T14:57:25.479 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345932 2024-09-18T14:57:25.479 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:57:25.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345932 2024-09-18T14:57:25.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345932' 2024-09-18T14:57:25.481 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 85899345932 2024-09-18T14:57:25.481 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T14:57:25.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345932 -lt 85899345932 2024-09-18T14:57:25.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:57:25.883 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182409 2024-09-18T14:57:25.883 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:57:25.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T14:57:25.884 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182409 2024-09-18T14:57:25.884 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:57:25.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182409 2024-09-18T14:57:25.886 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.4 seq 107374182409 2024-09-18T14:57:25.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182409' 2024-09-18T14:57:25.886 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T14:57:26.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182407 -lt 107374182409 2024-09-18T14:57:26.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:57:27.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:57:27.259 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T14:57:27.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182410 -lt 107374182409 2024-09-18T14:57:27.682 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:57:27.682 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:57:27.682 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:57:28.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 10 == 0 2024-09-18T14:57:28.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:57:28.179 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:57:28.179 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:57:28.180 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:57:28.180 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:57:28.180 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:57:28.180 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:57:28.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=10 2024-09-18T14:57:28.541 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:57:28.542 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:57:28.542 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:57:29.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 10 = 10 2024-09-18T14:57:29.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:57:29.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:57:29.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:70: TEST_recovery_priority: ceph pg dump pgs 2024-09-18T14:57:29.428 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T14:57:29.428 INFO:tasks.workunit.client.0.smithi047.stdout:10.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:57:18.405518+0000 0'0 81:20 [2,1] 2 [2,1] 2 0'0 2024-09-18T14:57:16.023325+0000 0'0 2024-09-18T14:57:16.023325+0000 0 0 periodic scrub scheduled @ 2024-09-19T19:03:14.557845+0000 0 0 2024-09-18T14:57:29.428 INFO:tasks.workunit.client.0.smithi047.stdout:9.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:57:16.292935+0000 0'0 80:25 [1,2] 1 [1,2] 1 0'0 2024-09-18T14:57:13.141601+0000 0'0 2024-09-18T14:57:13.141601+0000 0 0 periodic scrub scheduled @ 2024-09-19T23:47:16.930607+0000 0 0 2024-09-18T14:57:29.428 INFO:tasks.workunit.client.0.smithi047.stdout:8.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:57:13.510660+0000 0'0 78:27 [3,4] 3 [3,4] 3 0'0 2024-09-18T14:57:10.241372+0000 0'0 2024-09-18T14:57:10.241372+0000 0 0 periodic scrub scheduled @ 2024-09-20T02:38:02.193485+0000 0 0 2024-09-18T14:57:29.428 INFO:tasks.workunit.client.0.smithi047.stdout:7.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:57:10.803818+0000 0'0 80:33 [1,4] 1 [1,4] 1 0'0 2024-09-18T14:57:07.342435+0000 0'0 2024-09-18T14:57:07.342435+0000 0 0 periodic scrub scheduled @ 2024-09-19T18:41:09.171383+0000 0 0 2024-09-18T14:57:29.429 INFO:tasks.workunit.client.0.smithi047.stdout:6.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:57:07.267454+0000 0'0 79:35 [4,2] 4 [4,2] 4 0'0 2024-09-18T14:57:04.572790+0000 0'0 2024-09-18T14:57:04.572790+0000 0 0 periodic scrub scheduled @ 2024-09-19T19:11:05.779487+0000 0 0 2024-09-18T14:57:29.429 INFO:tasks.workunit.client.0.smithi047.stdout:5.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:57:04.895122+0000 0'0 78:32 [3,4] 3 [3,4] 3 0'0 2024-09-18T14:57:01.870823+0000 0'0 2024-09-18T14:57:01.870823+0000 0 0 periodic scrub scheduled @ 2024-09-19T22:03:13.118295+0000 0 0 2024-09-18T14:57:29.429 INFO:tasks.workunit.client.0.smithi047.stdout:4.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:57:02.022750+0000 0'0 79:42 [4,0] 4 [4,0] 4 0'0 2024-09-18T14:56:59.055555+0000 0'0 2024-09-18T14:56:59.055555+0000 0 0 periodic scrub scheduled @ 2024-09-19T15:11:45.593052+0000 0 0 2024-09-18T14:57:29.429 INFO:tasks.workunit.client.0.smithi047.stdout:3.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:56:59.366354+0000 0'0 80:48 [1,2] 1 [1,2] 1 0'0 2024-09-18T14:56:56.247413+0000 0'0 2024-09-18T14:56:56.247413+0000 0 0 periodic scrub scheduled @ 2024-09-19T19:46:22.791144+0000 0 0 2024-09-18T14:57:29.429 INFO:tasks.workunit.client.0.smithi047.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:56:56.893801+0000 0'0 78:46 [3,1] 3 [3,1] 3 0'0 2024-09-18T14:56:53.708618+0000 0'0 2024-09-18T14:56:53.708618+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:11:21.702343+0000 0 0 2024-09-18T14:57:29.429 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T14:56:54.246811+0000 0'0 80:55 [1,0] 1 [1,0] 1 0'0 2024-09-18T14:56:50.949946+0000 0'0 2024-09-18T14:56:50.949946+0000 0 0 periodic scrub scheduled @ 2024-09-19T18:23:03.164056+0000 0 0 2024-09-18T14:57:29.429 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T14:57:29.429 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T14:57:29.429 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T14:57:29.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:74: TEST_recovery_priority: local PG1 2024-09-18T14:57:29.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:75: TEST_recovery_priority: local POOLNUM1 2024-09-18T14:57:29.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:76: TEST_recovery_priority: local pool1 2024-09-18T14:57:29.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:77: TEST_recovery_priority: local chk_osd1_1 2024-09-18T14:57:29.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:78: TEST_recovery_priority: local chk_osd1_2 2024-09-18T14:57:29.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:80: TEST_recovery_priority: local PG2 2024-09-18T14:57:29.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:81: TEST_recovery_priority: local POOLNUM2 2024-09-18T14:57:29.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:82: TEST_recovery_priority: local pool2 2024-09-18T14:57:29.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:83: TEST_recovery_priority: local chk_osd2 2024-09-18T14:57:29.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:85: TEST_recovery_priority: local PG3 2024-09-18T14:57:29.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: local POOLNUM3 2024-09-18T14:57:29.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:87: TEST_recovery_priority: local pool3 2024-09-18T14:57:29.440 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: seq 1 10 2024-09-18T14:57:29.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-18T14:57:29.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 1.0 --format=json 2024-09-18T14:57:29.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-09-18T14:57:29.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-09-18T14:57:29.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=1 2024-09-18T14:57:29.849 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-09-18T14:57:29.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=0 2024-09-18T14:57:29.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z '' ']' 2024-09-18T14:57:29.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:96: TEST_recovery_priority: PG1=1.0 2024-09-18T14:57:29.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:97: TEST_recovery_priority: POOLNUM1=1 2024-09-18T14:57:29.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: pool1=test1 2024-09-18T14:57:29.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:99: TEST_recovery_priority: chk_osd1_1=1 2024-09-18T14:57:29.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:100: TEST_recovery_priority: chk_osd1_2=0 2024-09-18T14:57:29.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-18T14:57:29.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 2.0 --format=json 2024-09-18T14:57:29.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-09-18T14:57:30.219 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-09-18T14:57:30.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=3 2024-09-18T14:57:30.220 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-09-18T14:57:30.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=1 2024-09-18T14:57:30.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-09-18T14:57:30.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z '' -a 1 = 3 -a 0 '!=' 1 ']' 2024-09-18T14:57:30.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n '' -a 1 = 3 -a 0 '!=' 1 -a '' '!=' 1 ']' 2024-09-18T14:57:30.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-18T14:57:30.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 3.0 --format=json 2024-09-18T14:57:30.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-09-18T14:57:30.587 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-09-18T14:57:30.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=1 2024-09-18T14:57:30.588 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-09-18T14:57:30.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=2 2024-09-18T14:57:30.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-09-18T14:57:30.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z '' -a 1 = 1 -a 0 '!=' 2 ']' 2024-09-18T14:57:30.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:103: TEST_recovery_priority: PG2=3.0 2024-09-18T14:57:30.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:104: TEST_recovery_priority: POOLNUM2=3 2024-09-18T14:57:30.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:105: TEST_recovery_priority: pool2=test3 2024-09-18T14:57:30.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:106: TEST_recovery_priority: chk_osd2=2 2024-09-18T14:57:30.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-18T14:57:30.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 4.0 --format=json 2024-09-18T14:57:30.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-09-18T14:57:30.957 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-09-18T14:57:30.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=4 2024-09-18T14:57:30.958 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-09-18T14:57:30.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=0 2024-09-18T14:57:30.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-09-18T14:57:30.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 0 ']' 2024-09-18T14:57:30.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 0 -a 2 '!=' 0 ']' 2024-09-18T14:57:30.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-18T14:57:30.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 5.0 --format=json 2024-09-18T14:57:30.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-09-18T14:57:31.322 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-09-18T14:57:31.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=3 2024-09-18T14:57:31.323 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-09-18T14:57:31.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=4 2024-09-18T14:57:31.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-09-18T14:57:31.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z 3.0 -a 1 = 3 -a 0 '!=' 4 ']' 2024-09-18T14:57:31.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n 3.0 -a 1 = 3 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2024-09-18T14:57:31.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-18T14:57:31.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 6.0 --format=json 2024-09-18T14:57:31.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-09-18T14:57:31.686 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-09-18T14:57:31.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=4 2024-09-18T14:57:31.687 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-09-18T14:57:31.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=2 2024-09-18T14:57:31.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-09-18T14:57:31.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 2 ']' 2024-09-18T14:57:31.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 2 -a 2 '!=' 2 ']' 2024-09-18T14:57:31.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-18T14:57:31.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 7.0 --format=json 2024-09-18T14:57:31.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-09-18T14:57:32.056 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-09-18T14:57:32.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=1 2024-09-18T14:57:32.057 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-09-18T14:57:32.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=4 2024-09-18T14:57:32.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-09-18T14:57:32.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z 3.0 -a 1 = 1 -a 0 '!=' 4 ']' 2024-09-18T14:57:32.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n 3.0 -a 1 = 1 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2024-09-18T14:57:32.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:109: TEST_recovery_priority: PG3=7.0 2024-09-18T14:57:32.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:110: TEST_recovery_priority: POOLNUM3=7 2024-09-18T14:57:32.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:111: TEST_recovery_priority: pool3=test7 2024-09-18T14:57:32.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:112: TEST_recovery_priority: break 2024-09-18T14:57:32.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:115: TEST_recovery_priority: rm -f td/osd-recovery-prio/acting 2024-09-18T14:57:32.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:117: TEST_recovery_priority: '[' test3 = '' -o pool3 = '' ']' 2024-09-18T14:57:32.065 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: seq 1 10 2024-09-18T14:57:32.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-18T14:57:32.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 1 '!=' 1 -a 1 '!=' 3 -a 1 '!=' 7 ']' 2024-09-18T14:57:32.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-18T14:57:32.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 2 '!=' 1 -a 2 '!=' 3 -a 2 '!=' 7 ']' 2024-09-18T14:57:32.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test2 2024-09-18T14:57:32.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test2 2024-09-18T14:57:32.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2024-09-18T14:57:32.577 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test2' does not exist 2024-09-18T14:57:32.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-18T14:57:32.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 3 '!=' 1 -a 3 '!=' 3 -a 3 '!=' 7 ']' 2024-09-18T14:57:32.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-18T14:57:32.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 4 '!=' 1 -a 4 '!=' 3 -a 4 '!=' 7 ']' 2024-09-18T14:57:32.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test4 2024-09-18T14:57:32.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test4 2024-09-18T14:57:32.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test4 test4 --yes-i-really-really-mean-it 2024-09-18T14:57:33.207 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test4' does not exist 2024-09-18T14:57:33.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-18T14:57:33.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 5 '!=' 1 -a 5 '!=' 3 -a 5 '!=' 7 ']' 2024-09-18T14:57:33.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test5 2024-09-18T14:57:33.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test5 2024-09-18T14:57:33.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test5 test5 --yes-i-really-really-mean-it 2024-09-18T14:57:33.829 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test5' does not exist 2024-09-18T14:57:33.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-18T14:57:33.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 6 '!=' 1 -a 6 '!=' 3 -a 6 '!=' 7 ']' 2024-09-18T14:57:33.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test6 2024-09-18T14:57:33.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test6 2024-09-18T14:57:33.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test6 test6 --yes-i-really-really-mean-it 2024-09-18T14:57:34.581 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test6' does not exist 2024-09-18T14:57:34.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-18T14:57:34.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 7 '!=' 1 -a 7 '!=' 3 -a 7 '!=' 7 ']' 2024-09-18T14:57:34.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-18T14:57:34.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 8 '!=' 1 -a 8 '!=' 3 -a 8 '!=' 7 ']' 2024-09-18T14:57:34.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test8 2024-09-18T14:57:34.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test8 2024-09-18T14:57:34.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test8 test8 --yes-i-really-really-mean-it 2024-09-18T14:57:35.145 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test8' does not exist 2024-09-18T14:57:35.155 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-18T14:57:35.155 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 9 '!=' 1 -a 9 '!=' 3 -a 9 '!=' 7 ']' 2024-09-18T14:57:35.155 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test9 2024-09-18T14:57:35.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test9 2024-09-18T14:57:35.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test9 test9 --yes-i-really-really-mean-it 2024-09-18T14:57:35.728 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test9' does not exist 2024-09-18T14:57:35.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-18T14:57:35.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 10 '!=' 1 -a 10 '!=' 3 -a 10 '!=' 7 ']' 2024-09-18T14:57:35.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test10 2024-09-18T14:57:35.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test10 2024-09-18T14:57:35.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test10 test10 --yes-i-really-really-mean-it 2024-09-18T14:57:36.376 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test10' does not exist 2024-09-18T14:57:36.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:131: TEST_recovery_priority: ceph osd pool set test3 size 1 --yes-i-really-mean-it 2024-09-18T14:57:37.246 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 3 size to 1 2024-09-18T14:57:37.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:132: TEST_recovery_priority: ceph osd pool set test7 size 1 --yes-i-really-mean-it 2024-09-18T14:57:38.361 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 7 size to 1 2024-09-18T14:57:38.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:133: TEST_recovery_priority: wait_for_clean 2024-09-18T14:57:38.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T14:57:38.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T14:57:38.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T14:57:38.374 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T14:57:38.374 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T14:57:38.374 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T14:57:38.374 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T14:57:38.374 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T14:57:38.375 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T14:57:38.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T14:57:38.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T14:57:38.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T14:57:38.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T14:57:38.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T14:57:38.467 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T14:57:38.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T14:57:38.951 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T14:57:38.951 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T14:57:38.951 INFO:tasks.workunit.client.0.smithi047.stderr:3 2024-09-18T14:57:38.951 INFO:tasks.workunit.client.0.smithi047.stderr:4' 2024-09-18T14:57:38.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T14:57:38.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:57:38.951 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T14:57:39.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836505 2024-09-18T14:57:39.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836505 2024-09-18T14:57:39.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505' 2024-09-18T14:57:39.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:57:39.062 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T14:57:39.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672982 2024-09-18T14:57:39.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672982 2024-09-18T14:57:39.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505 1-42949672982' 2024-09-18T14:57:39.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:57:39.184 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T14:57:39.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509460 2024-09-18T14:57:39.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509460 2024-09-18T14:57:39.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505 1-42949672982 2-64424509460' 2024-09-18T14:57:39.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:57:39.295 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T14:57:39.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345937 2024-09-18T14:57:39.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345937 2024-09-18T14:57:39.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505 1-42949672982 2-64424509460 3-85899345937' 2024-09-18T14:57:39.406 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T14:57:39.406 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T14:57:39.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182414 2024-09-18T14:57:39.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182414 2024-09-18T14:57:39.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505 1-42949672982 2-64424509460 3-85899345937 4-107374182414' 2024-09-18T14:57:39.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:57:39.517 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836505 2024-09-18T14:57:39.517 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:57:39.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T14:57:39.518 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836505 2024-09-18T14:57:39.518 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:57:39.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836505 2024-09-18T14:57:39.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-09-18T14:57:39.519 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836505 2024-09-18T14:57:39.520 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:57:39.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836503 -lt 21474836505 2024-09-18T14:57:39.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T14:57:40.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T14:57:40.897 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T14:57:41.285 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836505 -lt 21474836505 2024-09-18T14:57:41.285 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:57:41.285 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672982 2024-09-18T14:57:41.285 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:57:41.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T14:57:41.287 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672982 2024-09-18T14:57:41.287 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:57:41.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672982 2024-09-18T14:57:41.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-09-18T14:57:41.288 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672982 2024-09-18T14:57:41.289 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T14:57:41.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672982 -lt 42949672982 2024-09-18T14:57:41.699 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:57:41.699 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509460 2024-09-18T14:57:41.699 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:57:41.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T14:57:41.700 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509460 2024-09-18T14:57:41.701 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:57:41.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509460 2024-09-18T14:57:41.702 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509460 2024-09-18T14:57:41.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-09-18T14:57:41.702 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T14:57:42.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509460 -lt 64424509460 2024-09-18T14:57:42.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:57:42.109 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345937 2024-09-18T14:57:42.109 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:57:42.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T14:57:42.111 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345937 2024-09-18T14:57:42.111 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:57:42.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345937 2024-09-18T14:57:42.112 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 85899345937 2024-09-18T14:57:42.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345937' 2024-09-18T14:57:42.112 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T14:57:42.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345937 -lt 85899345937 2024-09-18T14:57:42.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T14:57:42.519 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182414 2024-09-18T14:57:42.519 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T14:57:42.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T14:57:42.520 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182414 2024-09-18T14:57:42.520 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T14:57:42.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182414 2024-09-18T14:57:42.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182414' 2024-09-18T14:57:42.521 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.4 seq 107374182414 2024-09-18T14:57:42.522 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T14:57:42.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182415 -lt 107374182414 2024-09-18T14:57:42.922 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T14:57:42.923 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:57:42.923 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:57:43.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 == 0 2024-09-18T14:57:43.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T14:57:43.422 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T14:57:43.422 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T14:57:43.422 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T14:57:43.423 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T14:57:43.423 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T14:57:43.423 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T14:57:43.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-09-18T14:57:43.808 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T14:57:43.808 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T14:57:43.809 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T14:57:44.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 3 2024-09-18T14:57:44.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T14:57:44.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T14:57:44.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:135: TEST_recovery_priority: dd if=/dev/urandom of=td/osd-recovery-prio/data bs=1M count=10 2024-09-18T14:57:44.350 INFO:tasks.workunit.client.0.smithi047.stderr:10+0 records in 2024-09-18T14:57:44.350 INFO:tasks.workunit.client.0.smithi047.stderr:10+0 records out 2024-09-18T14:57:44.350 INFO:tasks.workunit.client.0.smithi047.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0279867 s, 375 MB/s 2024-09-18T14:57:44.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: p=1 2024-09-18T14:57:44.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:137: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2024-09-18T14:57:44.351 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: seq 1 200 2024-09-18T14:57:44.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:57:44.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj1-p1 td/osd-recovery-prio/data 2024-09-18T14:57:45.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:57:45.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj2-p1 td/osd-recovery-prio/data 2024-09-18T14:57:45.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:57:45.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj3-p1 td/osd-recovery-prio/data 2024-09-18T14:57:46.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:57:46.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj4-p1 td/osd-recovery-prio/data 2024-09-18T14:57:46.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:57:46.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj5-p1 td/osd-recovery-prio/data 2024-09-18T14:57:47.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:57:47.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj6-p1 td/osd-recovery-prio/data 2024-09-18T14:57:48.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:57:48.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj7-p1 td/osd-recovery-prio/data 2024-09-18T14:57:49.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:57:49.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj8-p1 td/osd-recovery-prio/data 2024-09-18T14:57:49.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:57:49.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj9-p1 td/osd-recovery-prio/data 2024-09-18T14:57:50.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:57:50.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj10-p1 td/osd-recovery-prio/data 2024-09-18T14:57:51.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:57:51.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj11-p1 td/osd-recovery-prio/data 2024-09-18T14:57:51.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:57:51.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj12-p1 td/osd-recovery-prio/data 2024-09-18T14:57:52.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:57:52.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj13-p1 td/osd-recovery-prio/data 2024-09-18T14:57:52.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:57:52.985 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj14-p1 td/osd-recovery-prio/data 2024-09-18T14:57:53.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:57:53.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj15-p1 td/osd-recovery-prio/data 2024-09-18T14:57:54.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:57:54.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj16-p1 td/osd-recovery-prio/data 2024-09-18T14:57:55.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:57:55.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj17-p1 td/osd-recovery-prio/data 2024-09-18T14:57:55.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:57:55.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj18-p1 td/osd-recovery-prio/data 2024-09-18T14:57:56.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:57:56.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj19-p1 td/osd-recovery-prio/data 2024-09-18T14:57:57.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:57:57.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj20-p1 td/osd-recovery-prio/data 2024-09-18T14:57:57.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:57:57.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj21-p1 td/osd-recovery-prio/data 2024-09-18T14:57:58.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:57:58.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj22-p1 td/osd-recovery-prio/data 2024-09-18T14:57:59.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:57:59.135 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj23-p1 td/osd-recovery-prio/data 2024-09-18T14:57:59.862 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:57:59.862 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj24-p1 td/osd-recovery-prio/data 2024-09-18T14:58:00.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:00.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj25-p1 td/osd-recovery-prio/data 2024-09-18T14:58:01.242 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:01.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj26-p1 td/osd-recovery-prio/data 2024-09-18T14:58:01.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:01.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj27-p1 td/osd-recovery-prio/data 2024-09-18T14:58:02.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:02.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj28-p1 td/osd-recovery-prio/data 2024-09-18T14:58:03.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:03.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj29-p1 td/osd-recovery-prio/data 2024-09-18T14:58:03.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:03.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj30-p1 td/osd-recovery-prio/data 2024-09-18T14:58:04.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:04.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj31-p1 td/osd-recovery-prio/data 2024-09-18T14:58:05.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:05.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj32-p1 td/osd-recovery-prio/data 2024-09-18T14:58:05.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:05.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj33-p1 td/osd-recovery-prio/data 2024-09-18T14:58:06.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:06.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj34-p1 td/osd-recovery-prio/data 2024-09-18T14:58:07.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:07.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj35-p1 td/osd-recovery-prio/data 2024-09-18T14:58:07.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:07.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj36-p1 td/osd-recovery-prio/data 2024-09-18T14:58:08.648 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:08.648 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj37-p1 td/osd-recovery-prio/data 2024-09-18T14:58:09.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:09.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj38-p1 td/osd-recovery-prio/data 2024-09-18T14:58:10.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:10.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj39-p1 td/osd-recovery-prio/data 2024-09-18T14:58:10.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:10.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj40-p1 td/osd-recovery-prio/data 2024-09-18T14:58:11.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:11.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj41-p1 td/osd-recovery-prio/data 2024-09-18T14:58:12.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:12.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj42-p1 td/osd-recovery-prio/data 2024-09-18T14:58:12.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:12.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj43-p1 td/osd-recovery-prio/data 2024-09-18T14:58:13.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:13.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj44-p1 td/osd-recovery-prio/data 2024-09-18T14:58:14.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:14.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj45-p1 td/osd-recovery-prio/data 2024-09-18T14:58:14.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:14.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj46-p1 td/osd-recovery-prio/data 2024-09-18T14:58:15.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:15.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj47-p1 td/osd-recovery-prio/data 2024-09-18T14:58:16.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:16.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj48-p1 td/osd-recovery-prio/data 2024-09-18T14:58:16.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:16.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj49-p1 td/osd-recovery-prio/data 2024-09-18T14:58:17.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:17.551 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj50-p1 td/osd-recovery-prio/data 2024-09-18T14:58:18.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:18.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj51-p1 td/osd-recovery-prio/data 2024-09-18T14:58:18.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:18.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj52-p1 td/osd-recovery-prio/data 2024-09-18T14:58:19.483 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:19.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj53-p1 td/osd-recovery-prio/data 2024-09-18T14:58:20.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:20.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj54-p1 td/osd-recovery-prio/data 2024-09-18T14:58:20.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:20.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj55-p1 td/osd-recovery-prio/data 2024-09-18T14:58:21.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:21.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj56-p1 td/osd-recovery-prio/data 2024-09-18T14:58:22.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:22.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj57-p1 td/osd-recovery-prio/data 2024-09-18T14:58:22.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:22.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj58-p1 td/osd-recovery-prio/data 2024-09-18T14:58:23.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:23.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj59-p1 td/osd-recovery-prio/data 2024-09-18T14:58:24.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:24.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj60-p1 td/osd-recovery-prio/data 2024-09-18T14:58:24.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:24.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj61-p1 td/osd-recovery-prio/data 2024-09-18T14:58:25.492 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:25.492 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj62-p1 td/osd-recovery-prio/data 2024-09-18T14:58:26.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:26.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj63-p1 td/osd-recovery-prio/data 2024-09-18T14:58:26.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:26.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj64-p1 td/osd-recovery-prio/data 2024-09-18T14:58:27.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:27.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj65-p1 td/osd-recovery-prio/data 2024-09-18T14:58:28.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:28.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj66-p1 td/osd-recovery-prio/data 2024-09-18T14:58:28.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:28.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj67-p1 td/osd-recovery-prio/data 2024-09-18T14:58:29.476 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:29.476 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj68-p1 td/osd-recovery-prio/data 2024-09-18T14:58:30.119 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:30.119 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj69-p1 td/osd-recovery-prio/data 2024-09-18T14:58:30.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:30.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj70-p1 td/osd-recovery-prio/data 2024-09-18T14:58:31.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:31.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj71-p1 td/osd-recovery-prio/data 2024-09-18T14:58:32.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:32.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj72-p1 td/osd-recovery-prio/data 2024-09-18T14:58:32.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:32.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj73-p1 td/osd-recovery-prio/data 2024-09-18T14:58:33.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:33.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj74-p1 td/osd-recovery-prio/data 2024-09-18T14:58:34.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:34.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj75-p1 td/osd-recovery-prio/data 2024-09-18T14:58:34.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:34.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj76-p1 td/osd-recovery-prio/data 2024-09-18T14:58:35.549 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:35.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj77-p1 td/osd-recovery-prio/data 2024-09-18T14:58:36.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:36.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj78-p1 td/osd-recovery-prio/data 2024-09-18T14:58:36.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:36.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj79-p1 td/osd-recovery-prio/data 2024-09-18T14:58:37.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:37.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj80-p1 td/osd-recovery-prio/data 2024-09-18T14:58:38.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:38.194 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj81-p1 td/osd-recovery-prio/data 2024-09-18T14:58:38.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:38.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj82-p1 td/osd-recovery-prio/data 2024-09-18T14:58:39.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:39.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj83-p1 td/osd-recovery-prio/data 2024-09-18T14:58:40.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:40.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj84-p1 td/osd-recovery-prio/data 2024-09-18T14:58:41.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:41.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj85-p1 td/osd-recovery-prio/data 2024-09-18T14:58:41.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:41.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj86-p1 td/osd-recovery-prio/data 2024-09-18T14:58:42.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:42.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj87-p1 td/osd-recovery-prio/data 2024-09-18T14:58:43.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:43.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj88-p1 td/osd-recovery-prio/data 2024-09-18T14:58:43.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:43.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj89-p1 td/osd-recovery-prio/data 2024-09-18T14:58:44.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:44.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj90-p1 td/osd-recovery-prio/data 2024-09-18T14:58:45.021 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:45.021 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj91-p1 td/osd-recovery-prio/data 2024-09-18T14:58:45.699 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:45.699 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj92-p1 td/osd-recovery-prio/data 2024-09-18T14:58:46.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:46.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj93-p1 td/osd-recovery-prio/data 2024-09-18T14:58:47.021 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:47.021 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj94-p1 td/osd-recovery-prio/data 2024-09-18T14:58:47.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:47.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj95-p1 td/osd-recovery-prio/data 2024-09-18T14:58:48.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:48.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj96-p1 td/osd-recovery-prio/data 2024-09-18T14:58:48.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:48.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj97-p1 td/osd-recovery-prio/data 2024-09-18T14:58:49.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:49.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj98-p1 td/osd-recovery-prio/data 2024-09-18T14:58:50.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:50.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj99-p1 td/osd-recovery-prio/data 2024-09-18T14:58:50.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:50.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj100-p1 td/osd-recovery-prio/data 2024-09-18T14:58:51.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:51.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj101-p1 td/osd-recovery-prio/data 2024-09-18T14:58:52.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:52.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj102-p1 td/osd-recovery-prio/data 2024-09-18T14:58:52.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:52.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj103-p1 td/osd-recovery-prio/data 2024-09-18T14:58:53.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:53.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj104-p1 td/osd-recovery-prio/data 2024-09-18T14:58:54.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:54.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj105-p1 td/osd-recovery-prio/data 2024-09-18T14:58:54.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:54.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj106-p1 td/osd-recovery-prio/data 2024-09-18T14:58:55.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:55.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj107-p1 td/osd-recovery-prio/data 2024-09-18T14:58:56.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:56.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj108-p1 td/osd-recovery-prio/data 2024-09-18T14:58:57.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:57.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj109-p1 td/osd-recovery-prio/data 2024-09-18T14:58:57.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:57.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj110-p1 td/osd-recovery-prio/data 2024-09-18T14:58:58.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:58.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj111-p1 td/osd-recovery-prio/data 2024-09-18T14:58:58.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:58.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj112-p1 td/osd-recovery-prio/data 2024-09-18T14:58:59.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:58:59.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj113-p1 td/osd-recovery-prio/data 2024-09-18T14:59:00.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:00.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj114-p1 td/osd-recovery-prio/data 2024-09-18T14:59:00.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:00.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj115-p1 td/osd-recovery-prio/data 2024-09-18T14:59:01.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:01.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj116-p1 td/osd-recovery-prio/data 2024-09-18T14:59:02.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:02.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj117-p1 td/osd-recovery-prio/data 2024-09-18T14:59:02.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:02.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj118-p1 td/osd-recovery-prio/data 2024-09-18T14:59:03.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:03.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj119-p1 td/osd-recovery-prio/data 2024-09-18T14:59:04.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:04.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj120-p1 td/osd-recovery-prio/data 2024-09-18T14:59:04.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:04.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj121-p1 td/osd-recovery-prio/data 2024-09-18T14:59:05.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:05.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj122-p1 td/osd-recovery-prio/data 2024-09-18T14:59:06.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:06.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj123-p1 td/osd-recovery-prio/data 2024-09-18T14:59:06.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:06.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj124-p1 td/osd-recovery-prio/data 2024-09-18T14:59:07.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:07.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj125-p1 td/osd-recovery-prio/data 2024-09-18T14:59:08.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:08.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj126-p1 td/osd-recovery-prio/data 2024-09-18T14:59:08.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:08.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj127-p1 td/osd-recovery-prio/data 2024-09-18T14:59:09.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:09.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj128-p1 td/osd-recovery-prio/data 2024-09-18T14:59:10.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:10.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj129-p1 td/osd-recovery-prio/data 2024-09-18T14:59:10.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:10.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj130-p1 td/osd-recovery-prio/data 2024-09-18T14:59:11.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:11.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj131-p1 td/osd-recovery-prio/data 2024-09-18T14:59:12.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:12.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj132-p1 td/osd-recovery-prio/data 2024-09-18T14:59:13.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:13.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj133-p1 td/osd-recovery-prio/data 2024-09-18T14:59:13.648 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:13.648 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj134-p1 td/osd-recovery-prio/data 2024-09-18T14:59:14.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:14.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj135-p1 td/osd-recovery-prio/data 2024-09-18T14:59:15.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:15.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj136-p1 td/osd-recovery-prio/data 2024-09-18T14:59:15.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:15.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj137-p1 td/osd-recovery-prio/data 2024-09-18T14:59:16.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:16.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj138-p1 td/osd-recovery-prio/data 2024-09-18T14:59:16.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:16.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj139-p1 td/osd-recovery-prio/data 2024-09-18T14:59:17.648 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:17.648 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj140-p1 td/osd-recovery-prio/data 2024-09-18T14:59:18.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:18.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj141-p1 td/osd-recovery-prio/data 2024-09-18T14:59:18.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:18.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj142-p1 td/osd-recovery-prio/data 2024-09-18T14:59:19.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:19.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj143-p1 td/osd-recovery-prio/data 2024-09-18T14:59:20.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:20.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj144-p1 td/osd-recovery-prio/data 2024-09-18T14:59:20.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:20.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj145-p1 td/osd-recovery-prio/data 2024-09-18T14:59:21.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:21.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj146-p1 td/osd-recovery-prio/data 2024-09-18T14:59:22.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:22.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj147-p1 td/osd-recovery-prio/data 2024-09-18T14:59:22.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:22.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj148-p1 td/osd-recovery-prio/data 2024-09-18T14:59:23.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:23.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj149-p1 td/osd-recovery-prio/data 2024-09-18T14:59:24.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:24.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj150-p1 td/osd-recovery-prio/data 2024-09-18T14:59:24.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:24.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj151-p1 td/osd-recovery-prio/data 2024-09-18T14:59:25.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:25.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj152-p1 td/osd-recovery-prio/data 2024-09-18T14:59:26.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:26.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj153-p1 td/osd-recovery-prio/data 2024-09-18T14:59:26.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:26.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj154-p1 td/osd-recovery-prio/data 2024-09-18T14:59:27.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:27.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj155-p1 td/osd-recovery-prio/data 2024-09-18T14:59:28.157 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:28.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj156-p1 td/osd-recovery-prio/data 2024-09-18T14:59:28.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:28.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj157-p1 td/osd-recovery-prio/data 2024-09-18T14:59:29.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:29.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj158-p1 td/osd-recovery-prio/data 2024-09-18T14:59:30.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:30.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj159-p1 td/osd-recovery-prio/data 2024-09-18T14:59:30.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:30.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj160-p1 td/osd-recovery-prio/data 2024-09-18T14:59:31.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:31.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj161-p1 td/osd-recovery-prio/data 2024-09-18T14:59:32.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:32.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj162-p1 td/osd-recovery-prio/data 2024-09-18T14:59:32.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:32.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj163-p1 td/osd-recovery-prio/data 2024-09-18T14:59:33.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:33.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj164-p1 td/osd-recovery-prio/data 2024-09-18T14:59:34.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:34.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj165-p1 td/osd-recovery-prio/data 2024-09-18T14:59:34.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:34.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj166-p1 td/osd-recovery-prio/data 2024-09-18T14:59:35.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:35.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj167-p1 td/osd-recovery-prio/data 2024-09-18T14:59:36.052 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:36.052 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj168-p1 td/osd-recovery-prio/data 2024-09-18T14:59:36.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:36.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj169-p1 td/osd-recovery-prio/data 2024-09-18T14:59:37.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:37.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj170-p1 td/osd-recovery-prio/data 2024-09-18T14:59:38.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:38.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj171-p1 td/osd-recovery-prio/data 2024-09-18T14:59:38.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:38.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj172-p1 td/osd-recovery-prio/data 2024-09-18T14:59:39.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:39.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj173-p1 td/osd-recovery-prio/data 2024-09-18T14:59:40.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:40.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj174-p1 td/osd-recovery-prio/data 2024-09-18T14:59:40.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:40.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj175-p1 td/osd-recovery-prio/data 2024-09-18T14:59:41.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:41.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj176-p1 td/osd-recovery-prio/data 2024-09-18T14:59:42.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:42.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj177-p1 td/osd-recovery-prio/data 2024-09-18T14:59:42.907 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:42.907 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj178-p1 td/osd-recovery-prio/data 2024-09-18T14:59:43.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:43.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj179-p1 td/osd-recovery-prio/data 2024-09-18T14:59:44.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:44.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj180-p1 td/osd-recovery-prio/data 2024-09-18T14:59:44.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:44.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj181-p1 td/osd-recovery-prio/data 2024-09-18T14:59:45.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:45.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj182-p1 td/osd-recovery-prio/data 2024-09-18T14:59:46.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:46.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj183-p1 td/osd-recovery-prio/data 2024-09-18T14:59:46.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:46.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj184-p1 td/osd-recovery-prio/data 2024-09-18T14:59:47.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:47.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj185-p1 td/osd-recovery-prio/data 2024-09-18T14:59:48.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:48.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj186-p1 td/osd-recovery-prio/data 2024-09-18T14:59:48.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:48.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj187-p1 td/osd-recovery-prio/data 2024-09-18T14:59:49.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:49.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj188-p1 td/osd-recovery-prio/data 2024-09-18T14:59:50.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:50.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj189-p1 td/osd-recovery-prio/data 2024-09-18T14:59:50.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:50.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj190-p1 td/osd-recovery-prio/data 2024-09-18T14:59:51.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:51.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj191-p1 td/osd-recovery-prio/data 2024-09-18T14:59:52.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:52.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj192-p1 td/osd-recovery-prio/data 2024-09-18T14:59:52.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:52.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj193-p1 td/osd-recovery-prio/data 2024-09-18T14:59:53.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:53.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj194-p1 td/osd-recovery-prio/data 2024-09-18T14:59:54.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:54.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj195-p1 td/osd-recovery-prio/data 2024-09-18T14:59:54.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:54.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj196-p1 td/osd-recovery-prio/data 2024-09-18T14:59:55.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:55.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj197-p1 td/osd-recovery-prio/data 2024-09-18T14:59:56.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:56.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj198-p1 td/osd-recovery-prio/data 2024-09-18T14:59:56.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:56.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj199-p1 td/osd-recovery-prio/data 2024-09-18T14:59:57.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:57.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj200-p1 td/osd-recovery-prio/data 2024-09-18T14:59:58.222 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: expr 1 + 1 2024-09-18T14:59:58.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: p=2 2024-09-18T14:59:58.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:137: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2024-09-18T14:59:58.223 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: seq 1 200 2024-09-18T14:59:58.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:58.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj1-p2 td/osd-recovery-prio/data 2024-09-18T14:59:58.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:58.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj2-p2 td/osd-recovery-prio/data 2024-09-18T14:59:59.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:59.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj3-p2 td/osd-recovery-prio/data 2024-09-18T14:59:59.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:59.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj4-p2 td/osd-recovery-prio/data 2024-09-18T14:59:59.761 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T14:59:59.761 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj5-p2 td/osd-recovery-prio/data 2024-09-18T15:00:00.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:00.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj6-p2 td/osd-recovery-prio/data 2024-09-18T15:00:00.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:00.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj7-p2 td/osd-recovery-prio/data 2024-09-18T15:00:00.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:00.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj8-p2 td/osd-recovery-prio/data 2024-09-18T15:00:01.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:01.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj9-p2 td/osd-recovery-prio/data 2024-09-18T15:00:01.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:01.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj10-p2 td/osd-recovery-prio/data 2024-09-18T15:00:01.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:01.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj11-p2 td/osd-recovery-prio/data 2024-09-18T15:00:02.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:02.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj12-p2 td/osd-recovery-prio/data 2024-09-18T15:00:02.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:02.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj13-p2 td/osd-recovery-prio/data 2024-09-18T15:00:03.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:03.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj14-p2 td/osd-recovery-prio/data 2024-09-18T15:00:03.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:03.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj15-p2 td/osd-recovery-prio/data 2024-09-18T15:00:03.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:03.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj16-p2 td/osd-recovery-prio/data 2024-09-18T15:00:04.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:04.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj17-p2 td/osd-recovery-prio/data 2024-09-18T15:00:04.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:04.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj18-p2 td/osd-recovery-prio/data 2024-09-18T15:00:04.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:04.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj19-p2 td/osd-recovery-prio/data 2024-09-18T15:00:05.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:05.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj20-p2 td/osd-recovery-prio/data 2024-09-18T15:00:05.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:05.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj21-p2 td/osd-recovery-prio/data 2024-09-18T15:00:05.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:05.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj22-p2 td/osd-recovery-prio/data 2024-09-18T15:00:06.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:06.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj23-p2 td/osd-recovery-prio/data 2024-09-18T15:00:06.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:06.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj24-p2 td/osd-recovery-prio/data 2024-09-18T15:00:07.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:07.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj25-p2 td/osd-recovery-prio/data 2024-09-18T15:00:07.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:07.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj26-p2 td/osd-recovery-prio/data 2024-09-18T15:00:07.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:07.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj27-p2 td/osd-recovery-prio/data 2024-09-18T15:00:08.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:08.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj28-p2 td/osd-recovery-prio/data 2024-09-18T15:00:08.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:08.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj29-p2 td/osd-recovery-prio/data 2024-09-18T15:00:08.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:08.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj30-p2 td/osd-recovery-prio/data 2024-09-18T15:00:09.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:09.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj31-p2 td/osd-recovery-prio/data 2024-09-18T15:00:09.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:09.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj32-p2 td/osd-recovery-prio/data 2024-09-18T15:00:09.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:09.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj33-p2 td/osd-recovery-prio/data 2024-09-18T15:00:10.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:10.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj34-p2 td/osd-recovery-prio/data 2024-09-18T15:00:10.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:10.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj35-p2 td/osd-recovery-prio/data 2024-09-18T15:00:11.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:11.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj36-p2 td/osd-recovery-prio/data 2024-09-18T15:00:11.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:11.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj37-p2 td/osd-recovery-prio/data 2024-09-18T15:00:11.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:11.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj38-p2 td/osd-recovery-prio/data 2024-09-18T15:00:12.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:12.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj39-p2 td/osd-recovery-prio/data 2024-09-18T15:00:12.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:12.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj40-p2 td/osd-recovery-prio/data 2024-09-18T15:00:12.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:12.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj41-p2 td/osd-recovery-prio/data 2024-09-18T15:00:13.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:13.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj42-p2 td/osd-recovery-prio/data 2024-09-18T15:00:13.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:13.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj43-p2 td/osd-recovery-prio/data 2024-09-18T15:00:14.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:14.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj44-p2 td/osd-recovery-prio/data 2024-09-18T15:00:14.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:14.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj45-p2 td/osd-recovery-prio/data 2024-09-18T15:00:14.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:14.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj46-p2 td/osd-recovery-prio/data 2024-09-18T15:00:15.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:15.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj47-p2 td/osd-recovery-prio/data 2024-09-18T15:00:15.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:15.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj48-p2 td/osd-recovery-prio/data 2024-09-18T15:00:16.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:16.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj49-p2 td/osd-recovery-prio/data 2024-09-18T15:00:16.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:16.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj50-p2 td/osd-recovery-prio/data 2024-09-18T15:00:16.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:16.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj51-p2 td/osd-recovery-prio/data 2024-09-18T15:00:17.264 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:17.265 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj52-p2 td/osd-recovery-prio/data 2024-09-18T15:00:17.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:17.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj53-p2 td/osd-recovery-prio/data 2024-09-18T15:00:17.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:17.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj54-p2 td/osd-recovery-prio/data 2024-09-18T15:00:18.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:18.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj55-p2 td/osd-recovery-prio/data 2024-09-18T15:00:18.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:18.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj56-p2 td/osd-recovery-prio/data 2024-09-18T15:00:19.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:19.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj57-p2 td/osd-recovery-prio/data 2024-09-18T15:00:19.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:19.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj58-p2 td/osd-recovery-prio/data 2024-09-18T15:00:19.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:19.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj59-p2 td/osd-recovery-prio/data 2024-09-18T15:00:20.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:20.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj60-p2 td/osd-recovery-prio/data 2024-09-18T15:00:20.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:20.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj61-p2 td/osd-recovery-prio/data 2024-09-18T15:00:20.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:20.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj62-p2 td/osd-recovery-prio/data 2024-09-18T15:00:21.247 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:21.248 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj63-p2 td/osd-recovery-prio/data 2024-09-18T15:00:21.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:21.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj64-p2 td/osd-recovery-prio/data 2024-09-18T15:00:21.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:21.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj65-p2 td/osd-recovery-prio/data 2024-09-18T15:00:22.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:22.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj66-p2 td/osd-recovery-prio/data 2024-09-18T15:00:22.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:22.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj67-p2 td/osd-recovery-prio/data 2024-09-18T15:00:23.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:23.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj68-p2 td/osd-recovery-prio/data 2024-09-18T15:00:23.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:23.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj69-p2 td/osd-recovery-prio/data 2024-09-18T15:00:23.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:23.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj70-p2 td/osd-recovery-prio/data 2024-09-18T15:00:24.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:24.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj71-p2 td/osd-recovery-prio/data 2024-09-18T15:00:24.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:24.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj72-p2 td/osd-recovery-prio/data 2024-09-18T15:00:24.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:24.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj73-p2 td/osd-recovery-prio/data 2024-09-18T15:00:25.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:25.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj74-p2 td/osd-recovery-prio/data 2024-09-18T15:00:25.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:25.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj75-p2 td/osd-recovery-prio/data 2024-09-18T15:00:26.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:26.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj76-p2 td/osd-recovery-prio/data 2024-09-18T15:00:26.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:26.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj77-p2 td/osd-recovery-prio/data 2024-09-18T15:00:26.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:26.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj78-p2 td/osd-recovery-prio/data 2024-09-18T15:00:27.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:27.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj79-p2 td/osd-recovery-prio/data 2024-09-18T15:00:27.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:27.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj80-p2 td/osd-recovery-prio/data 2024-09-18T15:00:27.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:27.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj81-p2 td/osd-recovery-prio/data 2024-09-18T15:00:28.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:28.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj82-p2 td/osd-recovery-prio/data 2024-09-18T15:00:28.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:28.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj83-p2 td/osd-recovery-prio/data 2024-09-18T15:00:28.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:28.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj84-p2 td/osd-recovery-prio/data 2024-09-18T15:00:29.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:29.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj85-p2 td/osd-recovery-prio/data 2024-09-18T15:00:29.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:29.642 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj86-p2 td/osd-recovery-prio/data 2024-09-18T15:00:29.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:29.985 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj87-p2 td/osd-recovery-prio/data 2024-09-18T15:00:30.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:30.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj88-p2 td/osd-recovery-prio/data 2024-09-18T15:00:30.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:30.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj89-p2 td/osd-recovery-prio/data 2024-09-18T15:00:31.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:31.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj90-p2 td/osd-recovery-prio/data 2024-09-18T15:00:31.432 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:31.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj91-p2 td/osd-recovery-prio/data 2024-09-18T15:00:31.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:31.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj92-p2 td/osd-recovery-prio/data 2024-09-18T15:00:32.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:32.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj93-p2 td/osd-recovery-prio/data 2024-09-18T15:00:32.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:32.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj94-p2 td/osd-recovery-prio/data 2024-09-18T15:00:32.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:32.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj95-p2 td/osd-recovery-prio/data 2024-09-18T15:00:33.299 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:33.299 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj96-p2 td/osd-recovery-prio/data 2024-09-18T15:00:33.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:33.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj97-p2 td/osd-recovery-prio/data 2024-09-18T15:00:34.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:34.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj98-p2 td/osd-recovery-prio/data 2024-09-18T15:00:34.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:34.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj99-p2 td/osd-recovery-prio/data 2024-09-18T15:00:34.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:34.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj100-p2 td/osd-recovery-prio/data 2024-09-18T15:00:35.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:35.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj101-p2 td/osd-recovery-prio/data 2024-09-18T15:00:35.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:35.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj102-p2 td/osd-recovery-prio/data 2024-09-18T15:00:35.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:35.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj103-p2 td/osd-recovery-prio/data 2024-09-18T15:00:36.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:36.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj104-p2 td/osd-recovery-prio/data 2024-09-18T15:00:36.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:36.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj105-p2 td/osd-recovery-prio/data 2024-09-18T15:00:37.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:37.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj106-p2 td/osd-recovery-prio/data 2024-09-18T15:00:37.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:37.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj107-p2 td/osd-recovery-prio/data 2024-09-18T15:00:37.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:37.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj108-p2 td/osd-recovery-prio/data 2024-09-18T15:00:38.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:38.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj109-p2 td/osd-recovery-prio/data 2024-09-18T15:00:38.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:38.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj110-p2 td/osd-recovery-prio/data 2024-09-18T15:00:39.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:39.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj111-p2 td/osd-recovery-prio/data 2024-09-18T15:00:39.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:39.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj112-p2 td/osd-recovery-prio/data 2024-09-18T15:00:39.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:39.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj113-p2 td/osd-recovery-prio/data 2024-09-18T15:00:40.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:40.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj114-p2 td/osd-recovery-prio/data 2024-09-18T15:00:40.419 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:40.420 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj115-p2 td/osd-recovery-prio/data 2024-09-18T15:00:40.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:40.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj116-p2 td/osd-recovery-prio/data 2024-09-18T15:00:41.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:41.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj117-p2 td/osd-recovery-prio/data 2024-09-18T15:00:41.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:41.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj118-p2 td/osd-recovery-prio/data 2024-09-18T15:00:41.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:41.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj119-p2 td/osd-recovery-prio/data 2024-09-18T15:00:42.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:42.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj120-p2 td/osd-recovery-prio/data 2024-09-18T15:00:42.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:42.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj121-p2 td/osd-recovery-prio/data 2024-09-18T15:00:43.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:43.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj122-p2 td/osd-recovery-prio/data 2024-09-18T15:00:43.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:43.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj123-p2 td/osd-recovery-prio/data 2024-09-18T15:00:43.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:43.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj124-p2 td/osd-recovery-prio/data 2024-09-18T15:00:44.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:44.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj125-p2 td/osd-recovery-prio/data 2024-09-18T15:00:44.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:44.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj126-p2 td/osd-recovery-prio/data 2024-09-18T15:00:45.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:45.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj127-p2 td/osd-recovery-prio/data 2024-09-18T15:00:45.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:45.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj128-p2 td/osd-recovery-prio/data 2024-09-18T15:00:45.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:45.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj129-p2 td/osd-recovery-prio/data 2024-09-18T15:00:46.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:46.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj130-p2 td/osd-recovery-prio/data 2024-09-18T15:00:46.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:46.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj131-p2 td/osd-recovery-prio/data 2024-09-18T15:00:47.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:47.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj132-p2 td/osd-recovery-prio/data 2024-09-18T15:00:47.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:47.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj133-p2 td/osd-recovery-prio/data 2024-09-18T15:00:47.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:47.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj134-p2 td/osd-recovery-prio/data 2024-09-18T15:00:48.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:48.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj135-p2 td/osd-recovery-prio/data 2024-09-18T15:00:48.551 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:48.551 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj136-p2 td/osd-recovery-prio/data 2024-09-18T15:00:48.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:48.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj137-p2 td/osd-recovery-prio/data 2024-09-18T15:00:49.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:49.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj138-p2 td/osd-recovery-prio/data 2024-09-18T15:00:49.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:49.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj139-p2 td/osd-recovery-prio/data 2024-09-18T15:00:50.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:50.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj140-p2 td/osd-recovery-prio/data 2024-09-18T15:00:50.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:50.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj141-p2 td/osd-recovery-prio/data 2024-09-18T15:00:50.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:50.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj142-p2 td/osd-recovery-prio/data 2024-09-18T15:00:51.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:51.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj143-p2 td/osd-recovery-prio/data 2024-09-18T15:00:51.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:51.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj144-p2 td/osd-recovery-prio/data 2024-09-18T15:00:51.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:51.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj145-p2 td/osd-recovery-prio/data 2024-09-18T15:00:52.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:52.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj146-p2 td/osd-recovery-prio/data 2024-09-18T15:00:52.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:52.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj147-p2 td/osd-recovery-prio/data 2024-09-18T15:00:53.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:53.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj148-p2 td/osd-recovery-prio/data 2024-09-18T15:00:53.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:53.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj149-p2 td/osd-recovery-prio/data 2024-09-18T15:00:53.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:53.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj150-p2 td/osd-recovery-prio/data 2024-09-18T15:00:54.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:54.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj151-p2 td/osd-recovery-prio/data 2024-09-18T15:00:54.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:54.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj152-p2 td/osd-recovery-prio/data 2024-09-18T15:00:54.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:54.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj153-p2 td/osd-recovery-prio/data 2024-09-18T15:00:55.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:55.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj154-p2 td/osd-recovery-prio/data 2024-09-18T15:00:55.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:55.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj155-p2 td/osd-recovery-prio/data 2024-09-18T15:00:56.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:56.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj156-p2 td/osd-recovery-prio/data 2024-09-18T15:00:56.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:56.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj157-p2 td/osd-recovery-prio/data 2024-09-18T15:00:56.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:56.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj158-p2 td/osd-recovery-prio/data 2024-09-18T15:00:57.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:57.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj159-p2 td/osd-recovery-prio/data 2024-09-18T15:00:57.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:57.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj160-p2 td/osd-recovery-prio/data 2024-09-18T15:00:57.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:57.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj161-p2 td/osd-recovery-prio/data 2024-09-18T15:00:58.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:58.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj162-p2 td/osd-recovery-prio/data 2024-09-18T15:00:58.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:58.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj163-p2 td/osd-recovery-prio/data 2024-09-18T15:00:59.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:59.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj164-p2 td/osd-recovery-prio/data 2024-09-18T15:00:59.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:59.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj165-p2 td/osd-recovery-prio/data 2024-09-18T15:00:59.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:00:59.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj166-p2 td/osd-recovery-prio/data 2024-09-18T15:01:00.096 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:00.096 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj167-p2 td/osd-recovery-prio/data 2024-09-18T15:01:00.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:00.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj168-p2 td/osd-recovery-prio/data 2024-09-18T15:01:00.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:00.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj169-p2 td/osd-recovery-prio/data 2024-09-18T15:01:01.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:01.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj170-p2 td/osd-recovery-prio/data 2024-09-18T15:01:01.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:01.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj171-p2 td/osd-recovery-prio/data 2024-09-18T15:01:02.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:02.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj172-p2 td/osd-recovery-prio/data 2024-09-18T15:01:02.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:02.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj173-p2 td/osd-recovery-prio/data 2024-09-18T15:01:02.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:02.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj174-p2 td/osd-recovery-prio/data 2024-09-18T15:01:03.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:03.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj175-p2 td/osd-recovery-prio/data 2024-09-18T15:01:03.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:03.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj176-p2 td/osd-recovery-prio/data 2024-09-18T15:01:03.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:03.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj177-p2 td/osd-recovery-prio/data 2024-09-18T15:01:04.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:04.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj178-p2 td/osd-recovery-prio/data 2024-09-18T15:01:04.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:04.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj179-p2 td/osd-recovery-prio/data 2024-09-18T15:01:05.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:05.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj180-p2 td/osd-recovery-prio/data 2024-09-18T15:01:05.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:05.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj181-p2 td/osd-recovery-prio/data 2024-09-18T15:01:05.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:05.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj182-p2 td/osd-recovery-prio/data 2024-09-18T15:01:06.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:06.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj183-p2 td/osd-recovery-prio/data 2024-09-18T15:01:06.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:06.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj184-p2 td/osd-recovery-prio/data 2024-09-18T15:01:06.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:06.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj185-p2 td/osd-recovery-prio/data 2024-09-18T15:01:07.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:07.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj186-p2 td/osd-recovery-prio/data 2024-09-18T15:01:07.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:07.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj187-p2 td/osd-recovery-prio/data 2024-09-18T15:01:07.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:07.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj188-p2 td/osd-recovery-prio/data 2024-09-18T15:01:08.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:08.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj189-p2 td/osd-recovery-prio/data 2024-09-18T15:01:08.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:08.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj190-p2 td/osd-recovery-prio/data 2024-09-18T15:01:09.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:09.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj191-p2 td/osd-recovery-prio/data 2024-09-18T15:01:09.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:09.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj192-p2 td/osd-recovery-prio/data 2024-09-18T15:01:09.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:09.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj193-p2 td/osd-recovery-prio/data 2024-09-18T15:01:10.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:10.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj194-p2 td/osd-recovery-prio/data 2024-09-18T15:01:10.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:10.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj195-p2 td/osd-recovery-prio/data 2024-09-18T15:01:10.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:10.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj196-p2 td/osd-recovery-prio/data 2024-09-18T15:01:11.248 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:11.248 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj197-p2 td/osd-recovery-prio/data 2024-09-18T15:01:11.599 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:11.599 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj198-p2 td/osd-recovery-prio/data 2024-09-18T15:01:11.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:11.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj199-p2 td/osd-recovery-prio/data 2024-09-18T15:01:12.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:12.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj200-p2 td/osd-recovery-prio/data 2024-09-18T15:01:12.646 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: expr 2 + 1 2024-09-18T15:01:12.648 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: p=3 2024-09-18T15:01:12.648 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:137: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2024-09-18T15:01:12.648 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: seq 1 200 2024-09-18T15:01:12.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:12.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj1-p3 td/osd-recovery-prio/data 2024-09-18T15:01:13.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:13.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj2-p3 td/osd-recovery-prio/data 2024-09-18T15:01:13.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:13.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj3-p3 td/osd-recovery-prio/data 2024-09-18T15:01:13.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:13.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj4-p3 td/osd-recovery-prio/data 2024-09-18T15:01:14.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:14.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj5-p3 td/osd-recovery-prio/data 2024-09-18T15:01:14.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:14.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj6-p3 td/osd-recovery-prio/data 2024-09-18T15:01:14.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:14.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj7-p3 td/osd-recovery-prio/data 2024-09-18T15:01:15.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:15.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj8-p3 td/osd-recovery-prio/data 2024-09-18T15:01:15.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:15.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj9-p3 td/osd-recovery-prio/data 2024-09-18T15:01:16.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:16.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj10-p3 td/osd-recovery-prio/data 2024-09-18T15:01:16.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:16.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj11-p3 td/osd-recovery-prio/data 2024-09-18T15:01:16.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:16.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj12-p3 td/osd-recovery-prio/data 2024-09-18T15:01:17.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:17.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj13-p3 td/osd-recovery-prio/data 2024-09-18T15:01:17.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:17.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj14-p3 td/osd-recovery-prio/data 2024-09-18T15:01:18.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:18.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj15-p3 td/osd-recovery-prio/data 2024-09-18T15:01:18.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:18.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj16-p3 td/osd-recovery-prio/data 2024-09-18T15:01:18.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:18.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj17-p3 td/osd-recovery-prio/data 2024-09-18T15:01:19.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:19.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj18-p3 td/osd-recovery-prio/data 2024-09-18T15:01:19.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:19.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj19-p3 td/osd-recovery-prio/data 2024-09-18T15:01:19.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:19.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj20-p3 td/osd-recovery-prio/data 2024-09-18T15:01:20.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:20.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj21-p3 td/osd-recovery-prio/data 2024-09-18T15:01:20.514 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:20.514 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj22-p3 td/osd-recovery-prio/data 2024-09-18T15:01:20.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:20.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj23-p3 td/osd-recovery-prio/data 2024-09-18T15:01:21.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:21.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj24-p3 td/osd-recovery-prio/data 2024-09-18T15:01:21.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:21.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj25-p3 td/osd-recovery-prio/data 2024-09-18T15:01:21.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:21.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj26-p3 td/osd-recovery-prio/data 2024-09-18T15:01:22.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:22.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj27-p3 td/osd-recovery-prio/data 2024-09-18T15:01:22.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:22.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj28-p3 td/osd-recovery-prio/data 2024-09-18T15:01:23.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:23.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj29-p3 td/osd-recovery-prio/data 2024-09-18T15:01:23.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:23.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj30-p3 td/osd-recovery-prio/data 2024-09-18T15:01:23.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:23.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj31-p3 td/osd-recovery-prio/data 2024-09-18T15:01:24.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:24.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj32-p3 td/osd-recovery-prio/data 2024-09-18T15:01:24.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:24.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj33-p3 td/osd-recovery-prio/data 2024-09-18T15:01:24.956 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:24.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj34-p3 td/osd-recovery-prio/data 2024-09-18T15:01:25.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:25.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj35-p3 td/osd-recovery-prio/data 2024-09-18T15:01:25.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:25.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj36-p3 td/osd-recovery-prio/data 2024-09-18T15:01:25.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:25.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj37-p3 td/osd-recovery-prio/data 2024-09-18T15:01:26.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:26.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj38-p3 td/osd-recovery-prio/data 2024-09-18T15:01:26.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:26.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj39-p3 td/osd-recovery-prio/data 2024-09-18T15:01:27.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:27.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj40-p3 td/osd-recovery-prio/data 2024-09-18T15:01:27.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:27.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj41-p3 td/osd-recovery-prio/data 2024-09-18T15:01:27.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:27.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj42-p3 td/osd-recovery-prio/data 2024-09-18T15:01:28.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:28.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj43-p3 td/osd-recovery-prio/data 2024-09-18T15:01:28.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:28.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj44-p3 td/osd-recovery-prio/data 2024-09-18T15:01:28.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:28.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj45-p3 td/osd-recovery-prio/data 2024-09-18T15:01:29.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:29.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj46-p3 td/osd-recovery-prio/data 2024-09-18T15:01:29.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:29.642 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj47-p3 td/osd-recovery-prio/data 2024-09-18T15:01:30.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:30.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj48-p3 td/osd-recovery-prio/data 2024-09-18T15:01:30.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:30.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj49-p3 td/osd-recovery-prio/data 2024-09-18T15:01:30.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:30.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj50-p3 td/osd-recovery-prio/data 2024-09-18T15:01:31.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:31.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj51-p3 td/osd-recovery-prio/data 2024-09-18T15:01:31.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:31.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj52-p3 td/osd-recovery-prio/data 2024-09-18T15:01:31.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:31.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj53-p3 td/osd-recovery-prio/data 2024-09-18T15:01:32.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:32.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj54-p3 td/osd-recovery-prio/data 2024-09-18T15:01:32.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:32.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj55-p3 td/osd-recovery-prio/data 2024-09-18T15:01:32.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:32.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj56-p3 td/osd-recovery-prio/data 2024-09-18T15:01:33.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:33.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj57-p3 td/osd-recovery-prio/data 2024-09-18T15:01:33.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:33.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj58-p3 td/osd-recovery-prio/data 2024-09-18T15:01:33.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:33.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj59-p3 td/osd-recovery-prio/data 2024-09-18T15:01:34.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:34.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj60-p3 td/osd-recovery-prio/data 2024-09-18T15:01:34.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:34.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj61-p3 td/osd-recovery-prio/data 2024-09-18T15:01:35.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:35.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj62-p3 td/osd-recovery-prio/data 2024-09-18T15:01:35.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:35.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj63-p3 td/osd-recovery-prio/data 2024-09-18T15:01:35.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:35.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj64-p3 td/osd-recovery-prio/data 2024-09-18T15:01:36.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:36.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj65-p3 td/osd-recovery-prio/data 2024-09-18T15:01:36.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:36.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj66-p3 td/osd-recovery-prio/data 2024-09-18T15:01:37.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:37.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj67-p3 td/osd-recovery-prio/data 2024-09-18T15:01:37.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:37.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj68-p3 td/osd-recovery-prio/data 2024-09-18T15:01:37.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:37.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj69-p3 td/osd-recovery-prio/data 2024-09-18T15:01:38.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:38.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj70-p3 td/osd-recovery-prio/data 2024-09-18T15:01:38.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:38.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj71-p3 td/osd-recovery-prio/data 2024-09-18T15:01:38.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:38.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj72-p3 td/osd-recovery-prio/data 2024-09-18T15:01:39.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:39.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj73-p3 td/osd-recovery-prio/data 2024-09-18T15:01:39.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:39.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj74-p3 td/osd-recovery-prio/data 2024-09-18T15:01:39.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:39.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj75-p3 td/osd-recovery-prio/data 2024-09-18T15:01:40.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:40.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj76-p3 td/osd-recovery-prio/data 2024-09-18T15:01:40.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:40.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj77-p3 td/osd-recovery-prio/data 2024-09-18T15:01:41.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:41.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj78-p3 td/osd-recovery-prio/data 2024-09-18T15:01:41.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:41.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj79-p3 td/osd-recovery-prio/data 2024-09-18T15:01:41.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:41.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj80-p3 td/osd-recovery-prio/data 2024-09-18T15:01:42.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:42.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj81-p3 td/osd-recovery-prio/data 2024-09-18T15:01:42.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:42.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj82-p3 td/osd-recovery-prio/data 2024-09-18T15:01:42.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:42.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj83-p3 td/osd-recovery-prio/data 2024-09-18T15:01:43.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:43.212 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj84-p3 td/osd-recovery-prio/data 2024-09-18T15:01:43.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:43.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj85-p3 td/osd-recovery-prio/data 2024-09-18T15:01:43.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:43.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj86-p3 td/osd-recovery-prio/data 2024-09-18T15:01:44.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:44.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj87-p3 td/osd-recovery-prio/data 2024-09-18T15:01:44.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:44.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj88-p3 td/osd-recovery-prio/data 2024-09-18T15:01:45.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:45.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj89-p3 td/osd-recovery-prio/data 2024-09-18T15:01:45.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:45.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj90-p3 td/osd-recovery-prio/data 2024-09-18T15:01:45.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:45.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj91-p3 td/osd-recovery-prio/data 2024-09-18T15:01:46.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:46.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj92-p3 td/osd-recovery-prio/data 2024-09-18T15:01:46.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:46.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj93-p3 td/osd-recovery-prio/data 2024-09-18T15:01:47.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:47.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj94-p3 td/osd-recovery-prio/data 2024-09-18T15:01:47.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:47.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj95-p3 td/osd-recovery-prio/data 2024-09-18T15:01:47.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:47.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj96-p3 td/osd-recovery-prio/data 2024-09-18T15:01:48.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:48.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj97-p3 td/osd-recovery-prio/data 2024-09-18T15:01:48.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:48.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj98-p3 td/osd-recovery-prio/data 2024-09-18T15:01:48.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:48.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj99-p3 td/osd-recovery-prio/data 2024-09-18T15:01:49.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:49.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj100-p3 td/osd-recovery-prio/data 2024-09-18T15:01:49.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:49.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj101-p3 td/osd-recovery-prio/data 2024-09-18T15:01:50.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:50.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj102-p3 td/osd-recovery-prio/data 2024-09-18T15:01:50.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:50.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj103-p3 td/osd-recovery-prio/data 2024-09-18T15:01:50.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:50.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj104-p3 td/osd-recovery-prio/data 2024-09-18T15:01:51.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:51.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj105-p3 td/osd-recovery-prio/data 2024-09-18T15:01:51.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:51.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj106-p3 td/osd-recovery-prio/data 2024-09-18T15:01:51.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:51.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj107-p3 td/osd-recovery-prio/data 2024-09-18T15:01:52.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:52.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj108-p3 td/osd-recovery-prio/data 2024-09-18T15:01:52.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:52.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj109-p3 td/osd-recovery-prio/data 2024-09-18T15:01:53.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:53.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj110-p3 td/osd-recovery-prio/data 2024-09-18T15:01:53.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:53.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj111-p3 td/osd-recovery-prio/data 2024-09-18T15:01:53.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:53.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj112-p3 td/osd-recovery-prio/data 2024-09-18T15:01:54.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:54.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj113-p3 td/osd-recovery-prio/data 2024-09-18T15:01:54.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:54.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj114-p3 td/osd-recovery-prio/data 2024-09-18T15:01:54.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:54.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj115-p3 td/osd-recovery-prio/data 2024-09-18T15:01:55.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:55.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj116-p3 td/osd-recovery-prio/data 2024-09-18T15:01:55.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:55.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj117-p3 td/osd-recovery-prio/data 2024-09-18T15:01:55.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:55.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj118-p3 td/osd-recovery-prio/data 2024-09-18T15:01:56.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:56.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj119-p3 td/osd-recovery-prio/data 2024-09-18T15:01:56.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:56.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj120-p3 td/osd-recovery-prio/data 2024-09-18T15:01:57.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:57.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj121-p3 td/osd-recovery-prio/data 2024-09-18T15:01:57.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:57.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj122-p3 td/osd-recovery-prio/data 2024-09-18T15:01:57.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:57.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj123-p3 td/osd-recovery-prio/data 2024-09-18T15:01:58.096 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:58.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj124-p3 td/osd-recovery-prio/data 2024-09-18T15:01:58.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:58.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj125-p3 td/osd-recovery-prio/data 2024-09-18T15:01:58.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:58.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj126-p3 td/osd-recovery-prio/data 2024-09-18T15:01:59.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:59.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj127-p3 td/osd-recovery-prio/data 2024-09-18T15:01:59.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:59.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj128-p3 td/osd-recovery-prio/data 2024-09-18T15:01:59.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:01:59.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj129-p3 td/osd-recovery-prio/data 2024-09-18T15:02:00.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:00.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj130-p3 td/osd-recovery-prio/data 2024-09-18T15:02:00.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:00.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj131-p3 td/osd-recovery-prio/data 2024-09-18T15:02:01.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:01.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj132-p3 td/osd-recovery-prio/data 2024-09-18T15:02:01.420 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:01.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj133-p3 td/osd-recovery-prio/data 2024-09-18T15:02:01.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:01.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj134-p3 td/osd-recovery-prio/data 2024-09-18T15:02:02.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:02.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj135-p3 td/osd-recovery-prio/data 2024-09-18T15:02:02.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:02.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj136-p3 td/osd-recovery-prio/data 2024-09-18T15:02:02.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:02.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj137-p3 td/osd-recovery-prio/data 2024-09-18T15:02:03.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:03.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj138-p3 td/osd-recovery-prio/data 2024-09-18T15:02:03.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:03.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj139-p3 td/osd-recovery-prio/data 2024-09-18T15:02:03.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:03.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj140-p3 td/osd-recovery-prio/data 2024-09-18T15:02:04.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:04.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj141-p3 td/osd-recovery-prio/data 2024-09-18T15:02:04.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:04.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj142-p3 td/osd-recovery-prio/data 2024-09-18T15:02:05.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:05.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj143-p3 td/osd-recovery-prio/data 2024-09-18T15:02:05.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:05.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj144-p3 td/osd-recovery-prio/data 2024-09-18T15:02:05.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:05.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj145-p3 td/osd-recovery-prio/data 2024-09-18T15:02:06.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:06.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj146-p3 td/osd-recovery-prio/data 2024-09-18T15:02:06.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:06.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj147-p3 td/osd-recovery-prio/data 2024-09-18T15:02:06.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:06.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj148-p3 td/osd-recovery-prio/data 2024-09-18T15:02:07.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:07.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj149-p3 td/osd-recovery-prio/data 2024-09-18T15:02:07.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:07.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj150-p3 td/osd-recovery-prio/data 2024-09-18T15:02:07.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:07.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj151-p3 td/osd-recovery-prio/data 2024-09-18T15:02:08.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:08.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj152-p3 td/osd-recovery-prio/data 2024-09-18T15:02:08.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:08.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj153-p3 td/osd-recovery-prio/data 2024-09-18T15:02:09.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:09.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj154-p3 td/osd-recovery-prio/data 2024-09-18T15:02:09.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:09.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj155-p3 td/osd-recovery-prio/data 2024-09-18T15:02:09.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:09.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj156-p3 td/osd-recovery-prio/data 2024-09-18T15:02:10.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:10.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj157-p3 td/osd-recovery-prio/data 2024-09-18T15:02:10.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:10.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj158-p3 td/osd-recovery-prio/data 2024-09-18T15:02:11.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:11.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj159-p3 td/osd-recovery-prio/data 2024-09-18T15:02:11.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:11.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj160-p3 td/osd-recovery-prio/data 2024-09-18T15:02:11.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:11.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj161-p3 td/osd-recovery-prio/data 2024-09-18T15:02:12.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:12.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj162-p3 td/osd-recovery-prio/data 2024-09-18T15:02:12.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:12.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj163-p3 td/osd-recovery-prio/data 2024-09-18T15:02:12.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:12.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj164-p3 td/osd-recovery-prio/data 2024-09-18T15:02:13.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:13.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj165-p3 td/osd-recovery-prio/data 2024-09-18T15:02:13.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:13.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj166-p3 td/osd-recovery-prio/data 2024-09-18T15:02:13.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:13.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj167-p3 td/osd-recovery-prio/data 2024-09-18T15:02:14.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:14.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj168-p3 td/osd-recovery-prio/data 2024-09-18T15:02:14.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:14.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj169-p3 td/osd-recovery-prio/data 2024-09-18T15:02:15.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:15.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj170-p3 td/osd-recovery-prio/data 2024-09-18T15:02:15.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:15.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj171-p3 td/osd-recovery-prio/data 2024-09-18T15:02:15.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:15.760 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj172-p3 td/osd-recovery-prio/data 2024-09-18T15:02:16.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:16.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj173-p3 td/osd-recovery-prio/data 2024-09-18T15:02:16.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:16.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj174-p3 td/osd-recovery-prio/data 2024-09-18T15:02:16.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:16.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj175-p3 td/osd-recovery-prio/data 2024-09-18T15:02:17.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:17.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj176-p3 td/osd-recovery-prio/data 2024-09-18T15:02:17.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:17.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj177-p3 td/osd-recovery-prio/data 2024-09-18T15:02:18.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:18.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj178-p3 td/osd-recovery-prio/data 2024-09-18T15:02:18.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:18.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj179-p3 td/osd-recovery-prio/data 2024-09-18T15:02:19.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:19.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj180-p3 td/osd-recovery-prio/data 2024-09-18T15:02:19.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:19.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj181-p3 td/osd-recovery-prio/data 2024-09-18T15:02:19.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:19.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj182-p3 td/osd-recovery-prio/data 2024-09-18T15:02:20.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:20.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj183-p3 td/osd-recovery-prio/data 2024-09-18T15:02:20.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:20.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj184-p3 td/osd-recovery-prio/data 2024-09-18T15:02:20.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:20.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj185-p3 td/osd-recovery-prio/data 2024-09-18T15:02:21.231 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:21.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj186-p3 td/osd-recovery-prio/data 2024-09-18T15:02:21.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:21.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj187-p3 td/osd-recovery-prio/data 2024-09-18T15:02:21.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:21.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj188-p3 td/osd-recovery-prio/data 2024-09-18T15:02:22.285 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:22.285 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj189-p3 td/osd-recovery-prio/data 2024-09-18T15:02:22.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:22.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj190-p3 td/osd-recovery-prio/data 2024-09-18T15:02:23.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:23.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj191-p3 td/osd-recovery-prio/data 2024-09-18T15:02:23.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:23.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj192-p3 td/osd-recovery-prio/data 2024-09-18T15:02:23.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:23.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj193-p3 td/osd-recovery-prio/data 2024-09-18T15:02:24.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:24.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj194-p3 td/osd-recovery-prio/data 2024-09-18T15:02:24.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:24.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj195-p3 td/osd-recovery-prio/data 2024-09-18T15:02:24.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:24.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj196-p3 td/osd-recovery-prio/data 2024-09-18T15:02:25.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:25.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj197-p3 td/osd-recovery-prio/data 2024-09-18T15:02:25.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:25.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj198-p3 td/osd-recovery-prio/data 2024-09-18T15:02:25.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:25.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj199-p3 td/osd-recovery-prio/data 2024-09-18T15:02:26.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-18T15:02:26.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj200-p3 td/osd-recovery-prio/data 2024-09-18T15:02:26.672 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: expr 3 + 1 2024-09-18T15:02:26.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: p=4 2024-09-18T15:02:26.673 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:146: TEST_recovery_priority: get_not_primary test1 obj1-p1 2024-09-18T15:02:26.673 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test1 2024-09-18T15:02:26.673 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1-p1 2024-09-18T15:02:26.674 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test1 obj1-p1 2024-09-18T15:02:26.674 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test1 2024-09-18T15:02:26.674 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1-p1 2024-09-18T15:02:26.674 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test1 obj1-p1 2024-09-18T15:02:26.674 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:02:27.054 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-18T15:02:27.054 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test1 obj1-p1 2024-09-18T15:02:27.054 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-18T15:02:27.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:146: TEST_recovery_priority: local otherosd=0 2024-09-18T15:02:27.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:148: TEST_recovery_priority: ceph pg dump pgs 2024-09-18T15:02:27.805 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T15:02:27.805 INFO:tasks.workunit.client.0.smithi047.stdout:7.0 191 0 0 0 0 1996488704 0 0 571 0 571 active+clean 2024-09-18T14:57:38.955326+0000 94'571 94:621 [1] 1 [1] 1 0'0 2024-09-18T14:57:07.342435+0000 0'0 2024-09-18T14:57:07.342435+0000 0 0 periodic scrub scheduled @ 2024-09-19T18:41:18.000364+0000 0 0 2024-09-18T15:02:27.805 INFO:tasks.workunit.client.0.smithi047.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-09-18T14:57:38.300716+0000 94'600 94:665 [1] 1 [1] 1 0'0 2024-09-18T14:56:56.247413+0000 0'0 2024-09-18T14:56:56.247413+0000 0 0 periodic scrub scheduled @ 2024-09-20T00:29:11.351156+0000 0 0 2024-09-18T15:02:27.805 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-09-18T14:56:54.246811+0000 94'600 94:664 [1,0] 1 [1,0] 1 0'0 2024-09-18T14:56:50.949946+0000 0'0 2024-09-18T14:56:50.949946+0000 0 0 periodic scrub scheduled @ 2024-09-19T18:23:03.164056+0000 0 0 2024-09-18T15:02:27.806 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T15:02:27.806 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T15:02:27.806 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T15:02:27.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:149: TEST_recovery_priority: ERRORS=0 2024-09-18T15:02:27.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:151: TEST_recovery_priority: ceph osd set norecover 2024-09-18T15:02:28.608 INFO:tasks.workunit.client.0.smithi047.stderr:norecover is set 2024-09-18T15:02:28.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:152: TEST_recovery_priority: ceph osd set noout 2024-09-18T15:02:29.273 INFO:tasks.workunit.client.0.smithi047.stderr:noout is set 2024-09-18T15:02:29.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:156: TEST_recovery_priority: ceph osd pool set test7 size 2 2024-09-18T15:02:30.049 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 7 size to 2 2024-09-18T15:02:30.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:157: TEST_recovery_priority: sleep 2 2024-09-18T15:02:32.062 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: get_asok_path osd.1 2024-09-18T15:02:32.062 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:02:32.062 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:02:32.062 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:02:32.062 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:02:32.062 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215238 2024-09-18T15:02:32.062 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.215238/ceph-osd.1.asok 2024-09-18T15:02:32.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: CEPH_ARGS= 2024-09-18T15:02:32.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:02:32.147 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:02:32.147 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:02:32.147 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:02:32.147 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:02:32.148 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:02:32.148 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:02:32.148 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:02:32.148 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:02:32.148 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:02:32.148 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:02:32.148 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:02:32.148 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:02:32.148 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:02:32.148 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:02:32.148 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:02:32.148 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:02:32.148 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:02:32.148 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:02:32.148 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:02:32.149 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:02:32.155 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:161: TEST_recovery_priority: seq 1 10 2024-09-18T15:02:32.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:161: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2024-09-18T15:02:32.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: ceph pg force-recovery 7.0 2024-09-18T15:02:32.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2024-09-18T15:02:32.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:166: TEST_recovery_priority: '[' 1 = 10 ']' 2024-09-18T15:02:32.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:170: TEST_recovery_priority: sleep 2 2024-09-18T15:02:34.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:161: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2024-09-18T15:02:34.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: ceph pg force-recovery 7.0 2024-09-18T15:02:34.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2024-09-18T15:02:34.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:164: TEST_recovery_priority: break 2024-09-18T15:02:34.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:172: TEST_recovery_priority: flush_pg_stats 2024-09-18T15:02:34.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:02:34.899 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:02:35.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:02:35.306 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:02:35.306 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T15:02:35.306 INFO:tasks.workunit.client.0.smithi047.stderr:3 2024-09-18T15:02:35.306 INFO:tasks.workunit.client.0.smithi047.stderr:4' 2024-09-18T15:02:35.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:02:35.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:02:35.306 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:02:35.420 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836566 2024-09-18T15:02:35.420 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836566 2024-09-18T15:02:35.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836566' 2024-09-18T15:02:35.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:02:35.421 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:02:35.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673043 2024-09-18T15:02:35.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673043 2024-09-18T15:02:35.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836566 1-42949673043' 2024-09-18T15:02:35.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:02:35.651 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:02:35.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509522 2024-09-18T15:02:35.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509522 2024-09-18T15:02:35.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836566 1-42949673043 2-64424509522' 2024-09-18T15:02:35.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:02:35.793 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:02:35.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345998 2024-09-18T15:02:35.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345998 2024-09-18T15:02:35.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836566 1-42949673043 2-64424509522 3-85899345998' 2024-09-18T15:02:35.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:02:35.903 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:02:36.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182476 2024-09-18T15:02:36.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182476 2024-09-18T15:02:36.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836566 1-42949673043 2-64424509522 3-85899345998 4-107374182476' 2024-09-18T15:02:36.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:02:36.014 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836566 2024-09-18T15:02:36.014 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:02:36.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:02:36.015 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836566 2024-09-18T15:02:36.015 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:02:36.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836566 2024-09-18T15:02:36.016 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836566 2024-09-18T15:02:36.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836566' 2024-09-18T15:02:36.017 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:02:36.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836566 -lt 21474836566 2024-09-18T15:02:36.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:02:36.401 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673043 2024-09-18T15:02:36.401 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:02:36.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:02:36.403 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673043 2024-09-18T15:02:36.403 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:02:36.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673043 2024-09-18T15:02:36.404 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949673043 2024-09-18T15:02:36.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673043' 2024-09-18T15:02:36.404 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:02:36.780 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673043 -lt 42949673043 2024-09-18T15:02:36.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:02:36.781 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509522 2024-09-18T15:02:36.781 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:02:36.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:02:36.782 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509522 2024-09-18T15:02:36.782 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:02:36.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509522 2024-09-18T15:02:36.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509522' 2024-09-18T15:02:36.784 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509522 2024-09-18T15:02:36.784 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:02:37.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509522 -lt 64424509522 2024-09-18T15:02:37.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:02:37.166 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345998 2024-09-18T15:02:37.166 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:02:37.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:02:37.167 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345998 2024-09-18T15:02:37.167 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:02:37.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345998 2024-09-18T15:02:37.168 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 85899345998 2024-09-18T15:02:37.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345998' 2024-09-18T15:02:37.169 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:02:37.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345998 -lt 85899345998 2024-09-18T15:02:37.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:02:37.540 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182476 2024-09-18T15:02:37.540 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:02:37.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:02:37.541 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182476 2024-09-18T15:02:37.542 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:02:37.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182476 2024-09-18T15:02:37.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182476' 2024-09-18T15:02:37.543 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.4 seq 107374182476 2024-09-18T15:02:37.543 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:02:37.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182476 -lt 107374182476 2024-09-18T15:02:37.951 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:173: TEST_recovery_priority: get_asok_path osd.1 2024-09-18T15:02:37.951 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:02:37.951 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:02:37.951 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:02:37.951 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:02:37.951 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215238 2024-09-18T15:02:37.951 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.215238/ceph-osd.1.asok 2024-09-18T15:02:37.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:173: TEST_recovery_priority: CEPH_ARGS= 2024-09-18T15:02:37.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:173: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:02:38.026 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:02:38.026 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:02:38.027 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:02:38.027 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:02:38.027 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:02:38.027 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:02:38.027 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:02:38.027 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:02:38.027 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:02:38.027 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:02:38.027 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:02:38.027 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:02:38.027 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:02:38.027 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:02:38.027 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:02:38.027 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:02:38.028 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:02:38.028 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:02:38.028 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:02:38.028 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:02:38.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:175: TEST_recovery_priority: ceph osd out osd.0 2024-09-18T15:02:38.626 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already out. 2024-09-18T15:02:38.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:176: TEST_recovery_priority: sleep 2 2024-09-18T15:02:40.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:177: TEST_recovery_priority: flush_pg_stats 2024-09-18T15:02:40.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:02:40.639 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:02:41.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:02:41.016 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:02:41.016 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T15:02:41.016 INFO:tasks.workunit.client.0.smithi047.stderr:3 2024-09-18T15:02:41.016 INFO:tasks.workunit.client.0.smithi047.stderr:4' 2024-09-18T15:02:41.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:02:41.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:02:41.017 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:02:41.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836569 2024-09-18T15:02:41.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836569 2024-09-18T15:02:41.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836569' 2024-09-18T15:02:41.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:02:41.128 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:02:41.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673046 2024-09-18T15:02:41.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673046 2024-09-18T15:02:41.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836569 1-42949673046' 2024-09-18T15:02:41.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:02:41.262 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:02:41.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509525 2024-09-18T15:02:41.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509525 2024-09-18T15:02:41.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836569 1-42949673046 2-64424509525' 2024-09-18T15:02:41.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:02:41.373 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:02:41.483 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899346001 2024-09-18T15:02:41.483 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899346001 2024-09-18T15:02:41.483 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836569 1-42949673046 2-64424509525 3-85899346001' 2024-09-18T15:02:41.483 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:02:41.483 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:02:41.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182479 2024-09-18T15:02:41.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182479 2024-09-18T15:02:41.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836569 1-42949673046 2-64424509525 3-85899346001 4-107374182479' 2024-09-18T15:02:41.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:02:41.594 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836569 2024-09-18T15:02:41.594 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:02:41.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:02:41.595 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836569 2024-09-18T15:02:41.595 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:02:41.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836569 2024-09-18T15:02:41.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836569' 2024-09-18T15:02:41.597 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836569 2024-09-18T15:02:41.597 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:02:41.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 0 -lt 21474836569 2024-09-18T15:02:41.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:02:42.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:02:42.966 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:02:43.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836570 -lt 21474836569 2024-09-18T15:02:43.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:02:43.347 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673046 2024-09-18T15:02:43.347 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:02:43.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:02:43.348 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673046 2024-09-18T15:02:43.348 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:02:43.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673046 2024-09-18T15:02:43.349 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949673046 2024-09-18T15:02:43.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673046' 2024-09-18T15:02:43.350 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:02:43.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673046 -lt 42949673046 2024-09-18T15:02:43.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:02:43.722 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509525 2024-09-18T15:02:43.722 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:02:43.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:02:43.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509525 2024-09-18T15:02:43.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:02:43.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509525 2024-09-18T15:02:43.725 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509525 2024-09-18T15:02:43.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509525' 2024-09-18T15:02:43.726 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:02:44.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509525 -lt 64424509525 2024-09-18T15:02:44.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:02:44.092 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899346001 2024-09-18T15:02:44.092 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:02:44.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:02:44.093 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899346001 2024-09-18T15:02:44.093 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:02:44.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899346001 2024-09-18T15:02:44.094 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 85899346001 2024-09-18T15:02:44.095 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899346001' 2024-09-18T15:02:44.095 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:02:44.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899346002 -lt 85899346001 2024-09-18T15:02:44.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:02:44.522 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182479 2024-09-18T15:02:44.522 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:02:44.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:02:44.523 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182479 2024-09-18T15:02:44.524 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:02:44.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182479 2024-09-18T15:02:44.525 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.4 seq 107374182479 2024-09-18T15:02:44.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182479' 2024-09-18T15:02:44.525 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:02:44.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182479 -lt 107374182479 2024-09-18T15:02:44.923 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:178: TEST_recovery_priority: get_asok_path osd.1 2024-09-18T15:02:44.923 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:02:44.924 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:02:44.924 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:02:44.924 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:02:44.924 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215238 2024-09-18T15:02:44.924 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.215238/ceph-osd.1.asok 2024-09-18T15:02:44.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:178: TEST_recovery_priority: CEPH_ARGS= 2024-09-18T15:02:44.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:178: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:02:44.999 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:02:44.999 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:02:45.000 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:02:45.000 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:02:45.000 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:02:45.000 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:02:45.000 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:02:45.000 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:02:45.000 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:02:45.000 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:02:45.000 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:02:45.000 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:02:45.000 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:02:45.000 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:02:45.000 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:02:45.001 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:02:45.001 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:02:45.001 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:02:45.001 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:02:45.001 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:02:45.001 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:02:45.001 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:02:45.001 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:02:45.001 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:02:45.001 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:02:45.001 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:02:45.001 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:02:45.001 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:02:45.001 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:02:45.002 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:02:45.002 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:02:45.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:179: TEST_recovery_priority: ceph pg dump pgs 2024-09-18T15:02:45.354 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T15:02:45.354 INFO:tasks.workunit.client.0.smithi047.stdout:7.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovering+forced_recovery+undersized+degraded+remapped 2024-09-18T15:02:34.787587+0000 94'600 105:670 [1,4] 1 [1] 1 0'0 2024-09-18T14:57:07.342435+0000 0'0 2024-09-18T14:57:07.342435+0000 0 0 periodic scrub scheduled @ 2024-09-20T00:00:03.561008+0000 0 0 2024-09-18T15:02:45.354 INFO:tasks.workunit.client.0.smithi047.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-09-18T14:57:38.300716+0000 94'600 105:674 [1] 1 [1] 1 0'0 2024-09-18T14:56:56.247413+0000 0'0 2024-09-18T14:56:56.247413+0000 0 0 periodic scrub scheduled @ 2024-09-20T00:29:11.351156+0000 0 0 2024-09-18T15:02:45.354 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovery_wait+undersized+degraded+remapped 2024-09-18T15:02:40.508138+0000 94'600 106:682 [1,2] 1 [1] 1 0'0 2024-09-18T14:56:50.949946+0000 0'0 2024-09-18T14:56:50.949946+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:32:47.739223+0000 0 0 2024-09-18T15:02:45.354 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T15:02:45.354 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T15:02:45.354 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T15:02:45.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:181: TEST_recovery_priority: ceph osd pool set test3 size 2 2024-09-18T15:02:46.023 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 3 size to 2 2024-09-18T15:02:46.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:182: TEST_recovery_priority: sleep 2 2024-09-18T15:02:48.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:183: TEST_recovery_priority: flush_pg_stats 2024-09-18T15:02:48.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:02:48.037 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:02:48.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:02:48.561 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:02:48.562 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T15:02:48.562 INFO:tasks.workunit.client.0.smithi047.stderr:3 2024-09-18T15:02:48.562 INFO:tasks.workunit.client.0.smithi047.stderr:4' 2024-09-18T15:02:48.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:02:48.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:02:48.562 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:02:48.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836573 2024-09-18T15:02:48.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836573 2024-09-18T15:02:48.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836573' 2024-09-18T15:02:48.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:02:48.564 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:02:48.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673050 2024-09-18T15:02:48.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673050 2024-09-18T15:02:48.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836573 1-42949673050' 2024-09-18T15:02:48.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:02:48.735 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:02:48.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509528 2024-09-18T15:02:48.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509528 2024-09-18T15:02:48.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836573 1-42949673050 2-64424509528' 2024-09-18T15:02:48.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:02:48.846 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:02:48.956 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899346005 2024-09-18T15:02:48.956 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899346005 2024-09-18T15:02:48.956 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836573 1-42949673050 2-64424509528 3-85899346005' 2024-09-18T15:02:48.956 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:02:48.956 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:02:49.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182482 2024-09-18T15:02:49.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182482 2024-09-18T15:02:49.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836573 1-42949673050 2-64424509528 3-85899346005 4-107374182482' 2024-09-18T15:02:49.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:02:49.067 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836573 2024-09-18T15:02:49.067 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:02:49.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:02:49.068 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836573 2024-09-18T15:02:49.068 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:02:49.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836573 2024-09-18T15:02:49.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836573' 2024-09-18T15:02:49.069 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836573 2024-09-18T15:02:49.069 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:02:49.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836571 -lt 21474836573 2024-09-18T15:02:49.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:02:50.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:02:50.437 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:02:50.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836573 -lt 21474836573 2024-09-18T15:02:50.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:02:50.825 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673050 2024-09-18T15:02:50.825 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:02:50.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:02:50.826 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673050 2024-09-18T15:02:50.826 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:02:50.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673050 2024-09-18T15:02:50.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673050' 2024-09-18T15:02:50.827 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949673050 2024-09-18T15:02:50.828 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:02:51.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673050 -lt 42949673050 2024-09-18T15:02:51.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:02:51.260 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509528 2024-09-18T15:02:51.260 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:02:51.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:02:51.262 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509528 2024-09-18T15:02:51.262 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:02:51.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509528 2024-09-18T15:02:51.263 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509528 2024-09-18T15:02:51.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509528' 2024-09-18T15:02:51.264 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:02:51.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509528 -lt 64424509528 2024-09-18T15:02:51.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:02:51.629 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899346005 2024-09-18T15:02:51.629 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:02:51.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:02:51.630 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899346005 2024-09-18T15:02:51.630 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:02:51.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899346005 2024-09-18T15:02:51.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899346005' 2024-09-18T15:02:51.632 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 85899346005 2024-09-18T15:02:51.632 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:02:52.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899346005 -lt 85899346005 2024-09-18T15:02:52.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:02:52.006 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182482 2024-09-18T15:02:52.006 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:02:52.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:02:52.007 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182482 2024-09-18T15:02:52.008 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:02:52.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182482 2024-09-18T15:02:52.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182482' 2024-09-18T15:02:52.009 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.4 seq 107374182482 2024-09-18T15:02:52.009 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:02:52.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182483 -lt 107374182482 2024-09-18T15:02:52.379 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:184: TEST_recovery_priority: get_asok_path osd.1 2024-09-18T15:02:52.379 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:02:52.379 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:02:52.379 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:02:52.379 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:02:52.379 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215238 2024-09-18T15:02:52.379 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.215238/ceph-osd.1.asok 2024-09-18T15:02:52.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:184: TEST_recovery_priority: CEPH_ARGS= 2024-09-18T15:02:52.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:184: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:02:52.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:185: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-18T15:02:52.463 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:02:52.464 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:02:52.464 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:02:52.464 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:02:52.464 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:02:52.464 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:02:52.464 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:02:52.464 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:02:52.465 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:02:52.465 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:02:52.465 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:02:52.465 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:02:52.465 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:02:52.465 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:02:52.465 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:02:52.465 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:02:52.465 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:02:52.465 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:02:52.465 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:02:52.465 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:02:52.465 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:02:52.466 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:02:52.466 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:02:52.466 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:02:52.466 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:02:52.466 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:02:52.466 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:02:52.466 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:02:52.467 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:02:52.467 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:02:52.467 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:02:52.467 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:02:52.467 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:02:52.468 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:02:52.468 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:02:52.468 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:02:52.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:186: TEST_recovery_priority: ceph pg dump pgs 2024-09-18T15:02:52.841 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T15:02:52.841 INFO:tasks.workunit.client.0.smithi047.stdout:7.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovering+forced_recovery+undersized+degraded+remapped 2024-09-18T15:02:34.787587+0000 94'600 109:674 [1,4] 1 [1] 1 0'0 2024-09-18T14:57:07.342435+0000 0'0 2024-09-18T14:57:07.342435+0000 0 0 periodic scrub scheduled @ 2024-09-20T00:00:03.561008+0000 0 0 2024-09-18T15:02:52.841 INFO:tasks.workunit.client.0.smithi047.stdout:3.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovery_wait+undersized+degraded+remapped 2024-09-18T15:02:48.006852+0000 94'600 110:687 [1,2] 1 [1] 1 0'0 2024-09-18T14:56:56.247413+0000 0'0 2024-09-18T14:56:56.247413+0000 0 0 periodic scrub scheduled @ 2024-09-19T18:33:24.604369+0000 0 0 2024-09-18T15:02:52.841 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovery_wait+undersized+degraded+remapped 2024-09-18T15:02:40.508138+0000 94'600 109:686 [1,2] 1 [1] 1 0'0 2024-09-18T14:56:50.949946+0000 0'0 2024-09-18T14:56:50.949946+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:32:47.739223+0000 0 0 2024-09-18T15:02:52.841 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T15:02:52.841 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T15:02:52.841 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T15:02:52.852 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:188: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-18T15:02:52.852 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:188: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "1.0")).prio' 2024-09-18T15:02:52.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:188: TEST_recovery_priority: PRIO=190 2024-09-18T15:02:52.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:189: TEST_recovery_priority: '[' 190 '!=' 190 ']' 2024-09-18T15:02:52.855 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:196: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-18T15:02:52.855 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:196: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2024-09-18T15:02:52.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:196: TEST_recovery_priority: eval 'ITEM="7.0"' 2024-09-18T15:02:52.858 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:196: TEST_recovery_priority: ITEM=7.0 2024-09-18T15:02:52.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:197: TEST_recovery_priority: '[' 7.0 '!=' 7.0 ']' 2024-09-18T15:02:52.858 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:202: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-18T15:02:52.858 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:202: TEST_recovery_priority: jq '.local_reservations.in_progress[0].prio' 2024-09-18T15:02:52.861 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:202: TEST_recovery_priority: PRIO=255 2024-09-18T15:02:52.861 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:203: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2024-09-18T15:02:52.861 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:211: TEST_recovery_priority: seq 1 10 2024-09-18T15:02:52.861 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:211: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2024-09-18T15:02:52.862 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:213: TEST_recovery_priority: ceph pg force-recovery 3.0 2024-09-18T15:02:52.862 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:213: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2024-09-18T15:02:53.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:214: TEST_recovery_priority: break 2024-09-18T15:02:53.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:222: TEST_recovery_priority: sleep 2 2024-09-18T15:02:55.226 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:223: TEST_recovery_priority: get_asok_path osd.1 2024-09-18T15:02:55.227 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:02:55.227 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:02:55.227 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:02:55.227 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:02:55.227 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215238 2024-09-18T15:02:55.227 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.215238/ceph-osd.1.asok 2024-09-18T15:02:55.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:223: TEST_recovery_priority: CEPH_ARGS= 2024-09-18T15:02:55.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:223: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:02:55.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:224: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-18T15:02:55.321 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:02:55.322 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:02:55.322 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:02:55.322 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:02:55.322 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:02:55.322 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:02:55.323 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:02:55.323 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:02:55.323 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:02:55.323 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:02:55.323 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:02:55.323 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:02:55.324 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:02:55.324 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:02:55.324 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:02:55.324 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:02:55.324 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 255, 2024-09-18T15:02:55.324 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:02:55.324 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:02:55.324 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:02:55.324 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:02:55.325 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:02:55.325 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:02:55.325 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:02:55.325 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:02:55.325 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:02:55.325 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:02:55.326 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:02:55.326 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:02:55.326 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:02:55.326 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:02:55.326 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:02:55.326 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:02:55.326 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:02:55.326 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:02:55.326 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:02:55.326 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:02:55.326 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:02:55.326 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:02:55.326 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:02:55.326 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:02:55.328 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:225: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-18T15:02:55.328 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:225: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2024-09-18T15:02:55.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:225: TEST_recovery_priority: PRIO=255 2024-09-18T15:02:55.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:226: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2024-09-18T15:02:55.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:231: TEST_recovery_priority: flush_pg_stats 2024-09-18T15:02:55.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:02:55.329 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:02:55.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:02:55.693 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:02:55.694 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T15:02:55.694 INFO:tasks.workunit.client.0.smithi047.stderr:3 2024-09-18T15:02:55.694 INFO:tasks.workunit.client.0.smithi047.stderr:4' 2024-09-18T15:02:55.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:02:55.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:02:55.694 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:02:55.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836576 2024-09-18T15:02:55.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836576 2024-09-18T15:02:55.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836576' 2024-09-18T15:02:55.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:02:55.805 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:02:55.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673053 2024-09-18T15:02:55.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673053 2024-09-18T15:02:55.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836576 1-42949673053' 2024-09-18T15:02:55.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:02:55.942 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:02:56.052 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509532 2024-09-18T15:02:56.052 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509532 2024-09-18T15:02:56.052 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836576 1-42949673053 2-64424509532' 2024-09-18T15:02:56.052 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:02:56.052 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:02:56.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899346008 2024-09-18T15:02:56.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899346008 2024-09-18T15:02:56.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836576 1-42949673053 2-64424509532 3-85899346008' 2024-09-18T15:02:56.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:02:56.162 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:02:56.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182486 2024-09-18T15:02:56.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182486 2024-09-18T15:02:56.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836576 1-42949673053 2-64424509532 3-85899346008 4-107374182486' 2024-09-18T15:02:56.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:02:56.274 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836576 2024-09-18T15:02:56.274 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:02:56.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:02:56.275 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836576 2024-09-18T15:02:56.275 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:02:56.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836576 2024-09-18T15:02:56.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836576' 2024-09-18T15:02:56.276 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836576 2024-09-18T15:02:56.277 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:02:56.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836576 -lt 21474836576 2024-09-18T15:02:56.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:02:56.678 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673053 2024-09-18T15:02:56.678 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:02:56.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:02:56.679 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673053 2024-09-18T15:02:56.679 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:02:56.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673053 2024-09-18T15:02:56.680 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949673053 2024-09-18T15:02:56.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673053' 2024-09-18T15:02:56.681 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:02:57.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673053 -lt 42949673053 2024-09-18T15:02:57.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:02:57.087 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509532 2024-09-18T15:02:57.087 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:02:57.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:02:57.088 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509532 2024-09-18T15:02:57.089 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:02:57.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509532 2024-09-18T15:02:57.090 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509532 2024-09-18T15:02:57.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509532' 2024-09-18T15:02:57.090 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:02:57.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509532 -lt 64424509532 2024-09-18T15:02:57.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:02:57.462 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899346008 2024-09-18T15:02:57.462 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:02:57.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:02:57.463 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899346008 2024-09-18T15:02:57.463 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:02:57.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899346008 2024-09-18T15:02:57.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899346008' 2024-09-18T15:02:57.465 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 85899346008 2024-09-18T15:02:57.465 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:02:57.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899346008 -lt 85899346008 2024-09-18T15:02:57.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:02:57.834 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182486 2024-09-18T15:02:57.835 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:02:57.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:02:57.836 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182486 2024-09-18T15:02:57.836 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:02:57.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182486 2024-09-18T15:02:57.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182486' 2024-09-18T15:02:57.837 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.4 seq 107374182486 2024-09-18T15:02:57.837 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:02:58.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182486 -lt 107374182486 2024-09-18T15:02:58.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:235: TEST_recovery_priority: ceph pg cancel-force-recovery 7.0 2024-09-18T15:02:58.598 INFO:tasks.workunit.client.0.smithi047.stderr:instructing pg(s) [7.0] on osd.1 to cancel-force-recovery; 2024-09-18T15:02:58.599 INFO:tasks.workunit.client.0.smithi047.stderr: 2024-09-18T15:02:58.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:236: TEST_recovery_priority: sleep 2 2024-09-18T15:03:00.611 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:238: TEST_recovery_priority: get_asok_path osd.1 2024-09-18T15:03:00.612 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:03:00.612 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:03:00.612 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:03:00.612 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:03:00.612 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215238 2024-09-18T15:03:00.612 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.215238/ceph-osd.1.asok 2024-09-18T15:03:00.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:238: TEST_recovery_priority: CEPH_ARGS= 2024-09-18T15:03:00.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:238: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:00.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:239: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-18T15:03:00.706 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:03:00.706 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:03:00.706 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:00.706 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:00.707 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:03:00.707 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:00.707 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:03:00.707 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:03:00.707 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:00.707 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:03:00.707 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:00.707 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:00.707 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:00.707 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:00.707 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:03:00.707 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:00.708 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:00.708 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:00.708 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:00.708 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:00.708 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:03:00.708 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:03:00.708 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:00.708 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:03:00.708 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:03:00.708 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:00.708 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:00.708 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:00.708 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:00.708 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:03:00.709 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:00.709 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:00.709 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:03:00.709 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:03:00.709 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:00.709 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:03:00.709 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:240: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-18T15:03:00.709 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:240: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "7.0")).prio' 2024-09-18T15:03:00.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:240: TEST_recovery_priority: PRIO=190 2024-09-18T15:03:00.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:241: TEST_recovery_priority: '[' 190 '!=' 190 ']' 2024-09-18T15:03:00.711 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:247: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-18T15:03:00.711 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:247: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2024-09-18T15:03:00.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:247: TEST_recovery_priority: eval 'ITEM="3.0"' 2024-09-18T15:03:00.713 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:247: TEST_recovery_priority: ITEM=3.0 2024-09-18T15:03:00.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:248: TEST_recovery_priority: '[' 3.0 '!=' 3.0 ']' 2024-09-18T15:03:00.714 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:253: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-18T15:03:00.714 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:253: TEST_recovery_priority: jq '.local_reservations.in_progress[0].prio' 2024-09-18T15:03:00.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:253: TEST_recovery_priority: PRIO=255 2024-09-18T15:03:00.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:254: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2024-09-18T15:03:00.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:261: TEST_recovery_priority: ceph pg cancel-force-recovery 3.0 2024-09-18T15:03:01.069 INFO:tasks.workunit.client.0.smithi047.stderr:instructing pg(s) [3.0] on osd.1 to cancel-force-recovery; 2024-09-18T15:03:01.069 INFO:tasks.workunit.client.0.smithi047.stderr: 2024-09-18T15:03:01.079 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:262: TEST_recovery_priority: sleep 5 2024-09-18T15:03:06.082 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:263: TEST_recovery_priority: get_asok_path osd.1 2024-09-18T15:03:06.082 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:03:06.082 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:03:06.082 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:03:06.082 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:03:06.082 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215238 2024-09-18T15:03:06.083 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.215238/ceph-osd.1.asok 2024-09-18T15:03:06.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:263: TEST_recovery_priority: CEPH_ARGS= 2024-09-18T15:03:06.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:263: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:06.168 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:03:06.168 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:03:06.168 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:06.168 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:06.168 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:03:06.168 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:06.169 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:03:06.169 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:03:06.169 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:06.169 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:03:06.169 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:06.169 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:06.169 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:06.169 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:06.169 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:03:06.169 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:06.169 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:06.169 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:06.170 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:06.170 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:06.170 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:03:06.170 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:03:06.170 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:06.170 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:03:06.170 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:06.170 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:06.170 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:06.170 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:06.170 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:06.171 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:03:06.171 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:06.171 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:06.171 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:03:06.171 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:03:06.171 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:06.171 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:03:06.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:266: TEST_recovery_priority: flush_pg_stats 2024-09-18T15:03:06.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:03:06.176 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:03:06.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:03:06.572 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:03:06.572 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T15:03:06.572 INFO:tasks.workunit.client.0.smithi047.stderr:3 2024-09-18T15:03:06.573 INFO:tasks.workunit.client.0.smithi047.stderr:4' 2024-09-18T15:03:06.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:03:06.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:03:06.573 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:03:06.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836581 2024-09-18T15:03:06.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836581 2024-09-18T15:03:06.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836581' 2024-09-18T15:03:06.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:03:06.683 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:03:06.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673057 2024-09-18T15:03:06.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673057 2024-09-18T15:03:06.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836581 1-42949673057' 2024-09-18T15:03:06.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:03:06.793 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:03:06.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509536 2024-09-18T15:03:06.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509536 2024-09-18T15:03:06.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836581 1-42949673057 2-64424509536' 2024-09-18T15:03:06.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:03:06.904 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:03:07.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899346012 2024-09-18T15:03:07.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899346012 2024-09-18T15:03:07.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836581 1-42949673057 2-64424509536 3-85899346012' 2024-09-18T15:03:07.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:03:07.015 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:03:07.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182490 2024-09-18T15:03:07.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182490 2024-09-18T15:03:07.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836581 1-42949673057 2-64424509536 3-85899346012 4-107374182490' 2024-09-18T15:03:07.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:03:07.126 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836581 2024-09-18T15:03:07.126 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:03:07.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:03:07.127 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836581 2024-09-18T15:03:07.127 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:03:07.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836581 2024-09-18T15:03:07.128 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836581 2024-09-18T15:03:07.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836581' 2024-09-18T15:03:07.129 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:03:07.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836578 -lt 21474836581 2024-09-18T15:03:07.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:03:08.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:03:08.498 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:03:08.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836581 -lt 21474836581 2024-09-18T15:03:08.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:03:08.894 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673057 2024-09-18T15:03:08.894 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:03:08.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:03:08.895 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673057 2024-09-18T15:03:08.895 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:03:08.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673057 2024-09-18T15:03:08.897 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949673057 2024-09-18T15:03:08.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673057' 2024-09-18T15:03:08.897 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:03:09.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673057 -lt 42949673057 2024-09-18T15:03:09.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:03:09.311 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509536 2024-09-18T15:03:09.311 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:03:09.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:03:09.312 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509536 2024-09-18T15:03:09.312 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:03:09.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509536 2024-09-18T15:03:09.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509536' 2024-09-18T15:03:09.314 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509536 2024-09-18T15:03:09.314 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:03:09.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509536 -lt 64424509536 2024-09-18T15:03:09.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:03:09.682 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899346012 2024-09-18T15:03:09.682 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:03:09.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:03:09.683 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899346012 2024-09-18T15:03:09.683 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:03:09.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899346012 2024-09-18T15:03:09.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899346012' 2024-09-18T15:03:09.685 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 85899346012 2024-09-18T15:03:09.685 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:03:10.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899346013 -lt 85899346012 2024-09-18T15:03:10.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:03:10.064 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182490 2024-09-18T15:03:10.064 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:03:10.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:03:10.065 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182490 2024-09-18T15:03:10.065 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:03:10.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182490 2024-09-18T15:03:10.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182490' 2024-09-18T15:03:10.066 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.4 seq 107374182490 2024-09-18T15:03:10.066 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:03:10.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182491 -lt 107374182490 2024-09-18T15:03:10.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:267: TEST_recovery_priority: ceph pg force-recovery 7.0 2024-09-18T15:03:10.854 INFO:tasks.workunit.client.0.smithi047.stderr:instructing pg(s) [7.0] on osd.1 to force-recovery; 2024-09-18T15:03:10.854 INFO:tasks.workunit.client.0.smithi047.stderr: 2024-09-18T15:03:10.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:268: TEST_recovery_priority: sleep 2 2024-09-18T15:03:12.868 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:270: TEST_recovery_priority: get_asok_path osd.1 2024-09-18T15:03:12.868 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:03:12.868 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:03:12.868 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:03:12.868 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:03:12.868 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215238 2024-09-18T15:03:12.868 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.215238/ceph-osd.1.asok 2024-09-18T15:03:12.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:270: TEST_recovery_priority: CEPH_ARGS= 2024-09-18T15:03:12.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:270: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:12.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:271: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-18T15:03:12.962 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:03:12.962 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:03:12.962 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:12.962 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:12.962 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:03:12.962 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:12.962 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:03:12.962 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:03:12.962 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:12.962 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:03:12.962 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:12.962 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:12.963 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:12.963 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:12.963 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:03:12.963 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:12.963 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:12.963 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:12.963 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:12.963 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:12.963 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:03:12.963 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:03:12.963 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:12.963 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:03:12.963 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:03:12.963 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:12.963 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:12.964 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:12.964 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:12.964 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:03:12.964 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:12.964 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:12.964 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:03:12.964 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:03:12.964 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:12.964 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:03:12.964 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:272: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-18T15:03:12.965 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:272: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2024-09-18T15:03:12.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:272: TEST_recovery_priority: PRIO=190 2024-09-18T15:03:12.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:273: TEST_recovery_priority: '[' 190 '!=' 190 ']' 2024-09-18T15:03:12.966 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:279: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-18T15:03:12.966 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:279: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2024-09-18T15:03:12.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:279: TEST_recovery_priority: eval 'ITEM="7.0"' 2024-09-18T15:03:12.969 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:279: TEST_recovery_priority: ITEM=7.0 2024-09-18T15:03:12.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:280: TEST_recovery_priority: '[' 7.0 '!=' 7.0 ']' 2024-09-18T15:03:12.969 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:285: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-18T15:03:12.969 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:285: TEST_recovery_priority: jq '.local_reservations.in_progress[0].prio' 2024-09-18T15:03:12.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:285: TEST_recovery_priority: PRIO=255 2024-09-18T15:03:12.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:286: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2024-09-18T15:03:12.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:293: TEST_recovery_priority: ceph osd unset noout 2024-09-18T15:03:13.641 INFO:tasks.workunit.client.0.smithi047.stderr:noout is unset 2024-09-18T15:03:13.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:294: TEST_recovery_priority: ceph osd unset norecover 2024-09-18T15:03:14.646 INFO:tasks.workunit.client.0.smithi047.stderr:norecover is unset 2024-09-18T15:03:14.659 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:296: TEST_recovery_priority: get_asok_path osd.1 2024-09-18T15:03:14.659 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:03:14.659 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:03:14.659 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:03:14.659 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:03:14.659 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215238 2024-09-18T15:03:14.659 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.215238/ceph-osd.1.asok 2024-09-18T15:03:14.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:296: TEST_recovery_priority: wait_for_clean 'CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations' 2024-09-18T15:03:14.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local 'cmd=CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations' 2024-09-18T15:03:14.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:03:14.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:03:14.660 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:03:14.660 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:03:14.660 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:03:14.661 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:03:14.661 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:03:14.661 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:03:14.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:03:14.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:03:14.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:03:14.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:03:14.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:03:14.751 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:03:15.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:03:15.151 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:03:15.151 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T15:03:15.151 INFO:tasks.workunit.client.0.smithi047.stderr:3 2024-09-18T15:03:15.151 INFO:tasks.workunit.client.0.smithi047.stderr:4' 2024-09-18T15:03:15.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:03:15.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:03:15.152 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:03:15.264 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836584 2024-09-18T15:03:15.264 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836584 2024-09-18T15:03:15.265 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836584' 2024-09-18T15:03:15.265 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:03:15.265 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:03:15.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673061 2024-09-18T15:03:15.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673061 2024-09-18T15:03:15.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836584 1-42949673061' 2024-09-18T15:03:15.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:03:15.426 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:03:15.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509540 2024-09-18T15:03:15.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509540 2024-09-18T15:03:15.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836584 1-42949673061 2-64424509540' 2024-09-18T15:03:15.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:03:15.537 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:03:15.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899346016 2024-09-18T15:03:15.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899346016 2024-09-18T15:03:15.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836584 1-42949673061 2-64424509540 3-85899346016' 2024-09-18T15:03:15.648 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:03:15.648 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:03:15.761 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182494 2024-09-18T15:03:15.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182494 2024-09-18T15:03:15.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836584 1-42949673061 2-64424509540 3-85899346016 4-107374182494' 2024-09-18T15:03:15.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:03:15.762 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836584 2024-09-18T15:03:15.762 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:03:15.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:03:15.763 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836584 2024-09-18T15:03:15.763 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:03:15.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836584 2024-09-18T15:03:15.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836584' 2024-09-18T15:03:15.765 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836584 2024-09-18T15:03:15.765 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:03:16.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836582 -lt 21474836584 2024-09-18T15:03:16.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:03:17.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:03:17.200 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:03:17.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836584 -lt 21474836584 2024-09-18T15:03:17.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:03:17.587 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673061 2024-09-18T15:03:17.587 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:03:17.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:03:17.588 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673061 2024-09-18T15:03:17.589 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:03:17.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673061 2024-09-18T15:03:17.590 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949673061 2024-09-18T15:03:17.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673061' 2024-09-18T15:03:17.590 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:03:18.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673061 -lt 42949673061 2024-09-18T15:03:18.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:03:18.004 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509540 2024-09-18T15:03:18.005 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:03:18.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:03:18.006 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509540 2024-09-18T15:03:18.006 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:03:18.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509540 2024-09-18T15:03:18.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509540' 2024-09-18T15:03:18.007 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509540 2024-09-18T15:03:18.008 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:03:18.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509540 -lt 64424509540 2024-09-18T15:03:18.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:03:18.429 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899346016 2024-09-18T15:03:18.429 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:03:18.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:03:18.431 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899346016 2024-09-18T15:03:18.431 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:03:18.432 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899346016 2024-09-18T15:03:18.432 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899346016' 2024-09-18T15:03:18.432 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 85899346016 2024-09-18T15:03:18.432 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:03:18.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899346017 -lt 85899346016 2024-09-18T15:03:18.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:03:18.998 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182494 2024-09-18T15:03:18.999 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:03:18.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:03:19.000 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182494 2024-09-18T15:03:19.000 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:03:19.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182494 2024-09-18T15:03:19.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182494' 2024-09-18T15:03:19.001 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.4 seq 107374182494 2024-09-18T15:03:19.001 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:03:19.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182494 -lt 107374182494 2024-09-18T15:03:19.525 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:03:19.526 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:03:19.526 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:03:20.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 == 0 2024-09-18T15:03:20.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:03:20.223 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:03:20.223 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:03:20.223 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:03:20.223 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:03:20.223 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:03:20.223 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:03:20.612 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:03:20.613 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:03:20.613 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:03:20.613 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:03:21.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:03:21.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-18T15:03:21.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T15:03:21.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-18T15:03:21.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:21.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:03:21.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:21.349 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:03:21.349 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:03:21.349 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:21.349 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:21.349 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:03:21.349 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:21.349 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:03:21.349 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:03:21.349 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:21.349 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:03:21.349 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:21.350 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:21.350 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:21.350 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:21.350 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:03:21.350 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:21.350 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:21.350 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:21.350 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:21.350 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:21.350 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:03:21.350 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:03:21.350 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:21.350 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:03:21.350 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:03:21.351 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:21.351 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:21.351 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:21.351 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:21.351 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:03:21.351 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:21.351 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:21.351 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:03:21.351 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:03:21.351 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:21.351 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:03:21.355 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:03:21.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:03:21.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:03:21.458 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:03:21.458 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:03:21.458 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:03:21.458 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:03:21.458 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:03:21.458 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:03:21.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:03:21.818 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:03:21.819 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:03:21.819 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:03:22.406 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:03:22.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:03:22.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:03:22.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:03:22.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:03:22.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:03:22.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:03:22.407 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:03:22.407 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:03:22.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6552544 2024-09-18T15:03:22.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6552544 '!=' null 2024-09-18T15:03:22.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:03:22.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:22.983 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:03:22.983 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:23.058 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:03:23.058 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:03:23.058 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:23.058 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:23.058 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:03:23.058 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:23.058 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:03:23.058 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:03:23.059 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:23.059 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:03:23.059 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:23.059 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:23.059 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:23.059 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:23.059 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:03:23.059 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:23.059 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:23.059 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:23.059 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:23.059 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:23.059 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:03:23.059 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:03:23.059 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:23.060 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:03:23.060 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:03:23.060 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:23.060 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:23.060 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:23.060 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:23.060 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:03:23.060 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:23.060 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:23.060 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:03:23.060 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:03:23.060 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:23.060 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:03:23.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:03:23.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:03:23.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:03:23.167 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:03:23.167 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:03:23.167 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:03:23.167 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:03:23.167 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:03:23.167 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:03:23.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:03:23.530 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:03:23.531 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:03:23.531 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:03:24.155 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:03:24.155 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:03:24.155 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:03:24.155 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:03:24.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:03:24.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:03:24.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:03:24.156 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:03:24.156 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:03:24.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14808455 2024-09-18T15:03:24.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14808455 '!=' null 2024-09-18T15:03:24.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:03:24.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:24.748 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:03:24.748 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:24.827 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:03:24.827 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:03:24.827 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:24.827 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:24.827 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:03:24.827 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:24.828 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:03:24.828 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:03:24.828 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:24.828 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:03:24.828 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:24.828 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:24.828 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:24.828 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:24.828 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:03:24.828 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:24.828 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:24.828 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:24.828 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:24.828 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:24.829 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:03:24.829 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:03:24.829 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:24.829 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:03:24.829 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:03:24.829 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:24.829 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:24.829 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:24.829 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:24.829 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:03:24.829 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:24.829 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:24.829 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:03:24.830 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:03:24.830 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:24.830 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:03:24.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:03:24.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:03:24.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:03:24.936 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:03:24.936 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:03:24.936 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:03:24.936 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:03:24.936 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:03:24.936 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:03:25.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:03:25.315 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:03:25.316 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:03:25.316 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:03:25.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:03:25.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:03:25.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:03:25.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:03:25.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:03:25.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:03:25.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:03:25.809 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:03:25.809 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:03:26.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14808455 2024-09-18T15:03:26.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14808455 '!=' null 2024-09-18T15:03:26.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:03:26.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:26.497 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:03:26.497 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:26.572 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:03:26.572 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:03:26.572 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:26.572 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:26.572 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:03:26.572 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:26.572 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:03:26.573 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:03:26.573 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:26.573 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:03:26.573 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:26.573 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:26.573 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:26.573 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:26.573 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:03:26.573 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:26.573 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:26.573 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:26.573 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:26.573 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:26.573 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:03:26.574 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:03:26.574 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:26.574 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:03:26.574 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:03:26.574 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:26.574 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:26.574 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:26.574 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:26.574 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:03:26.574 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:26.574 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:26.574 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:03:26.574 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:03:26.574 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:26.575 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:03:26.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:03:26.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:03:26.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:03:26.681 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:03:26.681 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:03:26.681 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:03:26.681 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:03:26.681 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:03:26.681 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:03:27.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:03:27.085 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:03:27.085 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:03:27.085 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:03:27.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:03:27.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:03:27.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:03:27.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:03:27.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:03:27.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:03:27.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:03:27.589 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:03:27.589 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:03:28.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12623793 2024-09-18T15:03:28.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12623793 '!=' null 2024-09-18T15:03:28.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:03:28.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:28.229 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:03:28.229 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:28.304 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:03:28.304 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:03:28.304 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:28.304 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:28.305 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:03:28.305 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:28.305 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:03:28.305 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:03:28.305 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:28.305 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:03:28.305 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:28.305 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:28.305 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:28.305 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:28.305 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:03:28.305 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:28.305 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:28.305 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:28.305 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:28.306 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:28.306 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:03:28.306 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:03:28.306 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:28.306 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:03:28.306 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:03:28.306 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:28.306 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:28.306 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:28.306 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:28.306 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:03:28.306 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:28.306 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:28.306 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:03:28.306 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:03:28.307 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:28.307 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:03:28.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:03:28.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:03:28.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:03:28.414 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:03:28.414 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:03:28.414 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:03:28.414 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:03:28.414 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:03:28.414 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:03:28.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:03:28.776 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:03:28.777 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:03:28.777 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:03:29.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:03:29.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:03:29.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:03:29.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:03:29.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:03:29.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:03:29.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:03:29.274 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:03:29.274 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:03:29.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12231269 2024-09-18T15:03:29.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12231269 '!=' null 2024-09-18T15:03:29.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:03:29.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:29.895 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:03:29.895 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:29.970 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:03:29.970 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:03:29.970 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:29.970 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:29.971 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:03:29.971 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:29.971 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:03:29.971 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:03:29.971 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:29.971 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:03:29.971 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:29.971 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:29.971 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:29.971 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:29.971 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:03:29.971 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:29.971 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:29.971 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:29.971 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:29.972 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:29.972 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:03:29.972 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:03:29.972 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:29.972 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:03:29.972 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:03:29.972 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:29.972 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:29.972 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:29.972 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:29.972 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:03:29.972 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:29.972 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:29.972 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:03:29.973 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:03:29.973 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:29.973 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:03:29.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:03:30.079 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:03:30.080 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:03:30.080 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:03:30.080 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:03:30.080 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:03:30.080 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:03:30.080 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:03:30.081 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:03:30.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:03:30.459 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:03:30.460 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:03:30.460 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:03:31.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:03:31.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:03:31.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:03:31.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:03:31.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:03:31.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:03:31.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:03:31.017 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:03:31.017 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:03:31.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20967889 2024-09-18T15:03:31.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20967889 '!=' null 2024-09-18T15:03:31.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:03:31.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:31.520 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:03:31.520 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:31.595 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:03:31.595 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:03:31.595 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:31.595 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:31.595 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:03:31.595 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:31.595 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:03:31.596 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:03:31.596 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:31.596 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:03:31.596 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:31.596 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:31.596 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:31.596 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:31.596 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:03:31.596 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:31.596 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:31.596 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:31.596 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:31.596 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:31.597 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:03:31.597 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:03:31.597 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:31.597 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:03:31.597 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:03:31.597 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:31.597 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:31.597 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:31.597 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:31.597 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:03:31.597 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:31.597 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:31.597 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:03:31.597 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:03:31.597 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:31.598 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:03:31.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:03:31.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:03:31.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:03:31.704 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:03:31.704 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:03:31.705 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:03:31.705 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:03:31.705 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:03:31.705 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:03:32.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:03:32.122 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:03:32.122 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:03:32.122 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:03:32.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:03:32.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:03:32.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:03:32.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:03:32.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:03:32.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:03:32.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:03:32.620 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:03:32.620 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:03:33.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599445 2024-09-18T15:03:33.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599445 '!=' null 2024-09-18T15:03:33.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:03:33.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:33.144 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:03:33.144 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:33.220 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:03:33.220 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:03:33.220 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:33.220 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:33.220 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:03:33.221 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:33.221 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:03:33.221 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:03:33.221 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:33.221 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:03:33.221 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:33.221 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:33.221 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:33.221 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:33.221 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:03:33.221 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:33.221 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:33.221 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:33.221 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:33.222 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:33.222 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:03:33.222 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:03:33.222 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:33.222 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:03:33.222 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:03:33.222 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:33.222 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:33.222 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:33.222 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:33.222 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:03:33.222 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:33.222 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:33.222 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:03:33.223 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:03:33.223 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:33.223 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:03:33.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:03:33.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:03:33.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:03:33.329 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:03:33.329 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:03:33.329 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:03:33.329 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:03:33.329 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:03:33.329 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:03:33.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:03:33.689 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:03:33.690 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:03:33.690 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:03:34.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:03:34.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:03:34.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:03:34.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:03:34.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:03:34.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:03:34.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:03:34.193 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:03:34.193 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:03:34.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25336080 2024-09-18T15:03:34.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25336080 '!=' null 2024-09-18T15:03:34.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:03:34.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:34.771 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:03:34.771 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:34.845 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:03:34.845 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:03:34.845 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:34.845 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:34.845 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:03:34.845 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:34.846 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:03:34.846 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:03:34.846 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:34.846 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:03:34.846 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:34.846 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:34.846 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:34.846 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:34.846 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:03:34.846 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:34.846 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:34.846 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:34.846 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:34.846 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:34.847 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:03:34.847 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:03:34.847 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:34.847 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:03:34.847 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:03:34.847 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:34.847 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:34.847 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:34.847 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:34.847 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:03:34.847 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:34.847 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:34.847 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:03:34.848 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:03:34.848 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:34.848 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:03:34.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:03:34.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:03:34.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:03:34.954 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:03:34.954 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:03:34.954 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:03:34.954 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:03:34.954 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:03:34.954 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:03:35.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:03:35.381 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:03:35.381 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:03:35.381 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:03:35.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:03:35.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:03:35.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:03:35.991 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:03:35.991 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:03:35.991 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:03:35.991 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:03:35.991 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:03:35.991 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:03:36.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473114 2024-09-18T15:03:36.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473114 '!=' null 2024-09-18T15:03:36.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:03:36.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:36.503 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:03:36.504 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:36.577 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:03:36.577 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:03:36.577 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:36.577 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:36.577 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:03:36.577 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:36.577 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:03:36.578 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:03:36.578 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:36.578 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:03:36.578 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:36.578 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:36.578 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:36.578 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:36.578 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:03:36.578 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:36.578 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:36.578 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:36.578 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:36.578 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:36.578 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:03:36.579 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:03:36.579 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:36.579 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:03:36.579 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:03:36.579 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:36.579 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:36.579 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:36.579 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:36.579 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:03:36.579 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:36.579 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:36.579 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:03:36.580 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:03:36.580 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:36.580 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:03:36.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:03:36.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:03:36.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:03:36.686 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:03:36.686 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:03:36.686 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:03:36.686 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:03:36.686 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:03:36.686 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:03:37.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:03:37.066 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:03:37.067 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:03:37.067 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:03:37.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:03:37.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:03:37.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:03:37.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:03:37.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:03:37.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:03:37.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:03:37.586 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:03:37.586 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:03:38.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473114 2024-09-18T15:03:38.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473114 '!=' null 2024-09-18T15:03:38.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:03:38.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:38.379 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:03:38.379 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:38.453 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:03:38.453 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:03:38.453 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:38.453 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:38.453 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:03:38.453 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:38.454 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:03:38.454 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:03:38.454 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:38.454 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:03:38.454 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:38.454 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:38.454 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:38.454 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:38.454 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:03:38.454 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:38.454 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:38.454 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:38.454 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:38.454 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:38.455 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:03:38.455 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:03:38.455 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:38.455 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:03:38.455 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:03:38.455 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:38.455 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:38.455 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:38.455 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:38.455 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:03:38.455 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:38.455 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:38.455 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:03:38.455 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:03:38.456 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:38.456 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:03:38.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:03:38.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:03:38.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:03:38.562 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:03:38.562 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:03:38.562 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:03:38.562 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:03:38.562 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:03:38.562 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:03:38.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:03:38.929 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:03:38.930 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:03:38.930 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:03:39.514 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:03:39.514 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:03:39.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:03:39.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:03:39.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:03:39.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:03:39.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:03:39.515 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:03:39.515 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:03:40.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473230 2024-09-18T15:03:40.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473230 '!=' null 2024-09-18T15:03:40.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:03:40.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:40.056 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:03:40.057 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:40.131 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:03:40.131 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:03:40.131 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:40.132 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:40.132 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:03:40.132 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:40.132 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:03:40.132 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:03:40.132 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:40.132 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:03:40.132 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:40.132 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:40.132 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:40.132 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:40.132 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:03:40.132 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:40.133 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:40.133 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:40.133 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:40.133 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:40.133 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:03:40.133 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:03:40.133 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:40.133 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:03:40.133 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:03:40.133 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:40.133 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:40.133 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:40.133 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:40.134 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:03:40.134 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:40.134 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:40.134 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:03:40.134 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:03:40.134 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:40.134 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:03:40.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:03:40.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:03:40.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:03:40.240 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:03:40.240 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:03:40.240 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:03:40.240 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:03:40.240 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:03:40.241 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:03:40.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:03:40.605 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:03:40.605 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:03:40.605 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:03:41.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:03:41.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:03:41.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:03:41.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:03:41.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:03:41.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:03:41.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:03:41.140 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:03:41.140 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:03:41.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209680 2024-09-18T15:03:41.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209680 '!=' null 2024-09-18T15:03:41.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:03:41.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:41.682 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:03:41.682 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:41.756 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:03:41.756 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:03:41.756 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:41.757 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:41.757 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:03:41.757 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:41.757 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:03:41.757 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:03:41.757 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:41.757 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:03:41.757 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:41.757 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:41.757 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:41.757 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:41.757 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:03:41.757 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:41.758 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:41.758 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:41.758 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:41.758 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:41.758 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:03:41.758 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:03:41.758 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:41.758 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:03:41.758 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:03:41.758 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:41.758 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:41.758 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:41.758 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:41.759 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:03:41.759 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:41.759 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:41.759 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:03:41.759 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:03:41.759 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:41.759 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:03:41.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:03:41.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:03:41.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:03:41.865 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:03:41.865 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:03:41.865 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:03:41.865 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:03:41.865 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:03:41.865 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:03:42.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:03:42.225 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:03:42.226 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:03:42.226 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:03:42.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:03:42.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:03:42.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:03:42.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:03:42.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:03:42.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:03:42.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:03:42.732 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:03:42.732 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:03:43.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473060 2024-09-18T15:03:43.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473060 '!=' null 2024-09-18T15:03:43.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:03:43.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:43.337 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:03:43.337 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:43.411 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:03:43.411 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:03:43.411 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:43.411 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:43.412 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:03:43.412 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:43.412 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:03:43.412 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:03:43.412 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:43.412 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:03:43.412 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:43.412 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:43.412 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:43.412 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:43.412 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:03:43.412 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:43.412 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:43.412 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:43.413 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:43.413 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:43.413 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:03:43.413 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:03:43.413 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:43.413 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:03:43.413 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:03:43.413 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:43.413 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:43.413 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:43.413 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:43.413 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:03:43.413 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:43.413 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:43.413 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:03:43.414 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:03:43.414 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:43.414 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:03:43.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:03:43.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:03:43.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:03:43.520 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:03:43.520 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:03:43.520 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:03:43.520 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:03:43.520 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:03:43.520 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:03:43.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:03:43.885 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:03:43.885 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:03:43.885 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:03:44.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:03:44.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:03:44.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:03:44.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:03:44.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:03:44.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:03:44.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:03:44.448 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:03:44.448 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:03:45.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25336172 2024-09-18T15:03:45.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25336172 '!=' null 2024-09-18T15:03:45.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:03:45.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:45.069 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:03:45.069 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:45.145 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:03:45.145 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:03:45.145 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:45.145 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:45.145 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:03:45.145 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:45.145 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:03:45.145 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:03:45.145 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:45.146 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:03:45.146 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:45.146 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:45.146 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:45.146 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:45.146 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:03:45.146 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:45.146 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:45.146 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:45.146 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:45.146 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:45.146 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:03:45.146 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:03:45.146 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:45.147 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:03:45.147 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:03:45.147 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:45.147 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:45.147 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:45.147 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:45.147 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:03:45.147 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:45.147 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:45.147 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:03:45.147 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:03:45.147 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:45.147 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:03:45.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:03:45.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:03:45.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:03:45.254 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:03:45.254 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:03:45.254 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:03:45.254 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:03:45.254 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:03:45.254 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:03:45.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:03:45.615 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:03:45.615 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:03:45.615 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:03:46.231 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:03:46.231 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:03:46.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:03:46.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:03:46.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:03:46.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:03:46.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:03:46.232 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:03:46.232 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:03:46.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599521 2024-09-18T15:03:46.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599521 '!=' null 2024-09-18T15:03:46.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:03:46.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:46.843 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:03:46.843 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:46.918 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:03:46.918 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:03:46.918 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:46.918 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:46.918 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:03:46.918 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:46.918 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:03:46.918 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:03:46.918 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:46.918 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:03:46.918 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:46.918 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:46.918 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:46.918 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:46.919 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:03:46.919 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:46.919 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:46.919 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:46.919 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:46.919 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:46.919 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:03:46.919 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:03:46.919 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:46.919 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:03:46.919 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:03:46.919 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:46.920 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:46.920 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:46.920 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:46.920 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:03:46.920 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:46.920 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:46.920 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:03:46.920 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:03:46.921 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:46.921 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:03:46.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:03:47.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:03:47.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:03:47.027 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:03:47.027 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:03:47.027 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:03:47.027 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:03:47.027 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:03:47.027 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:03:47.390 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:03:47.390 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:03:47.390 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:03:47.390 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:03:47.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:03:47.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:03:47.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:03:47.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:03:47.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:03:47.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:03:47.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:03:47.912 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:03:47.912 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:03:48.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599597 2024-09-18T15:03:48.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599597 '!=' null 2024-09-18T15:03:48.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:03:48.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:48.613 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:03:48.614 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:48.688 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:03:48.689 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:03:48.689 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:48.689 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:48.689 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:03:48.689 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:48.689 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:03:48.689 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:03:48.689 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:48.689 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:03:48.689 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:48.689 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:48.689 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:48.689 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:48.689 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:03:48.690 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:48.690 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:48.690 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:48.690 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:48.690 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:48.690 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:03:48.690 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:03:48.690 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:48.690 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:03:48.690 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:03:48.690 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:48.690 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:48.690 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:48.691 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:48.691 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:03:48.691 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:48.691 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:48.691 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:03:48.691 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:03:48.691 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:48.691 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:03:48.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:03:48.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:03:48.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:03:48.797 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:03:48.797 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:03:48.797 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:03:48.797 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:03:48.797 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:03:48.797 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:03:49.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:03:49.243 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:03:49.244 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:03:49.244 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:03:49.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:03:49.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:03:49.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:03:49.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:03:49.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:03:49.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:03:49.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:03:49.744 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:03:49.744 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:03:50.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599597 2024-09-18T15:03:50.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599597 '!=' null 2024-09-18T15:03:50.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:03:50.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:50.381 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:03:50.381 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:50.455 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:03:50.455 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:03:50.455 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:50.456 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:50.456 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:03:50.456 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:50.456 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:03:50.456 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:03:50.456 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:50.456 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:03:50.456 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:50.456 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:50.456 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:50.456 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:50.456 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:03:50.457 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:50.457 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:50.457 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:50.457 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:50.457 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:50.457 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:03:50.457 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:03:50.457 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:50.457 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:03:50.457 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:03:50.457 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:50.457 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:50.457 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:50.457 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:50.457 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:03:50.458 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:50.458 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:50.458 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:03:50.458 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:03:50.458 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:50.458 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:03:50.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:03:50.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:03:50.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:03:50.564 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:03:50.564 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:03:50.564 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:03:50.564 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:03:50.564 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:03:50.564 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:03:50.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:03:50.934 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:03:50.935 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:03:50.935 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:03:51.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:03:51.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:03:51.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:03:51.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:03:51.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:03:51.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:03:51.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:03:51.474 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:03:51.474 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:03:52.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209848 2024-09-18T15:03:52.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209848 '!=' null 2024-09-18T15:03:52.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:03:52.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:52.074 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:03:52.074 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:52.148 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:03:52.148 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:03:52.148 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:52.148 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:52.148 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:03:52.148 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:52.148 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:03:52.148 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:03:52.148 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:52.148 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:03:52.148 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:52.148 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:52.148 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:52.149 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:52.149 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:03:52.149 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:52.149 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:52.149 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:52.149 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:52.149 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:52.149 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:03:52.149 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:03:52.149 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:52.149 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:03:52.150 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:03:52.150 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:52.150 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:52.150 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:52.150 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:52.150 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:03:52.150 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:52.150 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:52.150 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:03:52.150 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:03:52.150 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:52.150 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:03:52.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:03:52.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:03:52.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:03:52.256 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:03:52.257 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:03:52.257 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:03:52.257 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:03:52.257 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:03:52.257 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:03:52.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:03:52.617 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:03:52.618 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:03:52.618 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:03:53.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:03:53.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:03:53.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:03:53.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:03:53.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:03:53.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:03:53.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:03:53.170 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:03:53.170 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:03:53.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473092 2024-09-18T15:03:53.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473092 '!=' null 2024-09-18T15:03:53.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:03:53.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:53.796 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:03:53.796 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:53.871 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:03:53.871 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:03:53.871 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:53.871 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:53.871 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:03:53.871 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:53.871 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:03:53.871 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:03:53.871 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:53.871 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:03:53.872 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:53.872 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:53.872 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:53.872 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:53.872 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:03:53.872 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:53.872 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:53.872 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:53.872 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:53.872 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:53.872 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:03:53.872 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:03:53.872 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:53.873 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:03:53.873 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:03:53.873 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:53.873 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:53.873 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:53.873 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:53.873 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:03:53.873 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:53.873 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:53.873 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:03:53.873 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:03:53.873 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:53.874 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:03:53.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:03:53.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:03:53.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:03:53.979 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:03:53.980 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:03:53.980 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:03:53.980 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:03:53.980 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:03:53.980 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:03:54.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:03:54.350 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:03:54.350 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:03:54.350 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:03:54.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:03:54.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:03:54.946 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:03:54.946 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:03:54.946 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:03:54.946 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:03:54.946 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:03:54.946 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:03:54.946 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:03:55.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25336044 2024-09-18T15:03:55.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25336044 '!=' null 2024-09-18T15:03:55.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:03:55.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:55.603 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:03:55.603 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:55.677 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:03:55.678 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:03:55.678 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:55.678 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:55.678 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:03:55.678 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:55.678 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:03:55.678 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:03:55.678 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:55.678 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:03:55.678 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:55.678 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:55.678 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:55.678 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:55.679 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:03:55.679 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:55.679 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:55.679 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:55.679 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:55.679 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:55.679 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:03:55.679 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:03:55.679 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:55.679 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:03:55.679 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:03:55.679 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:55.679 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:55.679 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:55.679 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:55.679 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:03:55.680 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:55.680 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:55.680 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:03:55.680 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:03:55.680 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:55.680 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:03:55.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:03:55.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:03:55.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:03:55.786 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:03:55.787 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:03:55.787 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:03:55.787 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:03:55.787 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:03:55.787 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:03:56.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:03:56.148 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:03:56.148 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:03:56.148 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:03:56.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:03:56.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:03:56.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:03:56.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:03:56.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:03:56.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:03:56.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:03:56.774 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:03:56.774 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:03:57.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473008 2024-09-18T15:03:57.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473008 '!=' null 2024-09-18T15:03:57.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:03:57.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:57.279 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:03:57.279 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:57.354 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:03:57.354 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:03:57.354 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:57.354 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:57.354 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:03:57.354 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:57.354 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:03:57.354 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:03:57.354 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:57.355 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:03:57.355 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:57.355 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:57.355 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:57.355 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:57.355 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:03:57.355 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:57.355 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:57.355 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:57.355 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:57.355 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:57.355 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:03:57.355 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:03:57.355 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:57.355 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:03:57.356 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:03:57.356 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:57.356 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:57.356 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:57.356 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:57.356 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:03:57.356 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:57.356 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:57.356 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:03:57.356 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:03:57.356 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:57.356 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:03:57.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:03:57.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:03:57.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:03:57.463 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:03:57.463 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:03:57.463 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:03:57.463 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:03:57.463 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:03:57.463 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:03:57.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:03:57.934 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:03:57.935 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:03:57.935 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:03:58.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:03:58.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:03:58.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:03:58.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:03:58.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:03:58.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:03:58.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:03:58.444 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:03:58.444 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:03:59.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17472990 2024-09-18T15:03:59.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17472990 '!=' null 2024-09-18T15:03:59.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:03:59.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:59.148 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:03:59.148 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:03:59.223 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:03:59.224 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:03:59.224 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:59.224 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:59.224 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:03:59.224 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:59.224 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:03:59.224 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:03:59.224 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:59.224 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:03:59.224 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:59.224 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:59.224 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:59.224 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:59.225 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:03:59.225 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:03:59.225 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:59.225 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:59.225 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:59.225 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:59.225 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:03:59.225 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:03:59.225 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:03:59.225 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:03:59.225 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:03:59.225 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:03:59.225 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:59.225 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:03:59.226 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:03:59.226 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:03:59.226 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:03:59.226 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:03:59.226 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:03:59.226 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:03:59.226 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:03:59.226 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:03:59.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:03:59.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:03:59.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:03:59.332 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:03:59.332 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:03:59.332 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:03:59.332 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:03:59.332 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:03:59.332 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:03:59.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:03:59.694 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:03:59.695 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:03:59.695 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:00.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:04:00.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:04:00.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:04:00.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:04:00.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:04:00.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:04:00.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:04:00.236 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:04:00.236 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:04:00.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27083261 2024-09-18T15:04:00.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27083261 '!=' null 2024-09-18T15:04:00.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:04:00.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:00.852 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:04:00.852 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:00.932 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:04:00.932 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:04:00.932 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:00.932 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:00.932 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:04:00.932 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:00.932 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:04:00.932 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:04:00.932 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:00.933 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:04:00.933 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:00.933 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:00.933 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:00.933 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:00.933 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:04:00.933 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:00.933 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:00.933 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:00.933 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:00.933 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:00.933 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:04:00.934 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:04:00.934 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:00.934 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:04:00.934 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:04:00.934 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:00.934 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:00.934 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:00.934 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:00.934 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:04:00.934 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:00.934 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:00.934 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:04:00.934 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:04:00.934 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:00.935 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:04:00.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:04:01.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:04:01.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:01.041 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:01.041 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:01.041 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:01.041 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:01.041 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:01.041 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:01.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:04:01.438 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:01.439 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:01.439 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:01.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:04:01.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:04:01.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:04:01.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:04:01.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:04:01.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:04:01.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:04:01.945 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:04:01.945 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:04:02.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27083261 2024-09-18T15:04:02.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27083261 '!=' null 2024-09-18T15:04:02.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:04:02.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:02.463 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:04:02.463 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:02.538 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:04:02.538 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:04:02.538 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:02.538 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:02.538 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:04:02.538 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:02.538 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:04:02.538 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:04:02.538 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:02.539 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:04:02.539 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:02.539 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:02.539 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:02.539 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:02.539 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:04:02.539 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:02.539 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:02.539 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:02.539 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:02.539 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:02.539 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:04:02.539 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:04:02.539 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:02.540 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:04:02.540 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:04:02.540 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:02.540 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:02.540 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:02.540 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:02.540 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:04:02.540 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:02.540 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:02.540 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:04:02.540 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:04:02.540 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:02.540 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:04:02.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:04:02.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:04:02.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:02.647 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:02.647 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:02.647 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:02.647 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:02.647 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:02.647 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:03.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:04:03.023 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:03.023 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:03.023 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:03.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:04:03.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:04:03.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:04:03.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:04:03.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:04:03.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:04:03.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:04:03.598 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:04:03.598 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:04:04.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473118 2024-09-18T15:04:04.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473118 '!=' null 2024-09-18T15:04:04.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:04:04.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:04.404 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:04:04.404 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:04.479 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:04:04.479 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:04:04.479 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:04.479 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:04.479 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:04:04.479 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:04.479 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:04:04.479 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:04:04.480 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:04.480 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:04:04.480 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:04.480 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:04.480 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:04.480 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:04.480 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:04:04.480 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:04.480 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:04.480 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:04.480 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:04.481 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:04.481 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:04:04.481 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:04:04.481 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:04.481 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:04:04.481 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:04:04.481 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:04.481 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:04.481 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:04.481 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:04.481 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:04:04.481 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:04.481 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:04.481 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:04:04.482 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:04:04.482 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:04.482 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:04:04.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:04:04.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:04:04.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:04.588 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:04.588 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:04.588 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:04.588 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:04.588 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:04.588 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:04.947 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:04:04.947 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:04.947 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:04.948 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:05.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:04:05.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:04:05.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:04:05.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:04:05.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:04:05.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:04:05.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:04:05.443 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:04:05.443 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:04:06.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26210060 2024-09-18T15:04:06.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26210060 '!=' null 2024-09-18T15:04:06.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:04:06.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:06.007 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:04:06.007 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:06.082 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:04:06.082 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:04:06.082 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:06.082 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:06.082 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:04:06.082 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:06.082 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:04:06.082 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:04:06.082 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:06.082 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:04:06.082 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:06.082 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:06.082 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:06.083 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:06.083 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:04:06.083 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:06.083 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:06.083 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:06.083 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:06.083 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:06.083 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:04:06.083 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:04:06.083 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:06.083 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:04:06.083 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:04:06.083 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:06.083 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:06.083 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:06.084 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:06.084 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:04:06.084 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:06.084 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:06.084 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:04:06.084 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:04:06.084 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:06.084 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:04:06.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:04:06.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:04:06.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:06.190 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:06.190 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:06.191 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:06.191 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:06.191 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:06.191 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:06.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:04:06.555 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:06.555 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:06.555 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:07.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:04:07.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:04:07.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:04:07.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:04:07.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:04:07.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:04:07.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:04:07.180 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:04:07.181 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:04:07.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18347040 2024-09-18T15:04:07.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18347040 '!=' null 2024-09-18T15:04:07.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:04:07.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:07.830 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:04:07.831 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:07.905 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:04:07.905 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:04:07.905 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:07.905 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:07.905 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:04:07.905 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:07.906 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:04:07.906 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:04:07.906 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:07.906 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:04:07.906 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:07.906 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:07.906 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:07.906 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:07.906 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:04:07.906 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:07.906 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:07.906 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:07.906 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:07.906 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:07.906 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:04:07.907 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:04:07.907 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:07.907 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:04:07.907 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:04:07.907 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:07.907 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:07.907 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:07.907 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:07.907 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:04:07.907 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:07.907 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:07.907 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:04:07.907 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:04:07.907 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:07.907 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:04:07.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:04:08.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:04:08.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:08.014 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:08.014 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:08.014 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:08.014 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:08.014 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:08.014 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:08.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:04:08.379 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:08.379 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:08.379 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:08.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:04:08.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:04:08.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:04:08.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:04:08.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:04:08.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:04:08.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:04:08.980 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:04:08.980 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:04:09.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18347013 2024-09-18T15:04:09.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18347013 '!=' null 2024-09-18T15:04:09.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:04:09.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:09.589 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:04:09.589 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:09.664 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:04:09.664 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:04:09.664 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:09.664 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:09.664 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:04:09.664 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:09.664 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:04:09.664 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:04:09.664 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:09.665 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:04:09.665 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:09.665 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:09.665 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:09.665 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:09.665 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:04:09.665 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:09.665 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:09.665 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:09.665 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:09.665 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:09.665 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:04:09.665 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:04:09.666 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:09.666 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:04:09.666 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:04:09.666 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:09.666 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:09.666 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:09.666 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:09.666 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:04:09.666 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:09.666 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:09.666 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:04:09.666 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:04:09.666 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:09.666 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:04:09.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:04:09.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:04:09.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:09.773 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:09.773 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:09.773 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:09.773 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:09.773 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:09.773 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:10.135 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:04:10.136 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:10.136 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:10.136 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:10.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:04:10.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:04:10.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:04:10.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:04:10.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:04:10.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:04:10.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:04:10.801 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:04:10.801 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:04:11.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27083739 2024-09-18T15:04:11.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27083739 '!=' null 2024-09-18T15:04:11.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:04:11.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:11.339 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:04:11.339 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:11.413 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:04:11.413 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:04:11.413 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:11.413 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:11.413 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:04:11.413 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:11.413 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:04:11.414 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:04:11.414 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:11.414 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:04:11.414 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:11.414 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:11.414 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:11.414 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:11.414 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:04:11.414 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:11.414 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:11.414 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:11.414 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:11.414 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:11.415 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:04:11.415 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:04:11.415 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:11.415 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:04:11.415 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:04:11.415 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:11.415 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:11.415 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:11.415 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:11.415 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:04:11.415 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:11.415 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:11.415 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:04:11.415 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:04:11.415 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:11.416 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:04:11.419 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:04:11.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:04:11.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:11.522 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:11.522 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:11.522 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:11.522 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:11.522 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:11.522 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:11.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:04:11.929 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:11.929 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:11.929 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:12.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:04:12.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:04:12.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:04:12.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:04:12.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:04:12.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:04:12.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:04:12.540 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:04:12.540 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:04:13.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473331 2024-09-18T15:04:13.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473331 '!=' null 2024-09-18T15:04:13.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:04:13.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:13.117 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:04:13.117 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:13.192 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:04:13.192 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:04:13.192 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:13.192 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:13.192 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:04:13.193 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:13.193 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:04:13.193 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:04:13.193 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:13.193 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:04:13.193 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:13.193 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:13.193 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:13.193 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:13.193 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:04:13.193 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:13.194 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:13.194 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:13.194 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:13.194 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:13.194 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:04:13.194 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:04:13.194 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:13.194 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:04:13.194 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:04:13.194 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:13.194 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:13.194 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:13.195 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:13.195 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:04:13.195 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:13.195 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:13.195 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:04:13.195 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:04:13.195 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:13.195 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:04:13.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:04:13.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:04:13.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:13.301 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:13.301 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:13.301 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:13.301 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:13.301 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:13.301 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:13.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:04:13.664 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:13.664 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:13.665 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:14.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:04:14.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:04:14.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:04:14.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:04:14.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:04:14.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:04:14.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:04:14.256 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:04:14.256 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:04:14.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26210018 2024-09-18T15:04:14.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26210018 '!=' null 2024-09-18T15:04:14.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:04:14.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:14.774 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:04:14.775 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:14.850 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:04:14.850 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:04:14.850 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:14.851 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:14.851 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:04:14.851 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:14.851 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:04:14.851 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:04:14.851 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:14.851 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:04:14.851 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:14.851 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:14.851 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:14.851 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:14.851 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:04:14.851 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:14.851 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:14.852 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:14.852 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:14.852 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:14.852 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:04:14.852 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:04:14.852 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:14.852 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:04:14.852 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:04:14.852 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:14.852 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:14.852 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:14.852 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:14.852 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:04:14.852 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:14.853 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:14.853 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:04:14.853 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:04:14.853 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:14.853 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:04:14.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:04:14.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:04:14.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:14.959 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:14.959 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:14.959 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:14.959 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:14.959 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:14.959 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:15.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:04:15.327 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:15.327 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:15.327 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:15.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:04:15.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:04:15.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:04:15.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:04:15.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:04:15.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:04:15.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:04:15.929 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:04:15.929 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:04:16.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26210018 2024-09-18T15:04:16.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26210018 '!=' null 2024-09-18T15:04:16.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:04:16.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:16.639 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:04:16.639 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:16.714 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:04:16.714 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:04:16.714 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:16.714 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:16.714 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:04:16.714 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:16.714 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:04:16.715 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:04:16.715 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:16.715 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:04:16.715 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:16.715 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:16.715 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:16.715 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:16.715 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:04:16.715 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:16.715 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:16.715 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:16.715 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:16.715 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:16.715 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:04:16.716 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:04:16.716 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:16.716 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:04:16.716 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:04:16.716 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:16.716 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:16.716 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:16.716 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:16.716 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:04:16.716 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:16.716 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:16.716 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:04:16.716 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:04:16.717 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:16.717 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:04:16.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:04:16.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:04:16.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:16.823 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:16.823 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:16.823 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:16.823 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:16.823 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:16.823 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:17.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:04:17.188 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:17.188 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:17.188 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:17.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:04:17.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:04:17.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:04:17.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:04:17.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:04:17.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:04:17.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:04:17.697 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:04:17.697 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:04:18.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473018 2024-09-18T15:04:18.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473018 '!=' null 2024-09-18T15:04:18.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:04:18.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:18.338 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:04:18.338 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:18.413 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:04:18.413 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:04:18.413 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:18.413 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:18.413 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:04:18.413 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:18.414 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:04:18.414 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:04:18.414 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:18.414 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:04:18.414 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:18.414 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:18.414 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:18.414 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:18.414 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:04:18.414 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:18.414 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:18.414 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:18.414 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:18.414 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:18.415 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:04:18.415 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:04:18.415 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:18.415 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:04:18.415 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:04:18.415 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:18.415 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:18.415 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:18.415 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:18.415 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:04:18.415 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:18.415 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:18.415 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:04:18.415 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:04:18.416 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:18.416 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:04:18.420 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:04:18.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:04:18.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:18.522 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:18.522 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:18.522 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:18.522 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:18.522 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:18.522 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:18.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:04:18.901 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:18.901 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:18.902 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:19.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:04:19.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:04:19.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:04:19.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:04:19.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:04:19.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:04:19.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:04:19.508 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:04:19.508 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:04:20.021 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17472929 2024-09-18T15:04:20.021 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17472929 '!=' null 2024-09-18T15:04:20.021 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:04:20.021 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:20.021 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:04:20.022 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:20.096 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:04:20.096 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:04:20.096 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:20.097 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:20.097 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:04:20.097 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:20.097 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:04:20.097 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:04:20.097 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:20.097 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:04:20.097 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:20.097 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:20.097 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:20.097 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:20.097 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:04:20.097 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:20.098 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:20.098 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:20.098 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:20.098 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:20.098 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:04:20.098 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:04:20.098 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:20.098 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:04:20.098 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:04:20.098 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:20.098 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:20.098 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:20.098 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:20.099 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:04:20.099 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:20.099 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:20.099 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:04:20.099 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:04:20.099 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:20.099 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:04:20.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:04:20.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:04:20.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:20.205 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:20.205 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:20.205 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:20.205 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:20.206 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:20.206 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:20.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:04:20.568 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:20.568 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:20.568 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:21.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:04:21.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:04:21.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:04:21.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:04:21.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:04:21.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:04:21.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:04:21.082 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:04:21.082 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:04:21.676 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209589 2024-09-18T15:04:21.676 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209589 '!=' null 2024-09-18T15:04:21.676 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:04:21.676 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:21.676 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:04:21.676 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:21.751 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:04:21.751 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:04:21.752 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:21.752 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:21.752 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:04:21.752 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:21.752 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:04:21.752 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:04:21.752 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:21.752 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:04:21.752 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:21.752 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:21.753 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:21.753 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:21.753 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:04:21.753 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:21.753 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:21.753 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:21.753 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:21.753 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:21.753 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:04:21.753 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:04:21.754 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:21.754 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:04:21.754 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:04:21.754 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:21.754 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:21.754 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:21.754 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:21.754 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:04:21.754 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:21.754 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:21.754 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:04:21.754 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:04:21.754 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:21.754 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:04:21.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:04:21.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:04:21.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:21.860 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:21.860 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:21.860 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:21.861 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:21.861 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:21.861 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:22.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:04:22.339 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:22.339 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:22.339 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:22.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:04:22.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:04:22.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:04:22.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:04:22.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:04:22.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:04:22.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:04:22.844 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:04:22.844 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:04:23.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473038 2024-09-18T15:04:23.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473038 '!=' null 2024-09-18T15:04:23.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:04:23.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:23.516 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:04:23.516 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:23.591 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:04:23.591 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:04:23.591 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:23.591 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:23.591 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:04:23.591 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:23.591 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:04:23.592 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:04:23.592 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:23.592 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:04:23.592 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:23.592 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:23.592 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:23.592 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:23.592 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:04:23.592 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:23.592 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:23.592 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:23.592 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:23.592 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:23.593 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:04:23.593 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:04:23.593 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:23.593 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:04:23.593 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:04:23.593 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:23.593 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:23.593 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:23.593 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:23.593 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:04:23.593 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:23.593 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:23.593 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:04:23.593 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:04:23.593 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:23.594 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:04:23.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:04:23.699 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:04:23.699 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:23.700 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:23.700 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:23.700 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:23.700 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:23.700 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:23.700 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:24.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:04:24.062 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:24.063 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:24.063 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:24.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:04:24.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:04:24.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:04:24.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:04:24.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:04:24.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:04:24.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:04:24.628 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:04:24.628 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:04:25.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209633 2024-09-18T15:04:25.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209633 '!=' null 2024-09-18T15:04:25.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:04:25.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:25.323 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:04:25.323 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:25.398 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:04:25.398 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:04:25.398 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:25.398 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:25.398 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:04:25.398 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:25.399 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:04:25.399 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:04:25.399 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:25.399 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:04:25.399 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:25.399 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:25.399 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:25.399 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:25.399 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:04:25.399 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:25.399 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:25.399 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:25.399 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:25.399 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:25.399 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:04:25.400 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:04:25.400 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:25.400 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:04:25.400 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:04:25.400 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:25.400 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:25.400 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:25.400 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:25.400 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:04:25.400 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:25.400 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:25.400 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:04:25.400 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:04:25.400 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:25.401 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:04:25.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:04:25.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:04:25.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:25.507 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:25.507 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:25.507 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:25.507 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:25.507 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:25.507 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:25.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:04:25.876 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:25.877 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:25.877 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:26.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:04:26.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:04:26.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:04:26.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:04:26.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:04:26.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:04:26.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:04:26.474 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:04:26.474 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:04:27.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17472906 2024-09-18T15:04:27.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17472906 '!=' null 2024-09-18T15:04:27.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:04:27.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:27.089 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:04:27.089 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:27.163 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:04:27.164 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:04:27.164 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:27.164 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:27.164 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:04:27.164 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:27.164 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:04:27.164 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:04:27.164 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:27.164 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:04:27.164 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:27.164 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:27.164 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:27.164 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:27.164 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:04:27.165 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:27.165 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:27.165 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:27.165 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:27.165 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:27.165 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:04:27.165 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:04:27.165 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:27.165 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:04:27.165 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:04:27.165 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:27.165 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:27.165 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:27.165 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:27.166 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:04:27.166 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:27.166 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:27.166 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:04:27.166 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:04:27.166 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:27.166 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:04:27.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:04:27.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:04:27.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:27.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:27.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:27.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:27.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:27.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:27.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:27.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:04:27.633 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:27.633 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:27.633 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:28.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:04:28.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:04:28.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:04:28.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:04:28.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:04:28.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:04:28.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:04:28.251 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:04:28.251 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:04:28.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473207 2024-09-18T15:04:28.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473207 '!=' null 2024-09-18T15:04:28.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:04:28.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:28.795 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:04:28.795 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:28.870 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:04:28.870 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:04:28.870 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:28.870 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:28.870 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:04:28.870 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:28.871 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:04:28.871 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:04:28.871 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:28.871 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:04:28.871 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:28.871 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:28.871 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:28.871 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:28.871 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:04:28.871 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:28.871 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:28.871 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:28.871 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:28.871 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:28.872 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:04:28.872 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:04:28.872 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:28.872 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:04:28.872 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:04:28.872 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:28.872 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:28.872 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:28.872 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:28.872 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:04:28.872 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:28.872 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:28.872 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:04:28.872 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:04:28.873 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:28.873 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:04:28.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:04:28.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:04:28.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:28.979 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:28.979 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:28.980 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:28.980 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:28.980 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:28.980 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:29.343 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:04:29.343 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:29.344 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:29.344 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:29.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:04:29.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:04:29.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:04:29.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:04:29.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:04:29.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:04:29.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:04:29.837 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:04:29.837 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:04:30.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473207 2024-09-18T15:04:30.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473207 '!=' null 2024-09-18T15:04:30.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:04:30.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:30.435 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:04:30.435 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:30.510 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:04:30.510 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:04:30.510 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:30.510 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:30.511 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:04:30.511 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:30.511 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:04:30.511 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:04:30.511 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:30.511 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:04:30.511 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:30.511 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:30.511 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:30.511 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:30.511 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:04:30.511 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:30.511 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:30.511 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:30.512 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:30.512 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:30.512 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:04:30.512 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:04:30.512 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:30.512 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:04:30.512 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:04:30.512 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:30.512 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:30.512 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:30.512 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:30.512 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:04:30.513 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:30.513 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:30.513 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:04:30.513 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:04:30.513 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:30.513 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:04:30.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:04:30.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:04:30.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:30.619 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:30.619 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:30.619 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:30.620 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:30.620 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:30.620 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:30.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:04:30.978 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:30.979 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:30.979 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:31.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:04:31.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:04:31.599 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:04:31.599 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:04:31.599 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:04:31.599 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:04:31.599 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:04:31.599 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:04:31.599 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:04:32.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209898 2024-09-18T15:04:32.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209898 '!=' null 2024-09-18T15:04:32.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:04:32.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:32.101 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:04:32.101 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:32.175 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:04:32.175 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:04:32.175 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:32.175 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:32.175 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:04:32.175 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:32.175 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:04:32.175 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:04:32.175 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:32.175 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:04:32.175 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:32.176 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:32.176 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:32.176 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:32.176 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:04:32.176 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:32.176 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:32.176 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:32.176 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:32.176 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:32.176 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:04:32.176 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:04:32.176 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:32.176 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:04:32.176 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:04:32.176 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:32.177 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:32.177 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:32.177 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:32.177 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:04:32.177 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:32.177 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:32.177 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:04:32.177 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:04:32.177 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:32.177 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:04:32.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:04:32.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:04:32.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:32.284 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:32.284 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:32.284 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:32.284 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:32.284 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:32.284 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:32.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:04:32.716 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:32.716 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:32.716 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:33.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:04:33.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:04:33.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:04:33.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:04:33.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:04:33.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:04:33.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:04:33.363 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:04:33.363 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:04:34.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473237 2024-09-18T15:04:34.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473237 '!=' null 2024-09-18T15:04:34.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:04:34.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:34.040 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:04:34.040 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:34.115 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:04:34.115 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:04:34.115 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:34.115 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:34.115 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:04:34.115 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:34.116 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:04:34.116 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:04:34.116 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:34.116 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:04:34.116 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:34.116 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:34.116 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:34.116 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:34.116 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:04:34.116 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:34.116 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:34.116 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:34.116 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:34.116 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:34.117 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:04:34.117 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:04:34.117 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:34.117 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:04:34.117 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:04:34.117 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:34.117 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:34.117 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:34.117 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:34.117 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:04:34.117 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:34.117 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:34.117 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:04:34.117 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:04:34.117 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:34.118 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:04:34.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:04:34.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:04:34.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:34.224 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:34.224 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:34.224 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:34.224 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:34.224 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:34.224 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:34.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:04:34.587 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:34.587 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:34.587 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:35.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:04:35.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:04:35.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:04:35.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:04:35.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:04:35.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:04:35.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:04:35.171 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:04:35.171 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:04:35.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209850 2024-09-18T15:04:35.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209850 '!=' null 2024-09-18T15:04:35.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:04:35.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:35.773 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:04:35.773 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:35.847 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:04:35.847 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:04:35.847 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:35.848 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:35.848 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:04:35.848 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:35.848 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:04:35.848 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:04:35.848 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:35.848 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:04:35.848 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:35.848 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:35.848 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:35.848 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:35.848 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:04:35.848 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:35.848 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:35.849 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:35.849 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:35.849 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:35.849 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:04:35.849 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:04:35.849 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:35.849 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:04:35.849 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:04:35.849 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:35.849 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:35.849 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:35.849 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:35.849 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:04:35.849 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:35.849 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:35.850 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:04:35.850 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:04:35.850 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:35.850 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:04:35.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:04:35.956 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:04:35.956 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:35.956 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:35.956 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:35.956 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:35.956 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:35.956 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:35.957 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:36.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:04:36.319 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:36.319 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:36.320 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:36.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:04:36.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:04:36.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:04:36.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:04:36.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:04:36.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:04:36.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:04:36.825 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:04:36.825 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:04:37.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473134 2024-09-18T15:04:37.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473134 '!=' null 2024-09-18T15:04:37.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:04:37.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:37.397 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:04:37.397 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:37.472 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:04:37.472 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:04:37.472 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:37.472 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:37.472 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:04:37.472 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:37.472 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:04:37.472 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:04:37.472 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:37.473 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:04:37.473 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:37.473 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:37.473 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:37.473 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:37.473 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:04:37.473 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:37.473 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:37.473 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:37.473 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:37.473 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:37.473 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:04:37.474 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:04:37.474 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:37.474 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:04:37.474 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:04:37.474 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:37.474 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:37.474 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:37.474 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:37.474 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:04:37.474 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:37.475 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:37.475 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:04:37.475 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:04:37.475 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:37.475 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:04:37.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:04:37.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:04:37.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:37.581 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:37.581 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:37.581 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:37.581 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:37.581 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:37.581 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:37.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:04:37.983 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:37.983 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:37.984 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:38.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:04:38.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:04:38.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:04:38.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:04:38.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:04:38.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:04:38.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:04:38.489 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:04:38.489 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:04:39.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473325 2024-09-18T15:04:39.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473325 '!=' null 2024-09-18T15:04:39.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:04:39.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:39.152 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:04:39.152 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:39.227 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:04:39.227 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:04:39.227 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:39.227 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:39.227 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:04:39.227 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:39.228 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:04:39.228 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:04:39.228 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:39.228 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:04:39.228 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:39.228 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:39.228 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:39.228 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:39.228 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:04:39.228 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:39.229 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:39.229 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:39.229 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:39.229 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:39.229 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:04:39.229 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:04:39.229 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:39.229 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:04:39.229 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:04:39.229 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:39.229 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:39.230 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:39.230 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:39.230 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:04:39.230 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:39.230 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:39.230 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:04:39.230 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:04:39.230 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:39.230 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:04:39.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:04:39.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:04:39.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:39.336 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:39.336 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:39.336 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:39.336 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:39.336 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:39.336 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:39.699 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:04:39.699 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:39.699 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:39.699 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:40.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:04:40.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:04:40.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:04:40.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:04:40.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:04:40.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:04:40.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:04:40.340 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:04:40.340 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:04:41.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209914 2024-09-18T15:04:41.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209914 '!=' null 2024-09-18T15:04:41.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:04:41.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:41.034 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:04:41.034 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:41.109 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:04:41.109 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:04:41.109 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:41.109 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:41.109 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:04:41.109 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:41.109 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:04:41.109 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:04:41.109 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:41.109 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:04:41.109 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:41.109 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:41.110 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:41.110 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:41.110 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:04:41.110 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:41.110 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:41.110 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:41.110 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:41.110 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:41.110 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:04:41.110 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:04:41.110 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:41.110 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:04:41.110 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:04:41.110 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:41.111 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:41.111 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:41.111 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:41.111 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:04:41.111 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:41.111 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:41.111 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:04:41.111 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:04:41.111 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:41.111 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:04:41.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:04:41.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:04:41.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:41.217 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:41.218 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:41.218 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:41.218 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:41.218 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:41.218 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:41.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:04:41.580 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:41.581 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:41.581 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:42.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:04:42.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:04:42.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:04:42.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:04:42.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:04:42.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:04:42.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:04:42.139 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:04:42.139 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:04:42.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473113 2024-09-18T15:04:42.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473113 '!=' null 2024-09-18T15:04:42.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:04:42.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:42.720 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:04:42.720 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:42.797 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:04:42.797 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:04:42.798 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:42.798 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:42.798 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:04:42.798 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:42.798 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:04:42.798 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:04:42.798 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:42.798 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:04:42.798 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:42.798 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:42.798 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:42.799 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:42.799 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:04:42.799 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:42.799 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:42.799 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:42.799 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:42.799 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:42.800 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:04:42.800 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:04:42.800 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:42.800 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:04:42.800 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:04:42.800 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:42.800 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:42.800 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:42.800 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:42.800 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:04:42.800 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:42.800 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:42.800 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:04:42.800 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:04:42.801 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:42.801 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:04:42.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:04:42.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:04:42.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:42.906 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:42.906 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:42.906 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:42.907 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:42.907 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:42.907 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:43.285 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:04:43.285 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:43.286 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:43.286 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:43.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:04:43.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:04:43.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:04:43.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:04:43.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:04:43.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:04:43.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:04:43.783 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:04:43.784 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:04:44.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473113 2024-09-18T15:04:44.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473113 '!=' null 2024-09-18T15:04:44.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:04:44.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:44.455 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:04:44.455 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:44.530 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:04:44.530 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:04:44.531 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:44.531 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:44.531 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:04:44.531 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:44.531 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:04:44.531 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:04:44.531 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:44.531 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:04:44.531 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:44.531 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:44.532 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:44.532 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:44.532 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:04:44.532 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:44.532 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:44.532 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:44.532 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:44.532 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:44.532 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:04:44.532 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:04:44.532 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:44.533 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:04:44.533 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:04:44.533 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:44.533 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:44.533 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:44.533 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:44.533 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:04:44.533 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:44.533 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:44.533 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:04:44.533 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:04:44.534 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:44.534 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:04:44.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:04:44.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:04:44.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:44.640 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:44.640 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:44.640 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:44.640 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:44.640 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:44.640 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:45.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:04:45.002 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:45.003 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:45.003 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:45.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:04:45.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:04:45.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:04:45.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:04:45.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:04:45.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:04:45.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:04:45.577 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:04:45.577 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:04:46.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209698 2024-09-18T15:04:46.079 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209698 '!=' null 2024-09-18T15:04:46.079 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:04:46.079 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:46.079 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:04:46.079 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:46.153 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:04:46.154 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:04:46.154 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:46.154 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:46.154 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:04:46.154 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:46.154 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:04:46.154 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:04:46.154 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:46.154 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:04:46.154 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:46.154 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:46.155 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:46.155 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:46.155 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:04:46.155 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:46.155 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:46.155 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:46.155 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:46.155 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:46.155 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:04:46.155 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:04:46.155 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:46.155 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:04:46.155 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:04:46.155 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:46.156 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:46.156 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:46.156 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:46.156 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:04:46.156 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:46.156 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:46.156 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:04:46.156 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:04:46.156 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:46.156 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:04:46.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:04:46.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:04:46.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:46.263 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:46.263 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:46.263 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:46.263 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:46.263 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:46.263 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:46.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:04:46.702 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:46.703 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:46.703 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:47.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:04:47.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:04:47.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:04:47.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:04:47.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:04:47.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:04:47.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:04:47.203 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:04:47.203 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:04:47.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473135 2024-09-18T15:04:47.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473135 '!=' null 2024-09-18T15:04:47.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:04:47.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:47.811 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:04:47.811 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:47.885 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:04:47.885 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:04:47.886 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:47.886 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:47.886 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:04:47.886 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:47.886 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:04:47.886 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:04:47.886 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:47.886 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:04:47.886 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:47.886 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:47.886 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:47.886 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:47.886 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:04:47.887 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:47.887 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:47.887 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:47.887 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:47.887 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:47.887 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:04:47.887 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:04:47.887 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:47.887 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:04:47.887 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:04:47.887 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:47.887 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:47.887 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:47.887 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:47.888 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:04:47.888 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:47.888 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:47.888 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:04:47.888 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:04:47.888 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:47.888 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:04:47.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:04:47.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:04:47.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:47.995 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:47.995 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:47.995 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:47.995 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:47.995 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:47.995 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:48.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:04:48.439 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:48.440 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:48.440 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:48.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:04:48.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:04:48.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:04:48.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:04:48.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:04:48.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:04:48.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:04:48.983 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:04:48.983 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:04:49.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473195 2024-09-18T15:04:49.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473195 '!=' null 2024-09-18T15:04:49.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:04:49.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:49.627 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:04:49.627 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:49.701 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:04:49.702 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:04:49.702 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:49.702 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:49.702 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:04:49.702 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:49.702 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:04:49.703 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:04:49.703 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:49.703 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:04:49.703 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:49.703 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:49.703 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:49.703 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:49.703 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:04:49.703 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:49.704 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:49.704 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:49.704 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:49.704 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:49.704 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:04:49.704 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:04:49.704 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:49.704 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:04:49.704 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:04:49.704 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:49.704 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:49.704 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:49.705 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:49.705 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:04:49.705 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:49.705 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:49.705 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:04:49.705 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:04:49.705 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:49.705 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:04:49.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:04:49.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:04:49.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:49.810 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:49.811 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:49.811 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:49.811 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:49.811 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:49.811 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:50.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:04:50.176 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:50.176 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:50.176 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:50.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:04:50.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:04:50.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:04:50.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:04:50.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:04:50.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:04:50.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:04:50.703 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:04:50.703 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:04:51.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209670 2024-09-18T15:04:51.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209670 '!=' null 2024-09-18T15:04:51.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:04:51.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:51.311 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:04:51.311 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:51.355 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:04:51.355 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:04:51.355 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:51.355 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:51.355 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:04:51.355 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:51.355 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:04:51.355 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:04:51.355 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:51.355 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:04:51.355 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:51.355 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:51.355 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:51.356 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:51.356 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:04:51.356 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:51.356 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:51.356 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:51.356 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:51.356 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:51.356 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:04:51.356 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:04:51.356 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:51.356 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:04:51.356 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:04:51.356 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:51.357 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:51.357 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:51.357 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:51.357 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:04:51.357 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:51.357 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:51.357 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:04:51.357 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:04:51.357 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:51.357 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:04:51.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:04:51.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:04:51.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:51.464 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:51.464 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:51.464 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:51.464 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:51.464 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:51.464 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:51.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:04:51.868 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:51.868 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:51.868 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:52.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:04:52.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:04:52.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:04:52.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:04:52.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:04:52.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:04:52.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:04:52.371 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:04:52.371 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:04:53.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473076 2024-09-18T15:04:53.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473076 '!=' null 2024-09-18T15:04:53.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:04:53.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:53.189 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:04:53.189 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:53.264 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:04:53.264 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:04:53.264 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:53.264 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:53.264 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:04:53.264 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:53.265 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:04:53.265 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:04:53.265 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:53.265 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:04:53.265 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:53.265 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:53.265 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:53.265 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:53.265 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:04:53.265 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:53.265 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:53.265 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:53.265 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:53.265 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:53.265 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:04:53.266 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:04:53.266 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:53.266 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:04:53.266 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:04:53.266 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:53.266 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:53.266 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:53.266 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:53.266 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:04:53.266 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:53.266 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:53.266 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:04:53.266 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:04:53.266 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:53.267 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:04:53.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:04:53.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:04:53.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:53.373 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:53.373 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:53.373 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:53.373 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:53.373 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:53.373 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:53.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:04:53.737 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:53.738 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:53.738 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:54.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:04:54.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:04:54.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:04:54.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:04:54.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:04:54.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:04:54.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:04:54.261 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:04:54.261 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:04:54.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209892 2024-09-18T15:04:54.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209892 '!=' null 2024-09-18T15:04:54.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:04:54.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:54.980 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:04:54.980 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:55.056 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:04:55.056 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:04:55.056 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:55.056 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:55.056 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:04:55.056 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:55.057 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:04:55.057 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:04:55.057 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:55.057 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:04:55.057 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:55.057 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:55.057 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:55.057 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:55.057 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:04:55.057 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:55.057 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:55.057 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:55.057 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:55.057 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:55.057 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:04:55.058 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:04:55.058 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:55.058 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "7.0", 2024-09-18T15:04:55.058 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 255, 2024-09-18T15:04:55.058 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:55.058 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:55.058 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:55.058 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:55.058 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:04:55.058 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:55.058 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:55.058 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:04:55.058 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:04:55.058 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:55.059 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:04:55.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:04:55.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:04:55.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:55.165 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:55.165 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:55.165 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:55.165 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:55.165 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:55.165 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:55.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:04:55.524 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:55.525 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:55.525 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:56.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:04:56.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:04:56.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:04:56.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:04:56.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:04:56.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:04:56.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:04:56.201 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:04:56.201 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:04:56.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473066 2024-09-18T15:04:56.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473066 '!=' null 2024-09-18T15:04:56.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:04:56.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:56.759 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:04:56.759 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:56.834 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:04:56.834 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:04:56.834 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:56.834 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:56.834 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:04:56.834 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:56.834 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:04:56.834 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:04:56.834 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:56.835 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:04:56.835 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:56.835 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:56.835 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:56.835 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:56.835 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:56.835 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:04:56.835 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:04:56.835 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:56.835 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:04:56.835 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:56.835 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:56.835 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:56.835 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:56.835 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:56.836 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:04:56.836 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:56.836 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:56.836 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:04:56.836 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:04:56.836 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:56.836 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:04:56.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:04:56.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:04:56.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:56.943 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:56.943 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:56.944 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:56.944 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:56.944 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:56.944 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:57.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:04:57.400 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:57.400 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:57.401 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:58.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:04:58.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:04:58.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:04:58.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:04:58.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:04:58.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:04:58.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:04:58.112 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:04:58.112 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:04:58.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13104774 2024-09-18T15:04:58.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13104774 '!=' null 2024-09-18T15:04:58.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:04:58.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:58.736 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:04:58.736 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:04:58.810 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:04:58.810 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:04:58.811 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:58.811 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:58.811 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:04:58.811 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:58.811 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:04:58.811 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:04:58.811 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:58.811 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:04:58.811 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:58.811 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:58.811 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:58.811 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:58.811 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:58.811 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:04:58.811 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:04:58.811 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:04:58.812 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:04:58.812 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:04:58.812 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:04:58.812 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:58.812 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:04:58.812 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:04:58.812 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:04:58.812 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:04:58.812 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:04:58.812 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:04:58.812 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:04:58.812 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:04:58.812 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:04:58.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:04:58.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:04:58.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:04:58.919 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:04:58.919 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:04:58.919 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:04:58.919 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:04:58.919 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:04:58.920 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:04:59.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:04:59.290 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:04:59.290 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:04:59.290 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:04:59.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-18T15:04:59.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:04:59.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:04:59.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:04:59.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:04:59.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:04:59.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:04:59.919 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:04:59.920 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:05:00.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13104774 2024-09-18T15:05:00.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13104774 '!=' null 2024-09-18T15:05:00.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:05:00.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:00.494 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:05:00.494 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:00.568 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:05:00.568 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:05:00.569 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:00.569 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:00.569 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:05:00.569 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:00.569 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:05:00.569 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:05:00.569 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:00.569 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:05:00.569 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:00.569 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:00.569 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:00.569 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:00.569 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:00.570 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:05:00.570 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:05:00.570 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:00.570 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:05:00.570 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:00.570 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:00.570 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:00.570 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:00.570 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:05:00.570 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:05:00.570 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:00.570 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:00.570 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:05:00.570 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:05:00.570 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:00.571 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:05:00.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:05:00.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:05:00.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:00.678 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:00.678 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:00.678 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:00.678 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:00.678 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:00.678 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:01.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:01.097 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:01.097 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:01.097 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:01.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:05:01.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 0 2024-09-18T15:05:01.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T15:05:01.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=1 2024-09-18T15:05:01.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:01.595 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:05:01.595 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:01.670 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:05:01.670 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:05:01.670 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:01.671 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:01.671 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:05:01.671 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:01.671 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:05:01.671 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:05:01.671 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:01.671 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:05:01.671 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:01.671 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:01.671 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:01.671 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:01.671 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:01.671 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:05:01.672 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:05:01.672 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:01.672 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:05:01.672 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:01.672 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:01.672 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:01.672 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:01.672 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:05:01.672 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:05:01.672 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:01.672 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:01.672 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:05:01.672 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:05:01.672 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:01.673 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:05:01.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:05:01.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:05:01.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:01.779 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:01.779 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:01.779 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:01.779 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:01.780 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:01.780 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:02.155 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:02.156 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:02.156 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:02.156 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:02.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:05:02.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:05:02.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:05:02.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:05:02.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:05:02.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:05:02.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:05:02.655 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:05:02.655 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:05:03.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13104918 2024-09-18T15:05:03.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13104918 '!=' null 2024-09-18T15:05:03.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:05:03.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:03.254 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:05:03.254 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:03.329 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:05:03.329 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:05:03.329 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:03.329 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:03.329 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:05:03.329 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:03.329 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:05:03.329 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:05:03.329 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:03.329 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:05:03.330 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:03.330 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:03.330 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:03.330 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:03.330 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:03.330 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:05:03.330 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:05:03.330 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:03.330 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:05:03.330 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:03.330 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:03.330 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:03.330 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:03.330 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:05:03.331 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:05:03.331 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:03.331 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:03.331 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:05:03.331 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:05:03.331 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:03.331 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:05:03.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:05:03.437 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:05:03.437 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:03.438 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:03.438 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:03.438 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:03.438 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:03.438 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:03.438 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:03.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:03.798 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:03.798 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:03.798 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:04.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:05:04.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:05:04.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:05:04.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:05:04.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:05:04.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:05:04.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:05:04.394 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:05:04.394 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:05:04.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26210049 2024-09-18T15:05:04.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26210049 '!=' null 2024-09-18T15:05:04.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:05:04.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:04.942 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:05:04.942 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:05.016 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:05:05.016 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:05:05.016 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:05.016 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:05.016 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:05:05.017 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:05.017 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:05:05.017 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:05:05.017 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:05.017 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:05:05.017 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:05.017 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:05.017 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:05.017 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:05.017 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:05.017 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:05:05.017 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:05:05.017 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:05.017 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:05:05.018 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:05.018 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:05.018 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:05.018 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:05.018 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:05:05.018 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:05:05.018 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:05.018 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:05.018 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:05:05.018 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:05:05.018 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:05.018 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:05:05.022 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:05:05.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:05:05.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:05.125 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:05.125 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:05.125 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:05.125 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:05.125 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:05.125 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:05.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:05.490 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:05.490 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:05.490 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:06.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:05:06.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:05:06.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:05:06.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:05:06.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:05:06.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:05:06.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:05:06.118 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:05:06.118 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:05:06.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21466320 2024-09-18T15:05:06.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21466320 '!=' null 2024-09-18T15:05:06.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:05:06.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:06.702 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:05:06.702 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:06.780 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:05:06.780 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:05:06.780 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:06.780 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:06.780 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:05:06.780 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:06.780 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:05:06.780 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:05:06.781 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:06.781 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:05:06.781 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:06.781 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:06.781 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:06.781 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:06.781 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:06.781 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:05:06.781 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:05:06.781 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:06.781 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:05:06.781 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:06.781 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:06.781 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:06.781 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:06.782 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:05:06.782 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:05:06.782 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:06.782 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:06.782 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:05:06.782 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:05:06.782 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:06.782 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:05:06.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:05:06.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:05:06.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:06.888 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:06.889 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:06.889 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:06.889 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:06.889 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:06.889 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:07.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:07.275 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:07.276 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:07.276 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:07.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:05:07.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:05:07.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:05:07.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:05:07.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:05:07.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:05:07.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:05:07.775 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:05:07.775 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:05:08.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21466320 2024-09-18T15:05:08.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21466320 '!=' null 2024-09-18T15:05:08.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:05:08.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:08.340 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:05:08.340 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:08.415 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:05:08.415 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:05:08.416 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:08.416 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:08.416 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:05:08.416 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:08.416 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:05:08.416 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:05:08.416 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:08.416 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:05:08.416 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:08.416 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:08.416 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:08.416 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:08.416 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:08.416 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:05:08.417 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:05:08.417 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:08.417 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:05:08.417 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:08.417 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:08.417 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:08.417 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:08.417 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:05:08.417 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:05:08.417 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:08.417 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:08.417 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:05:08.417 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:05:08.417 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:08.417 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:05:08.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:05:08.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:05:08.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:08.524 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:08.524 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:08.524 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:08.524 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:08.524 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:08.524 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:08.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:08.894 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:08.894 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:08.894 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:09.492 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:05:09.492 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:05:09.492 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:05:09.492 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:05:09.492 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:05:09.492 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:05:09.492 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:05:09.492 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:05:09.493 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:05:10.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18979637 2024-09-18T15:05:10.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18979637 '!=' null 2024-09-18T15:05:10.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:05:10.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:10.133 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:05:10.133 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:10.207 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:05:10.207 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:05:10.207 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:10.208 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:10.208 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:05:10.208 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:10.208 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:05:10.208 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:05:10.208 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:10.208 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:05:10.208 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:10.208 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:10.208 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:10.208 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:10.208 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:10.208 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:05:10.208 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:05:10.209 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:10.209 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:05:10.209 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:10.209 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:10.209 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:10.209 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:10.209 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:05:10.209 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:05:10.209 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:10.209 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:10.209 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:05:10.209 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:05:10.209 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:10.209 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:05:10.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:05:10.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:05:10.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:10.316 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:10.316 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:10.316 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:10.316 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:10.316 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:10.317 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:10.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:10.717 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:10.717 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:10.717 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:11.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:05:11.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:05:11.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:05:11.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:05:11.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:05:11.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:05:11.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:05:11.240 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:05:11.240 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:05:11.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209457 2024-09-18T15:05:11.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209457 '!=' null 2024-09-18T15:05:11.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:05:11.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:11.917 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:05:11.917 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:11.992 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:05:11.992 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:05:11.993 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:11.993 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:11.993 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:05:11.993 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:11.993 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:05:11.993 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:05:11.994 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:11.994 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:05:11.994 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:11.994 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:11.994 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:11.994 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:11.994 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:11.994 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:05:11.994 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:05:11.995 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:11.995 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:05:11.995 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:11.995 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:11.995 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:11.995 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:11.995 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:05:11.995 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:05:11.995 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:11.996 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:11.996 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:05:11.996 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:05:11.996 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:11.996 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:05:11.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:05:12.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:05:12.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:12.101 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:12.102 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:12.102 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:12.102 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:12.102 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:12.102 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:12.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:12.466 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:12.466 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:12.466 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:12.992 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:05:12.992 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:05:12.992 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:05:12.993 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:05:12.993 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:05:12.993 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:05:12.993 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:05:12.993 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:05:12.993 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:05:13.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17472720 2024-09-18T15:05:13.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17472720 '!=' null 2024-09-18T15:05:13.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:05:13.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:13.598 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:05:13.598 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:13.673 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:05:13.673 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:05:13.673 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:13.673 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:13.674 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:05:13.674 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:13.674 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:05:13.674 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:05:13.674 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:13.674 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:05:13.674 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:13.674 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:13.674 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:13.674 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:13.674 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:13.674 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:05:13.674 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:05:13.674 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:13.675 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:05:13.675 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:13.675 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:13.675 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:13.675 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:13.675 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:05:13.675 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:05:13.675 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:13.675 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:13.675 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:05:13.675 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:05:13.675 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:13.675 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:05:13.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:05:13.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:05:13.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:13.782 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:13.782 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:13.782 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:13.782 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:13.782 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:13.782 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:14.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:14.140 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:14.141 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:14.141 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:14.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:05:14.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:05:14.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:05:14.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:05:14.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:05:14.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:05:14.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:05:14.754 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:05:14.754 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:05:15.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26208985 2024-09-18T15:05:15.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26208985 '!=' null 2024-09-18T15:05:15.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:05:15.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:15.280 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:05:15.280 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:15.354 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:05:15.354 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:05:15.355 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:15.355 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:15.355 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:05:15.355 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:15.355 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:05:15.355 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:05:15.355 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:15.355 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:05:15.355 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:15.355 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:15.355 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:15.355 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:15.355 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:15.355 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:05:15.356 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:05:15.356 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:15.356 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:05:15.356 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:15.356 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:15.356 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:15.356 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:15.356 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:05:15.356 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:05:15.356 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:15.356 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:15.356 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:05:15.356 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:05:15.356 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:15.357 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:05:15.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:05:15.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:05:15.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:15.463 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:15.463 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:15.463 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:15.464 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:15.464 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:15.464 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:15.861 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:15.861 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:15.861 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:15.862 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:16.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:05:16.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:05:16.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:05:16.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:05:16.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:05:16.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:05:16.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:05:16.458 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:05:16.458 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:05:17.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17472756 2024-09-18T15:05:17.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17472756 '!=' null 2024-09-18T15:05:17.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:05:17.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:17.010 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:05:17.010 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:17.070 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:05:17.070 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:05:17.070 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:17.070 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:17.070 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:05:17.070 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:17.070 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:05:17.070 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:05:17.070 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:17.070 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:05:17.070 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:17.070 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:17.070 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:17.071 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:17.071 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:17.071 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:05:17.071 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:05:17.071 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:17.071 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:05:17.071 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:17.071 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:17.071 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:17.071 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:17.071 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:05:17.071 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:05:17.071 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:17.071 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:17.071 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:05:17.071 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:05:17.072 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:17.072 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:05:17.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:05:17.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:05:17.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:17.178 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:17.178 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:17.178 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:17.179 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:17.179 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:17.179 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:17.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:17.539 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:17.540 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:17.540 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:18.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:05:18.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:05:18.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:05:18.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:05:18.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:05:18.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:05:18.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:05:18.145 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:05:18.145 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:05:18.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473074 2024-09-18T15:05:18.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473074 '!=' null 2024-09-18T15:05:18.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:05:18.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:18.799 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:05:18.799 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:18.874 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:05:18.874 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:05:18.874 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:18.874 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:18.874 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:05:18.874 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:18.875 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:05:18.875 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:05:18.875 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:18.875 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:05:18.875 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:18.875 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:18.875 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:18.875 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:18.875 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:18.875 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:05:18.875 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:05:18.875 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:18.875 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:05:18.875 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:18.876 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:18.876 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:18.876 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:18.876 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:05:18.876 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:05:18.876 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:18.876 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:18.876 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:05:18.876 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:05:18.876 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:18.876 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:05:18.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:05:18.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:05:18.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:18.983 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:18.983 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:18.983 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:18.983 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:18.983 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:18.983 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:19.355 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:19.356 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:19.356 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:19.356 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:19.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:05:19.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:05:19.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:05:19.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:05:19.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:05:19.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:05:19.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:05:19.889 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:05:19.889 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:05:20.390 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473074 2024-09-18T15:05:20.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473074 '!=' null 2024-09-18T15:05:20.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:05:20.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:20.391 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:05:20.391 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:20.465 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:05:20.465 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:05:20.465 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:20.466 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:20.466 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:05:20.466 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:20.466 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:05:20.466 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:05:20.466 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:20.466 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:05:20.466 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:20.466 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:20.466 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:20.466 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:20.466 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:20.467 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:05:20.467 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:05:20.467 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:20.467 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:05:20.467 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:20.467 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:20.467 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:20.467 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:20.467 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:05:20.467 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:05:20.467 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:20.467 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:20.467 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:05:20.467 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:05:20.468 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:20.468 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:05:20.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:05:20.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:05:20.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:20.574 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:20.574 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:20.574 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:20.574 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:20.574 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:20.575 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:21.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:21.051 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:21.051 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:21.051 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:21.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:05:21.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:05:21.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:05:21.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:05:21.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:05:21.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:05:21.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:05:21.579 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:05:21.579 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:05:22.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27083239 2024-09-18T15:05:22.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27083239 '!=' null 2024-09-18T15:05:22.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:05:22.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:22.276 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:05:22.276 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:22.351 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:05:22.351 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:05:22.351 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:22.351 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:22.351 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:05:22.351 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:22.351 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:05:22.352 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:05:22.352 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:22.352 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:05:22.352 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:22.352 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:22.352 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:22.352 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:22.352 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:22.352 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:05:22.352 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:05:22.352 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:22.352 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:05:22.352 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:22.353 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:22.353 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:22.353 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:22.353 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:05:22.353 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:05:22.353 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:22.353 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:22.353 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:05:22.353 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:05:22.353 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:22.353 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:05:22.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:05:22.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:05:22.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:22.460 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:22.460 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:22.460 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:22.460 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:22.460 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:22.460 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:22.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:22.822 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:22.823 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:22.823 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:23.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:05:23.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:05:23.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:05:23.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:05:23.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:05:23.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:05:23.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:05:23.398 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:05:23.398 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:05:24.172 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18346824 2024-09-18T15:05:24.172 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18346824 '!=' null 2024-09-18T15:05:24.172 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:05:24.172 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:24.172 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:05:24.173 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:24.247 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:05:24.247 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:05:24.247 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:24.247 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:24.247 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:05:24.247 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:24.248 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:05:24.248 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:05:24.248 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:24.248 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:05:24.248 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:24.248 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:24.248 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:24.248 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:24.248 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:24.248 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:05:24.248 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:05:24.248 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:24.248 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:05:24.249 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:24.249 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:24.249 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:24.249 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:24.249 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:05:24.249 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:05:24.249 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:24.249 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:24.249 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:05:24.249 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:05:24.249 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:24.249 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:05:24.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:05:24.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:05:24.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:24.356 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:24.356 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:24.356 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:24.356 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:24.356 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:24.356 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:24.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:24.836 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:24.836 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:24.837 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:25.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:05:25.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:05:25.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:05:25.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:05:25.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:05:25.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:05:25.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:05:25.348 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:05:25.348 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:05:25.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209926 2024-09-18T15:05:25.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209926 '!=' null 2024-09-18T15:05:25.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:05:25.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:25.986 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:05:25.986 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:26.061 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:05:26.061 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:05:26.061 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:26.061 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:26.061 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:05:26.061 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:26.061 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:05:26.062 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:05:26.062 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:26.062 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:05:26.062 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:26.062 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:26.062 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:26.062 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:26.062 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:26.062 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:05:26.062 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:05:26.062 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:26.062 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:05:26.062 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:26.063 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:26.063 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:26.063 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:26.063 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:05:26.063 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:05:26.063 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:26.063 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:26.063 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:05:26.063 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:05:26.063 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:26.063 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:05:26.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:05:26.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:05:26.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:26.170 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:26.170 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:26.170 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:26.170 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:26.170 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:26.170 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:26.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:26.530 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:26.531 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:26.531 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:27.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:05:27.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:05:27.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:05:27.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:05:27.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:05:27.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:05:27.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:05:27.092 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:05:27.092 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:05:27.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473183 2024-09-18T15:05:27.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473183 '!=' null 2024-09-18T15:05:27.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:05:27.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:27.669 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:05:27.669 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:27.745 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:05:27.745 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:05:27.745 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:27.745 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:27.745 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:05:27.745 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:27.745 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:05:27.745 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:05:27.745 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:27.745 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:05:27.745 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:27.745 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:27.746 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:27.746 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:27.746 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:27.746 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:05:27.746 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:05:27.746 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:27.746 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:05:27.746 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:27.746 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:27.746 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:27.746 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:27.746 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:05:27.746 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:05:27.746 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:27.747 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:27.747 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:05:27.747 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:05:27.747 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:27.747 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:05:27.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:05:27.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:05:27.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:27.854 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:27.854 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:27.854 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:27.854 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:27.854 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:27.854 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:28.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:28.219 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:28.219 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:28.219 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:28.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:05:28.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:05:28.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:05:28.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:05:28.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:05:28.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:05:28.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:05:28.749 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:05:28.749 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:05:29.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473169 2024-09-18T15:05:29.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473169 '!=' null 2024-09-18T15:05:29.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:05:29.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:29.368 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:05:29.368 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:29.442 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:05:29.442 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:05:29.442 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:29.442 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:29.442 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:05:29.442 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:29.442 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:05:29.442 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:05:29.443 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:29.443 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:05:29.443 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:29.443 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:29.443 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:29.443 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:29.443 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:29.443 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:05:29.443 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:05:29.443 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:29.443 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:05:29.443 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:29.443 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:29.443 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:29.444 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:29.444 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:05:29.444 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:05:29.444 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:29.444 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:29.444 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:05:29.444 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:05:29.444 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:29.444 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:05:29.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:05:29.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:05:29.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:29.551 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:29.551 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:29.551 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:29.551 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:29.551 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:29.551 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:29.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:29.913 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:29.914 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:29.914 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:30.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:05:30.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:05:30.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:05:30.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:05:30.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:05:30.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:05:30.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:05:30.416 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:05:30.416 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:05:31.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209684 2024-09-18T15:05:31.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209684 '!=' null 2024-09-18T15:05:31.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:05:31.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:31.144 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:05:31.144 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:31.218 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:05:31.218 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:05:31.218 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:31.219 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:31.219 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:05:31.219 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:31.219 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:05:31.219 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:05:31.219 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:31.219 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:05:31.219 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:31.219 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:31.219 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:31.219 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:31.219 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:31.219 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:05:31.219 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:05:31.220 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:31.220 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:05:31.220 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:31.220 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:31.220 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:31.220 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:31.220 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:05:31.220 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:05:31.220 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:31.220 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:31.220 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:05:31.220 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:05:31.220 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:31.220 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:05:31.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:05:31.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:05:31.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:31.327 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:31.327 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:31.327 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:31.327 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:31.327 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:31.327 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:31.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:31.693 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:31.693 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:31.693 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:32.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:05:32.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:05:32.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:05:32.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:05:32.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:05:32.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:05:32.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:05:32.369 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:05:32.369 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:05:33.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599419 2024-09-18T15:05:33.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599419 '!=' null 2024-09-18T15:05:33.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:05:33.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:33.098 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:05:33.098 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:33.173 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:05:33.174 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:05:33.174 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:33.174 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:33.174 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:05:33.174 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:33.174 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:05:33.174 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:05:33.174 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:33.174 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:05:33.174 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:33.174 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:33.174 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:33.174 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:33.174 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:33.175 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:05:33.175 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:05:33.175 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:33.175 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:05:33.175 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:33.175 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:33.175 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:33.175 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:33.175 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:05:33.175 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:05:33.175 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:33.175 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:33.175 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:05:33.175 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:05:33.175 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:33.176 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:05:33.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:05:33.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:05:33.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:33.282 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:33.282 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:33.282 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:33.282 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:33.282 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:33.282 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:33.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:33.654 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:33.655 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:33.655 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:34.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:05:34.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:05:34.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:05:34.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:05:34.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:05:34.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:05:34.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:05:34.210 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:05:34.210 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:05:34.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25335943 2024-09-18T15:05:34.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25335943 '!=' null 2024-09-18T15:05:34.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:05:34.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:34.788 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:05:34.788 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:34.862 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:05:34.863 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:05:34.863 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:34.863 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:34.863 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:05:34.863 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:34.863 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:05:34.863 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:05:34.863 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:34.863 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:05:34.863 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:34.863 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:34.863 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:34.864 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:34.864 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:34.864 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:05:34.864 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:05:34.864 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:34.864 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:05:34.864 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:34.864 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:34.864 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:34.864 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:34.864 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:05:34.864 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:05:34.864 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:34.864 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:34.864 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:05:34.865 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:05:34.865 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:34.865 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:05:34.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:05:34.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:05:34.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:34.971 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:34.971 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:34.972 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:34.972 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:34.972 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:34.972 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:35.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:35.333 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:35.334 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:35.334 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:35.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:05:35.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:05:35.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:05:35.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:05:35.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:05:35.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:05:35.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:05:35.917 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:05:35.917 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:05:36.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25335943 2024-09-18T15:05:36.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25335943 '!=' null 2024-09-18T15:05:36.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:05:36.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:36.448 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:05:36.448 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:36.523 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:05:36.523 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:05:36.523 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:36.523 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:36.523 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:05:36.523 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:36.523 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:05:36.524 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:05:36.524 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:36.524 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:05:36.524 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:36.524 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:36.524 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:36.524 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:36.524 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:36.524 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:05:36.524 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:05:36.524 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:36.524 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:05:36.524 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:36.524 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:36.524 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:36.525 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:36.525 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:05:36.525 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:05:36.525 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:36.525 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:36.525 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:05:36.525 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:05:36.525 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:36.525 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:05:36.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:05:36.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:05:36.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:36.632 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:36.632 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:36.632 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:36.632 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:36.632 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:36.632 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:37.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:37.011 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:37.012 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:37.012 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:37.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:05:37.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:05:37.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:05:37.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:05:37.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:05:37.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:05:37.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:05:37.682 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:05:37.682 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:05:38.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473074 2024-09-18T15:05:38.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473074 '!=' null 2024-09-18T15:05:38.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:05:38.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:38.178 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:05:38.178 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:38.252 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:05:38.253 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:05:38.253 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:38.253 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:38.253 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:05:38.253 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:38.253 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:05:38.253 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:05:38.253 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:38.253 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:05:38.253 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:38.253 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:38.253 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:38.254 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:38.254 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:38.254 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:05:38.254 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:05:38.254 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:38.254 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:05:38.254 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:38.254 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:38.254 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:38.254 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:38.254 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:05:38.254 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:05:38.254 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:38.254 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:38.255 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:05:38.255 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:05:38.255 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:38.255 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:05:38.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:05:38.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:05:38.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:38.361 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:38.361 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:38.361 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:38.362 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:38.362 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:38.362 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:38.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:38.874 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:38.874 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:38.874 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:39.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:05:39.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:05:39.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:05:39.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:05:39.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:05:39.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:05:39.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:05:39.381 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:05:39.381 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:05:40.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473025 2024-09-18T15:05:40.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473025 '!=' null 2024-09-18T15:05:40.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:05:40.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:40.065 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:05:40.065 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:40.140 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:05:40.140 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:05:40.140 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:40.140 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:40.140 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:05:40.140 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:40.141 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:05:40.141 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:05:40.141 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:40.141 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:05:40.141 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:40.141 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:40.141 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:40.141 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:40.141 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:40.141 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:05:40.141 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:05:40.141 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:40.141 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:05:40.141 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:40.142 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:40.142 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:40.142 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:40.142 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:05:40.142 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:05:40.142 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:40.142 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:40.142 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:05:40.142 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:05:40.142 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:40.142 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:05:40.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:05:40.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:05:40.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:40.249 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:40.249 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:40.249 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:40.249 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:40.249 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:40.249 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:40.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:40.613 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:40.613 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:40.613 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:41.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:05:41.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:05:41.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:05:41.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:05:41.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:05:41.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:05:41.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:05:41.117 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:05:41.117 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:05:41.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209137 2024-09-18T15:05:41.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209137 '!=' null 2024-09-18T15:05:41.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:05:41.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:41.782 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:05:41.783 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:41.857 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:05:41.857 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:05:41.857 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:41.857 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:41.858 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:05:41.858 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:41.858 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:05:41.858 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:05:41.858 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:41.858 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:05:41.858 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:41.858 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:41.858 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:41.858 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:41.858 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:41.859 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:05:41.859 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:05:41.859 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:41.859 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:05:41.859 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:41.859 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:41.859 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:41.859 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:41.859 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:05:41.859 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:05:41.859 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:41.859 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:41.860 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:05:41.860 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:05:41.860 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:41.860 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:05:41.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:05:41.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:05:41.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:41.966 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:41.966 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:41.966 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:41.966 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:41.966 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:41.966 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:42.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:42.328 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:42.329 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:42.329 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:42.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:05:42.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:05:42.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:05:42.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:05:42.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:05:42.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:05:42.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:05:42.844 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:05:42.844 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:05:43.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17472882 2024-09-18T15:05:43.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17472882 '!=' null 2024-09-18T15:05:43.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:05:43.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:43.550 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:05:43.550 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:43.625 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:05:43.625 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:05:43.625 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:43.625 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:43.625 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:05:43.625 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:43.626 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:05:43.626 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:05:43.626 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:43.626 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:05:43.626 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:43.626 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:43.626 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:43.626 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:43.626 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:43.626 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:05:43.626 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:05:43.626 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:43.626 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:05:43.626 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:43.627 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:43.627 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:43.627 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:43.627 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:05:43.627 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:05:43.627 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:43.627 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:43.627 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:05:43.627 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:05:43.627 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:43.627 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:05:43.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:05:43.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:05:43.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:43.734 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:43.734 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:43.734 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:43.734 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:43.734 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:43.734 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:44.096 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:44.096 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:44.097 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:44.097 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:44.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:05:44.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:05:44.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:05:44.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:05:44.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:05:44.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:05:44.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:05:44.631 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:05:44.631 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:05:45.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209280 2024-09-18T15:05:45.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209280 '!=' null 2024-09-18T15:05:45.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:05:45.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:45.132 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:05:45.132 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:45.207 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:05:45.207 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:05:45.207 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:45.207 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:45.207 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:05:45.207 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:45.207 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:05:45.207 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:05:45.207 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:45.207 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:05:45.208 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:45.208 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:45.208 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:45.208 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:45.208 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:45.208 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:05:45.208 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:05:45.208 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:45.208 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:05:45.208 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:45.208 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:45.208 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:45.208 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:45.208 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:05:45.208 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:05:45.209 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:45.209 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:45.209 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:05:45.209 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:05:45.209 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:45.209 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:05:45.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:05:45.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:05:45.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:45.315 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:45.316 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:45.316 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:45.316 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:45.316 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:45.316 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:45.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:45.685 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:45.685 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:45.685 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:46.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:05:46.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:05:46.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:05:46.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:05:46.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:05:46.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:05:46.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:05:46.353 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:05:46.353 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:05:47.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17472704 2024-09-18T15:05:47.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17472704 '!=' null 2024-09-18T15:05:47.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:05:47.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:47.094 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:05:47.094 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:47.169 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:05:47.169 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:05:47.170 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:47.170 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:47.170 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:05:47.170 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:47.170 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:05:47.170 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:05:47.170 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:47.170 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:05:47.170 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:47.170 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:47.170 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:47.170 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:47.170 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:47.170 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:05:47.171 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:05:47.171 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:47.171 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:05:47.171 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:47.171 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:47.171 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:47.171 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:47.171 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:05:47.171 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:05:47.171 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:47.171 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:47.171 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:05:47.171 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:05:47.172 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:47.172 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:05:47.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:05:47.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:05:47.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:47.278 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:47.278 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:47.278 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:47.279 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:47.279 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:47.279 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:47.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:47.647 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:47.647 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:47.647 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:48.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:05:48.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:05:48.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:05:48.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:05:48.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:05:48.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:05:48.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:05:48.215 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:05:48.215 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:05:48.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17472824 2024-09-18T15:05:48.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17472824 '!=' null 2024-09-18T15:05:48.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:05:48.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:48.775 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:05:48.775 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:48.850 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:05:48.850 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:05:48.850 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:48.850 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:48.850 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:05:48.850 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:48.850 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:05:48.850 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:05:48.851 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:48.851 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:05:48.851 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:48.851 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:48.851 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:48.851 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:48.851 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:48.851 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:05:48.851 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:05:48.851 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:48.851 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:05:48.851 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:48.851 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:48.851 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:48.851 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:48.852 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:05:48.852 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:05:48.852 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:48.852 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:48.852 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:05:48.852 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:05:48.852 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:48.852 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:05:48.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:05:48.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:05:48.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:48.959 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:48.959 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:48.959 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:48.959 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:48.959 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:48.959 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:49.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:49.322 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:49.322 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:49.322 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:49.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:05:49.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:05:49.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:05:49.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:05:49.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:05:49.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:05:49.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:05:49.846 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:05:49.846 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:05:50.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17472824 2024-09-18T15:05:50.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17472824 '!=' null 2024-09-18T15:05:50.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:05:50.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:50.597 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:05:50.598 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:50.673 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:05:50.673 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:05:50.674 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:50.674 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:50.674 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:05:50.674 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:50.674 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:05:50.674 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:05:50.674 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:50.674 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:05:50.674 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:50.674 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:50.674 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:50.674 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:50.674 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:50.674 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:05:50.675 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:05:50.675 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:50.675 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:05:50.675 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:50.675 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:50.675 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:50.675 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:50.675 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:05:50.675 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:05:50.675 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:50.675 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:50.675 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:05:50.675 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:05:50.675 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:50.675 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:05:50.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:05:50.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:05:50.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:50.783 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:50.783 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:50.783 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:50.783 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:50.783 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:50.783 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:51.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:51.175 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:51.175 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:51.176 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:51.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:05:51.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:05:51.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:05:51.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:05:51.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:05:51.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:05:51.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:05:51.779 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:05:51.779 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:05:52.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209564 2024-09-18T15:05:52.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209564 '!=' null 2024-09-18T15:05:52.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:05:52.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:52.313 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:05:52.313 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:52.389 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:05:52.389 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:05:52.389 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:52.389 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:52.389 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:05:52.389 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:52.389 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:05:52.390 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:05:52.390 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:52.390 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:05:52.390 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:52.390 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:52.390 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:52.390 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:52.390 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:52.390 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:05:52.390 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:05:52.390 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:52.390 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:05:52.390 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:52.390 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:52.391 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:52.391 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:52.391 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:05:52.391 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:05:52.391 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:52.391 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:52.391 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:05:52.391 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:05:52.391 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:52.391 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:05:52.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:05:52.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:05:52.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:52.498 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:52.498 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:52.498 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:52.498 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:52.498 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:52.498 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:52.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:52.927 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:52.927 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:52.927 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:53.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:05:53.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:05:53.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:05:53.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:05:53.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:05:53.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:05:53.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:05:53.427 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:05:53.427 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:05:54.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473380 2024-09-18T15:05:54.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473380 '!=' null 2024-09-18T15:05:54.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:05:54.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:54.092 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:05:54.092 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:54.167 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:05:54.167 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:05:54.167 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:54.167 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:54.167 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:05:54.168 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:54.168 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:05:54.168 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:05:54.168 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:54.168 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:05:54.168 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:54.168 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:54.168 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:54.168 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:54.168 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:54.168 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:05:54.168 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:05:54.168 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:54.169 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:05:54.169 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:54.169 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:54.169 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:54.169 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:54.169 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:05:54.169 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:05:54.169 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:54.169 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:54.169 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:05:54.169 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:05:54.169 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:54.169 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:05:54.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:05:54.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:05:54.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:54.276 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:54.276 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:54.276 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:54.276 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:54.276 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:54.277 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:54.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:54.680 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:54.680 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:54.680 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:55.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:05:55.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:05:55.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:05:55.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:05:55.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:05:55.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:05:55.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:05:55.335 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:05:55.335 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:05:55.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26210015 2024-09-18T15:05:55.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26210015 '!=' null 2024-09-18T15:05:55.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:05:55.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:55.970 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:05:55.970 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:56.045 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:05:56.045 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:05:56.045 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:56.045 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:56.046 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:05:56.046 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:56.046 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:05:56.046 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:05:56.046 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:56.046 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:05:56.046 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:56.046 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:56.046 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:56.046 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:56.046 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:56.047 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:05:56.047 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:05:56.047 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:56.047 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:05:56.047 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:56.047 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:56.047 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:56.047 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:56.047 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:05:56.047 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:05:56.047 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:56.047 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:56.047 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:05:56.048 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:05:56.048 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:56.048 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:05:56.052 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:05:56.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:05:56.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:56.154 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:56.154 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:56.154 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:56.154 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:56.154 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:56.154 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:56.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:56.523 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:56.523 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:56.523 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:57.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:05:57.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:05:57.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:05:57.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:05:57.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:05:57.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:05:57.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:05:57.020 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:05:57.020 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:05:57.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473369 2024-09-18T15:05:57.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473369 '!=' null 2024-09-18T15:05:57.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:05:57.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:57.522 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:05:57.522 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:57.597 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:05:57.597 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:05:57.597 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:57.597 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:57.597 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:05:57.597 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:57.597 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:05:57.597 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:05:57.597 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:57.597 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:05:57.598 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:57.598 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:57.598 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:57.598 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:57.598 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:57.598 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:05:57.598 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:05:57.598 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:57.598 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:05:57.598 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:57.598 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:57.598 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:57.598 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:57.598 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:05:57.598 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:05:57.599 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:57.599 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:57.599 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:05:57.599 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:05:57.599 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:57.599 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:05:57.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:05:57.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:05:57.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:57.706 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:57.706 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:57.706 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:57.706 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:57.706 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:57.706 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:58.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:58.071 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:58.071 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:58.071 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:05:58.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:05:58.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:05:58.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:05:58.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:05:58.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:05:58.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:05:58.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:05:58.630 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:05:58.630 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:05:59.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473506 2024-09-18T15:05:59.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473506 '!=' null 2024-09-18T15:05:59.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:05:59.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:59.365 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:05:59.365 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:05:59.441 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:05:59.442 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:05:59.442 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:59.442 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:59.442 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:05:59.442 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:59.442 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:05:59.442 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:05:59.442 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:59.442 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:05:59.442 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:59.442 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:59.442 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:59.442 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:59.443 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:59.443 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:05:59.443 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:05:59.443 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:05:59.443 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:05:59.443 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:05:59.443 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:05:59.443 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:59.443 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:05:59.443 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:05:59.443 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:05:59.443 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:05:59.443 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:05:59.443 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:05:59.444 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:05:59.444 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:05:59.444 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:05:59.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:05:59.549 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:05:59.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:05:59.550 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:05:59.550 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:05:59.551 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:05:59.551 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:05:59.551 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:05:59.551 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:05:59.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:05:59.910 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:05:59.911 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:05:59.911 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:00.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:06:00.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:06:00.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:06:00.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:06:00.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:06:00.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:06:00.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:06:00.518 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:06:00.518 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:06:01.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25336642 2024-09-18T15:06:01.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25336642 '!=' null 2024-09-18T15:06:01.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:06:01.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:01.038 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:06:01.038 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:01.112 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:06:01.112 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:06:01.112 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:01.112 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:01.112 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:06:01.112 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:01.112 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:06:01.112 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:06:01.112 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:01.113 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:06:01.113 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:01.113 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:01.113 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:01.113 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:01.113 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:01.113 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:06:01.113 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:06:01.113 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:01.113 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:06:01.113 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:01.113 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:01.113 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:01.113 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:01.114 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:06:01.114 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:06:01.114 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:01.114 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:01.114 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:01.114 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:06:01.114 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:01.114 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:06:01.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:06:01.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:06:01.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:01.221 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:01.221 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:01.221 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:01.221 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:01.221 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:01.221 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:01.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:06:01.736 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:01.736 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:01.737 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:02.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:06:02.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:06:02.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:06:02.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:06:02.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:06:02.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:06:02.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:06:02.359 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:06:02.359 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:06:03.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599883 2024-09-18T15:06:03.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599883 '!=' null 2024-09-18T15:06:03.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:06:03.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:03.129 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:06:03.129 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:03.204 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:06:03.204 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:06:03.204 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:03.204 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:03.204 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:06:03.204 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:03.204 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:06:03.204 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:06:03.205 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:03.205 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:06:03.205 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:03.205 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:03.205 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:03.205 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:03.205 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:03.205 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:06:03.205 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:06:03.205 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:03.205 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:06:03.205 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:03.205 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:03.206 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:03.206 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:03.206 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:06:03.206 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:06:03.206 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:03.206 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:03.206 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:03.206 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:06:03.206 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:03.206 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:06:03.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:06:03.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:06:03.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:03.313 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:03.313 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:03.313 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:03.313 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:03.313 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:03.313 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:03.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:06:03.682 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:03.682 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:03.682 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:04.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:06:04.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:06:04.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:06:04.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:06:04.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:06:04.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:06:04.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:06:04.234 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:06:04.234 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:06:04.861 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25336610 2024-09-18T15:06:04.861 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25336610 '!=' null 2024-09-18T15:06:04.861 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:06:04.861 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:04.861 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:06:04.861 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:04.937 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:06:04.937 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:06:04.937 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:04.937 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:04.937 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:06:04.938 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:04.938 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:06:04.938 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:06:04.938 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:04.938 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:06:04.938 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:04.939 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:04.939 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:04.939 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:04.939 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:04.939 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:06:04.939 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:06:04.939 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:04.940 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:06:04.940 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:04.940 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:04.940 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:04.940 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:04.940 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:06:04.941 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:06:04.941 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:04.941 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:04.941 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:04.941 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:06:04.941 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:04.942 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:06:04.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:06:05.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:06:05.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:05.046 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:05.046 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:05.046 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:05.046 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:05.046 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:05.046 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:05.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:06:05.497 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:05.497 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:05.497 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:06.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:06:06.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:06:06.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:06:06.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:06:06.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:06:06.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:06:06.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:06:06.008 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:06:06.008 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:06:06.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599895 2024-09-18T15:06:06.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599895 '!=' null 2024-09-18T15:06:06.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:06:06.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:06.616 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:06:06.616 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:06.691 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:06:06.691 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:06:06.691 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:06.691 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:06.691 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:06:06.691 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:06.691 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:06:06.692 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:06:06.692 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:06.692 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:06:06.692 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:06.692 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:06.692 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:06.692 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:06.692 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:06.692 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:06:06.692 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:06:06.692 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:06.692 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:06:06.692 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:06.693 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:06.693 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:06.693 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:06.693 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:06:06.693 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:06:06.693 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:06.693 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:06.693 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:06.693 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:06:06.693 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:06.693 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:06:06.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:06:06.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:06:06.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:06.800 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:06.800 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:06.800 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:06.800 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:06.800 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:06.800 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:07.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:06:07.180 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:07.181 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:07.181 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:07.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:06:07.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:06:07.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:06:07.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:06:07.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:06:07.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:06:07.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:06:07.764 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:06:07.764 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:06:08.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599895 2024-09-18T15:06:08.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599895 '!=' null 2024-09-18T15:06:08.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:06:08.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:08.350 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:06:08.350 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:08.425 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:06:08.425 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:06:08.425 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:08.425 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:08.425 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:06:08.426 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:08.426 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:06:08.426 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:06:08.426 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:08.426 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:06:08.426 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:08.426 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:08.426 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:08.426 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:08.426 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:08.426 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:06:08.426 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:06:08.426 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:08.426 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:06:08.427 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:08.427 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:08.427 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:08.427 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:08.427 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:06:08.427 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:06:08.427 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:08.427 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:08.427 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:08.427 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:06:08.427 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:08.427 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:06:08.432 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:06:08.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:06:08.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:08.535 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:08.535 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:08.535 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:08.535 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:08.535 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:08.535 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:08.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:06:08.987 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:08.988 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:08.988 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:09.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:06:09.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:06:09.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:06:09.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:06:09.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:06:09.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:06:09.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:06:09.513 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:06:09.513 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:06:10.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599709 2024-09-18T15:06:10.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599709 '!=' null 2024-09-18T15:06:10.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:06:10.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:10.071 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:06:10.072 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:10.146 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:06:10.146 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:06:10.146 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:10.146 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:10.146 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:06:10.146 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:10.147 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:06:10.147 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:06:10.147 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:10.147 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:06:10.147 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:10.147 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:10.147 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:10.147 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:10.147 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:10.147 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:06:10.147 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:06:10.147 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:10.147 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:06:10.147 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:10.148 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:10.148 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:10.148 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:10.148 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:06:10.148 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:06:10.148 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:10.148 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:10.148 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:10.148 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:06:10.148 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:10.148 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:06:10.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:06:10.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:06:10.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:10.255 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:10.255 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:10.255 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:10.255 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:10.255 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:10.256 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:10.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:06:10.644 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:10.644 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:10.644 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:11.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:06:11.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:06:11.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:06:11.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:06:11.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:06:11.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:06:11.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:06:11.328 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:06:11.328 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:06:11.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25336334 2024-09-18T15:06:11.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25336334 '!=' null 2024-09-18T15:06:11.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:06:11.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:11.859 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:06:11.859 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:11.934 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:06:11.934 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:06:11.935 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:11.935 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:11.935 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:06:11.935 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:11.935 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:06:11.935 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:06:11.935 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:11.935 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:06:11.935 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:11.935 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:11.935 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:11.935 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:11.936 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:11.936 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:06:11.936 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:06:11.936 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:11.936 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:06:11.936 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:11.936 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:11.936 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:11.936 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:11.936 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:06:11.936 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:06:11.936 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:11.936 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:11.936 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:11.937 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:06:11.937 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:11.937 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:06:11.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:06:12.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:06:12.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:12.043 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:12.043 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:12.043 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:12.043 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:12.043 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:12.043 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:12.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:06:12.446 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:12.447 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:12.447 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:13.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:06:13.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:06:13.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:06:13.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:06:13.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:06:13.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:06:13.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:06:13.017 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:06:13.017 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:06:13.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473109 2024-09-18T15:06:13.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473109 '!=' null 2024-09-18T15:06:13.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:06:13.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:13.578 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:06:13.578 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:13.652 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:06:13.652 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:06:13.652 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:13.653 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:13.653 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:06:13.653 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:13.653 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:06:13.653 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:06:13.653 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:13.653 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:06:13.653 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:13.653 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:13.653 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:13.653 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:13.653 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:13.653 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:06:13.653 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:06:13.654 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:13.654 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:06:13.654 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:13.654 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:13.654 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:13.654 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:13.654 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:06:13.654 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:06:13.654 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:13.654 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:13.654 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:13.654 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:06:13.654 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:13.654 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:06:13.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:06:13.760 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:06:13.760 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:13.761 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:13.761 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:13.761 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:13.761 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:13.761 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:13.762 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:14.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:06:14.122 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:14.122 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:14.122 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:14.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:06:14.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:06:14.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:06:14.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:06:14.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:06:14.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:06:14.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:06:14.620 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:06:14.620 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:06:15.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25335786 2024-09-18T15:06:15.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25335786 '!=' null 2024-09-18T15:06:15.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:06:15.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:15.301 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:06:15.301 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:15.377 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:06:15.377 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:06:15.377 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:15.377 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:15.377 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:06:15.377 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:15.377 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:06:15.378 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:06:15.378 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:15.378 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:06:15.378 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:15.378 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:15.378 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:15.378 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:15.378 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:15.378 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:06:15.378 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:06:15.378 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:15.378 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:06:15.378 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:15.378 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:15.380 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:15.380 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:15.380 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:06:15.380 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:06:15.380 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:15.380 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:15.380 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:15.380 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:06:15.380 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:15.380 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:06:15.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:06:15.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:06:15.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:15.486 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:15.486 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:15.486 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:15.486 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:15.486 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:15.486 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:15.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:06:15.847 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:15.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:15.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:16.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:06:16.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:06:16.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:06:16.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:06:16.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:06:16.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:06:16.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:06:16.449 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:06:16.449 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:06:17.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599275 2024-09-18T15:06:17.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599275 '!=' null 2024-09-18T15:06:17.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:06:17.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:17.024 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:06:17.024 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:17.099 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:06:17.100 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:06:17.100 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:17.100 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:17.100 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:06:17.100 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:17.100 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:06:17.100 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:06:17.100 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:17.100 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:06:17.100 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:17.100 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:17.101 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:17.101 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:17.101 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:17.101 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:06:17.101 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:06:17.101 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:17.101 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:06:17.101 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:17.101 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:17.101 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:17.101 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:17.101 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:06:17.101 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:06:17.102 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:17.102 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:17.102 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:17.102 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:06:17.102 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:17.102 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:06:17.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:06:17.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:06:17.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:17.209 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:17.209 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:17.209 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:17.209 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:17.209 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:17.209 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:17.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:06:17.567 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:17.568 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:17.568 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:18.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:06:18.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:06:18.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:06:18.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:06:18.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:06:18.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:06:18.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:06:18.147 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:06:18.147 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:06:18.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599215 2024-09-18T15:06:18.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599215 '!=' null 2024-09-18T15:06:18.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:06:18.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:18.942 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:06:18.942 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:19.018 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:06:19.019 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:06:19.019 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:19.019 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:19.019 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:06:19.019 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:19.019 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:06:19.019 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:06:19.019 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:19.019 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:06:19.019 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:19.019 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:19.019 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:19.019 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:19.019 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:19.020 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:06:19.020 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:06:19.020 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:19.020 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:06:19.020 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:19.020 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:19.020 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:19.020 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:19.020 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:06:19.020 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:06:19.020 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:19.020 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:19.020 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:19.020 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:06:19.021 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:19.021 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:06:19.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:06:19.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:06:19.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:19.127 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:19.127 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:19.127 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:19.127 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:19.127 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:19.127 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:19.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:06:19.495 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:19.496 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:19.496 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:20.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:06:20.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:06:20.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:06:20.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:06:20.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:06:20.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:06:20.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:06:20.165 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:06:20.165 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:06:20.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25335970 2024-09-18T15:06:20.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25335970 '!=' null 2024-09-18T15:06:20.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:06:20.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:20.764 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:06:20.764 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:20.839 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:06:20.839 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:06:20.840 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:20.840 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:20.840 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:06:20.840 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:20.840 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:06:20.840 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:06:20.840 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:20.840 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:06:20.841 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:20.841 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:20.841 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:20.841 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:20.841 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:20.841 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:06:20.841 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:06:20.841 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:20.841 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:06:20.841 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:20.842 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:20.842 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:20.842 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:20.842 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:06:20.842 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:06:20.842 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:20.842 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:20.842 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:20.842 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:06:20.842 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:20.842 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:06:20.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:06:20.947 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:06:20.947 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:20.948 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:20.948 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:20.948 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:20.948 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:20.948 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:20.948 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:21.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:06:21.315 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:21.315 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:21.316 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:21.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:06:21.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:06:21.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:06:21.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:06:21.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:06:21.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:06:21.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:06:21.814 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:06:21.814 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:06:22.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25335970 2024-09-18T15:06:22.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25335970 '!=' null 2024-09-18T15:06:22.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:06:22.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:22.428 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:06:22.428 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:22.503 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:06:22.504 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:06:22.504 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:22.504 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:22.504 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:06:22.504 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:22.504 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:06:22.504 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:06:22.504 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:22.504 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:06:22.504 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:22.504 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:22.504 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:22.505 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:22.505 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:22.505 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:06:22.505 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:06:22.505 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:22.505 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:06:22.505 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:22.505 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:22.505 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:22.505 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:22.505 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:06:22.505 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:06:22.505 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:22.506 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:22.506 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:22.506 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:06:22.506 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:22.506 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:06:22.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:06:22.612 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:06:22.612 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:22.613 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:22.613 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:22.613 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:22.613 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:22.613 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:22.613 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:22.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:06:22.976 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:22.976 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:22.977 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:23.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:06:23.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:06:23.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:06:23.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:06:23.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:06:23.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:06:23.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:06:23.554 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:06:23.555 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:06:24.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599448 2024-09-18T15:06:24.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599448 '!=' null 2024-09-18T15:06:24.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:06:24.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:24.160 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:06:24.160 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:24.235 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:06:24.236 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:06:24.236 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:24.236 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:24.236 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:06:24.236 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:24.236 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:06:24.236 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:06:24.236 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:24.236 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:06:24.237 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:24.237 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:24.237 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:24.237 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:24.237 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:24.237 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:06:24.237 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:06:24.237 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:24.237 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:06:24.237 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:24.237 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:24.238 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:24.238 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:24.238 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:06:24.238 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:06:24.238 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:24.238 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:24.238 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:24.238 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:06:24.238 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:24.238 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:06:24.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:06:24.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:06:24.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:24.345 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:24.345 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:24.345 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:24.345 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:24.345 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:24.345 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:24.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:06:24.712 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:24.713 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:24.713 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:25.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:06:25.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:06:25.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:06:25.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:06:25.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:06:25.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:06:25.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:06:25.255 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:06:25.255 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:06:25.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25336056 2024-09-18T15:06:25.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25336056 '!=' null 2024-09-18T15:06:25.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:06:25.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:25.852 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:06:25.853 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:25.925 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:06:25.926 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:06:25.926 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:25.926 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:25.926 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:06:25.926 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:25.926 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:06:25.926 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:06:25.926 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:25.926 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:06:25.926 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:25.927 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:25.927 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:25.927 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:25.927 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:25.927 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:06:25.927 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:06:25.927 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:25.927 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:06:25.927 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:25.927 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:25.927 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:25.927 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:25.927 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:06:25.927 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:06:25.928 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:25.928 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:25.928 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:25.928 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:06:25.928 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:25.928 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:06:25.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:06:26.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:06:26.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:26.034 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:26.035 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:26.035 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:26.035 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:26.035 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:26.035 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:26.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:06:26.396 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:26.397 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:26.397 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:27.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:06:27.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:06:27.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:06:27.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:06:27.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:06:27.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:06:27.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:06:27.053 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:06:27.053 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:06:27.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473166 2024-09-18T15:06:27.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473166 '!=' null 2024-09-18T15:06:27.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:06:27.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:27.630 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:06:27.630 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:27.705 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:06:27.705 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:06:27.706 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:27.706 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:27.706 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:06:27.706 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:27.706 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:06:27.706 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:06:27.706 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:27.706 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:06:27.706 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:27.706 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:27.706 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:27.706 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:27.706 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:27.707 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:06:27.707 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:06:27.707 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:27.707 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:06:27.707 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:27.707 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:27.707 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:27.707 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:27.707 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:06:27.707 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:06:27.707 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:27.707 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:27.707 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:27.707 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:06:27.707 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:27.708 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:06:27.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:06:27.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:06:27.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:27.814 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:27.814 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:27.814 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:27.814 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:27.815 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:27.815 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:28.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:06:28.196 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:28.196 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:28.196 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:28.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:06:28.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:06:28.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:06:28.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:06:28.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:06:28.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:06:28.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:06:28.779 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:06:28.779 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:06:29.390 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473180 2024-09-18T15:06:29.390 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473180 '!=' null 2024-09-18T15:06:29.390 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:06:29.390 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:29.390 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:06:29.390 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:29.465 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:06:29.465 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:06:29.465 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:29.465 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:29.465 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:06:29.465 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:29.466 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:06:29.466 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:06:29.466 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:29.466 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:06:29.466 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:29.466 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:29.466 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:29.466 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:29.466 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:29.466 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:06:29.466 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:06:29.466 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:29.466 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:06:29.467 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:29.467 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:29.467 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:29.467 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:29.467 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:06:29.467 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:06:29.467 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:29.467 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:29.467 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:29.467 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:06:29.467 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:29.467 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:06:29.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:06:29.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:06:29.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:29.574 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:29.574 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:29.574 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:29.574 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:29.574 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:29.574 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:29.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:06:29.939 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:29.939 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:29.939 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:30.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:06:30.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:06:30.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:06:30.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:06:30.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:06:30.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:06:30.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:06:30.443 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:06:30.444 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:06:31.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209841 2024-09-18T15:06:31.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209841 '!=' null 2024-09-18T15:06:31.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:06:31.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:31.163 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:06:31.163 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:31.237 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:06:31.237 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:06:31.237 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:31.237 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:31.237 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:06:31.238 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:31.238 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:06:31.238 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:06:31.238 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:31.238 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:06:31.238 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:31.238 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:31.238 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:31.238 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:31.238 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:31.238 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:06:31.238 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:06:31.238 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:31.239 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:06:31.239 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:31.239 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:31.239 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:31.239 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:31.239 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:06:31.239 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:06:31.239 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:31.239 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:31.239 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:31.239 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:06:31.239 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:31.239 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:06:31.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:06:31.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:06:31.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:31.346 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:31.346 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:31.346 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:31.346 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:31.346 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:31.346 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:31.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:06:31.711 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:31.711 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:31.711 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:32.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:06:32.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:06:32.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:06:32.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:06:32.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:06:32.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:06:32.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:06:32.233 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:06:32.234 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:06:32.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473143 2024-09-18T15:06:32.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473143 '!=' null 2024-09-18T15:06:32.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:06:32.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:32.852 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:06:32.852 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:32.930 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:06:32.931 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:06:32.931 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:32.931 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:32.931 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:06:32.931 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:32.931 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:06:32.931 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:06:32.931 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:32.931 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:06:32.931 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:32.931 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:32.931 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:32.932 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:32.932 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:32.932 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:06:32.932 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:06:32.932 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:32.932 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:06:32.932 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:32.932 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:32.932 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:32.932 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:32.932 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:06:32.932 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:06:32.932 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:32.932 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:32.933 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:32.933 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:06:32.933 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:32.933 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:06:32.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:06:33.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:06:33.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:33.041 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:33.041 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:33.041 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:33.041 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:33.041 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:33.042 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:33.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:06:33.404 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:33.404 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:33.404 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:33.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:06:33.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:06:33.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:06:33.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:06:33.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:06:33.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:06:33.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:06:33.931 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:06:33.931 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:06:34.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473143 2024-09-18T15:06:34.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473143 '!=' null 2024-09-18T15:06:34.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:06:34.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:34.475 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:06:34.475 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:34.549 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:06:34.549 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:06:34.549 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:34.549 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:34.549 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:06:34.549 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:34.549 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:06:34.549 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:06:34.550 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:34.550 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:06:34.550 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:34.550 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:34.550 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:34.550 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:34.550 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:34.550 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:06:34.551 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:06:34.551 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:34.551 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:06:34.551 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:34.551 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:34.551 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:34.551 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:34.551 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:06:34.551 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:06:34.552 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:34.552 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:34.552 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:34.552 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:06:34.552 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:34.552 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:06:34.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:06:34.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:06:34.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:34.658 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:34.658 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:34.658 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:34.658 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:34.658 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:34.658 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:35.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:06:35.021 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:35.021 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:35.021 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:35.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:06:35.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:06:35.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:06:35.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:06:35.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:06:35.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:06:35.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:06:35.569 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:06:35.569 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:06:36.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209724 2024-09-18T15:06:36.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209724 '!=' null 2024-09-18T15:06:36.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:06:36.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:36.094 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:06:36.094 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:36.169 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:06:36.169 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:06:36.169 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:36.169 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:36.169 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [ 2024-09-18T15:06:36.169 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:36.169 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 190, 2024-09-18T15:06:36.169 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:06:36.169 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:36.169 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:06:36.170 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:36.170 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:36.170 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:36.170 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:36.170 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:36.170 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:06:36.170 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:06:36.170 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:36.170 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "1.0", 2024-09-18T15:06:36.170 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:36.170 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:36.170 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:36.171 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:36.171 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:06:36.171 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:06:36.171 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:36.171 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:36.171 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:36.171 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:06:36.171 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:36.171 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:06:36.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:06:36.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:06:36.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:36.278 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:36.278 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:36.278 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:36.278 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:36.278 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:36.278 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:36.642 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:06:36.642 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:36.643 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:36.643 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:37.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:06:37.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:06:37.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:06:37.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:06:37.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:06:37.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:06:37.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:06:37.245 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:06:37.245 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:06:37.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473299 2024-09-18T15:06:37.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473299 '!=' null 2024-09-18T15:06:37.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:06:37.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:37.812 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:06:37.812 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:37.887 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:06:37.887 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:06:37.887 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:37.887 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:37.887 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:37.887 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:06:37.887 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:37.887 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:06:37.887 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:37.888 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:37.888 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:37.888 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:37.888 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:06:37.888 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:06:37.888 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:37.888 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:37.888 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:37.888 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:06:37.888 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:37.888 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:06:37.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:06:37.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:06:37.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:37.996 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:37.996 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:37.996 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:37.996 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:37.996 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:37.996 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:38.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:06:38.440 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:38.440 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:38.440 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:39.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:06:39.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:06:39.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:06:39.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:06:39.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:06:39.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:06:39.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:06:39.296 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:06:39.296 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:06:39.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20968067 2024-09-18T15:06:39.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20968067 '!=' null 2024-09-18T15:06:39.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:06:39.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:39.866 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:06:39.866 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:39.941 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:06:39.941 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:06:39.941 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:39.941 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:39.941 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:39.942 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:06:39.942 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:39.942 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:06:39.942 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:39.942 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:39.942 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:39.942 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:39.942 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:06:39.942 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:06:39.942 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:39.942 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:39.942 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:39.942 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:06:39.942 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:39.942 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:06:39.947 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:06:40.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:06:40.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:40.050 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:40.050 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:40.050 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:40.050 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:40.050 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:40.050 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:40.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:06:40.421 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:40.422 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:40.422 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:40.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:06:40.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:06:40.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:06:40.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:06:40.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:06:40.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:06:40.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:06:40.978 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:06:40.978 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:06:41.514 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24899436 2024-09-18T15:06:41.514 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24899436 '!=' null 2024-09-18T15:06:41.514 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:06:41.514 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:41.514 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:06:41.514 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:41.588 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:06:41.588 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:06:41.588 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:41.588 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:41.588 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:41.589 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:06:41.589 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:41.589 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:06:41.589 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:41.589 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:41.589 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:41.589 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:41.589 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:06:41.589 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:06:41.589 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:41.589 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:41.589 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:41.589 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:06:41.589 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:41.590 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:06:41.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:06:41.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:06:41.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:41.697 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:41.697 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:41.697 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:41.697 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:41.697 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:41.698 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:42.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:06:42.066 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:42.066 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:42.066 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:42.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:06:42.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:06:42.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:06:42.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:06:42.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:06:42.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:06:42.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:06:42.575 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:06:42.576 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:06:43.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11794357 2024-09-18T15:06:43.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11794357 '!=' null 2024-09-18T15:06:43.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:06:43.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:43.243 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:06:43.243 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:43.319 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:06:43.319 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:06:43.319 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:43.319 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:43.319 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:43.319 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:06:43.319 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:43.320 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:06:43.320 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:43.320 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:43.320 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:43.320 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:43.320 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:06:43.320 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:06:43.320 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:43.320 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:43.320 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:43.320 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:06:43.320 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:43.320 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:06:43.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:06:43.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:06:43.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:43.428 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:43.428 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:43.428 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:43.428 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:43.428 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:43.428 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:43.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:06:43.793 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:43.794 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:43.794 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:44.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-18T15:06:44.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-18T15:06:44.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:06:44.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:06:44.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:06:44.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:06:44.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:06:44.366 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:06:44.366 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:06:44.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27520100 2024-09-18T15:06:44.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27520100 '!=' null 2024-09-18T15:06:44.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:06:44.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:44.918 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:06:44.919 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:44.993 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:06:44.993 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:06:44.993 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:44.993 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:44.993 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:44.993 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:06:44.993 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:44.993 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:06:44.993 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:44.994 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:44.994 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:44.994 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:44.994 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:06:44.994 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:06:44.994 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:44.994 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:44.994 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:44.994 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:06:44.994 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:44.994 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:06:45.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:06:45.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:06:45.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:45.102 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:45.102 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:45.102 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:45.102 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:45.102 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:45.102 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:45.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:06:45.461 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:45.462 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:45.462 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:45.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:06:45.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 1 2024-09-18T15:06:45.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T15:06:45.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-09-18T15:06:45.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:45.962 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:06:45.962 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:46.037 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:06:46.037 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:06:46.037 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:46.037 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:46.038 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:46.038 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:06:46.038 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:46.038 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:06:46.038 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:46.038 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:46.038 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:46.038 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:46.038 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:06:46.038 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:06:46.038 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:46.038 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:46.038 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:46.038 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:06:46.038 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:46.039 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:06:46.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:06:46.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:06:46.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:46.147 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:46.147 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:46.147 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:46.147 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:46.147 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:46.147 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:46.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:06:46.604 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:46.605 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:46.605 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:47.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:06:47.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:06:47.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:06:47.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:06:47.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:06:47.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:06:47.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:06:47.101 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:06:47.101 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:06:47.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25362043 2024-09-18T15:06:47.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25362043 '!=' null 2024-09-18T15:06:47.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:06:47.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:47.788 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:06:47.788 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:47.863 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:06:47.864 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:06:47.864 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:47.864 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:47.864 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:47.864 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:06:47.864 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:47.864 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:06:47.864 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:47.864 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:47.864 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:47.864 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:47.865 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:06:47.865 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:06:47.865 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:47.865 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:47.865 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:47.865 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:06:47.865 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:47.865 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:06:47.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:06:47.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:06:47.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:47.972 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:47.972 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:47.973 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:47.973 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:47.973 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:47.973 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:48.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:06:48.335 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:48.335 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:48.335 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:48.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:06:48.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:06:48.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:06:48.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:06:48.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:06:48.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:06:48.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:06:48.901 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:06:48.901 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:06:49.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22015895 2024-09-18T15:06:49.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22015895 '!=' null 2024-09-18T15:06:49.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:06:49.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:49.461 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:06:49.461 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:49.535 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:06:49.535 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:06:49.535 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:49.535 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:49.535 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:49.536 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:06:49.536 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:49.536 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:06:49.536 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:49.536 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:49.536 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:49.536 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:49.536 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:06:49.536 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:06:49.536 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:49.536 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:49.536 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:49.536 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:06:49.537 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:49.537 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:06:49.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:06:49.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:06:49.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:49.644 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:49.644 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:49.644 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:49.644 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:49.644 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:49.644 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:50.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:06:50.005 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:50.006 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:50.006 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:50.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:06:50.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:06:50.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:06:50.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:06:50.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:06:50.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:06:50.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:06:50.514 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:06:50.514 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:06:51.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27350036 2024-09-18T15:06:51.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27350036 '!=' null 2024-09-18T15:06:51.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:06:51.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:51.092 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:06:51.092 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:51.167 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:06:51.167 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:06:51.167 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:51.167 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:51.167 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:51.167 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:06:51.168 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:51.168 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:06:51.168 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:51.168 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:51.168 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:51.168 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:51.168 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:06:51.168 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:06:51.169 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:51.169 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:51.169 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:51.169 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:06:51.169 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:51.169 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:06:51.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:06:51.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:06:51.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:51.276 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:51.276 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:51.276 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:51.276 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:51.276 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:51.276 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:51.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:06:51.637 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:51.637 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:51.638 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:52.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:06:52.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:06:52.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:06:52.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:06:52.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:06:52.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:06:52.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:06:52.273 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:06:52.273 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:06:52.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=19220559 2024-09-18T15:06:52.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 19220559 '!=' null 2024-09-18T15:06:52.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:06:52.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:52.981 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:06:52.981 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:53.056 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:06:53.056 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:06:53.056 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:53.056 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:53.057 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:53.057 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:06:53.057 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:53.057 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:06:53.057 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:53.057 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:53.057 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:53.057 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:53.057 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:06:53.057 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:06:53.057 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:53.057 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:53.058 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:53.058 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:06:53.058 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:53.058 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:06:53.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:06:53.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:06:53.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:53.165 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:53.165 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:53.165 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:53.165 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:53.165 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:53.165 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:53.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:06:53.532 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:53.532 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:53.532 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:54.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:06:54.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:06:54.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:06:54.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:06:54.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:06:54.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:06:54.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:06:54.114 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:06:54.114 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:06:54.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27083732 2024-09-18T15:06:54.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27083732 '!=' null 2024-09-18T15:06:54.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:06:54.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:54.665 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:06:54.665 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:54.739 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:06:54.739 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:06:54.740 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:54.740 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:54.740 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:54.740 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:06:54.740 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:54.740 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:06:54.740 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:54.740 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:54.740 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:54.740 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:54.740 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:06:54.740 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:06:54.741 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:54.741 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:54.741 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:54.741 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:06:54.741 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:54.741 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:06:54.746 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:06:54.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:06:54.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:54.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:54.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:54.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:54.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:54.849 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:54.849 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:55.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:06:55.215 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:55.216 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:55.216 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:55.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:06:55.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:06:55.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:06:55.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:06:55.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:06:55.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:06:55.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:06:55.712 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:06:55.712 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:06:56.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27083732 2024-09-18T15:06:56.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27083732 '!=' null 2024-09-18T15:06:56.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:06:56.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:56.381 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:06:56.381 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:56.457 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:06:56.457 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:06:56.457 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:56.457 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:56.457 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:56.457 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:06:56.457 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:56.458 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:06:56.458 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:56.458 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:56.458 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:56.458 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:56.458 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:06:56.458 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:06:56.458 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:56.458 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:56.458 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:56.458 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:06:56.458 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:56.458 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:06:56.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:06:56.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:06:56.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:56.566 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:56.566 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:56.566 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:56.566 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:56.566 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:56.566 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:56.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:06:56.995 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:56.996 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:56.996 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:57.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:06:57.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:06:57.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:06:57.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:06:57.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:06:57.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:06:57.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:06:57.500 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:06:57.500 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:06:58.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599670 2024-09-18T15:06:58.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599670 '!=' null 2024-09-18T15:06:58.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:06:58.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:58.171 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:06:58.171 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:06:58.246 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:06:58.246 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:06:58.246 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:58.246 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:58.246 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:58.246 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:06:58.246 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:06:58.246 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:06:58.246 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:06:58.246 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:06:58.246 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:58.246 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:06:58.247 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:06:58.247 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:06:58.247 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:06:58.247 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:06:58.247 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:06:58.247 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:06:58.247 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:06:58.247 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:06:58.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:06:58.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:06:58.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:06:58.354 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:06:58.355 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:06:58.355 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:06:58.355 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:06:58.355 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:06:58.355 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:06:58.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:06:58.854 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:06:58.854 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:06:58.854 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:06:59.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:06:59.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:06:59.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:06:59.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:06:59.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:06:59.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:06:59.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:06:59.353 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:06:59.353 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:07:00.079 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599602 2024-09-18T15:07:00.079 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599602 '!=' null 2024-09-18T15:07:00.079 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:07:00.079 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:00.079 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:07:00.079 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:00.154 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:07:00.154 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:07:00.154 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:00.154 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:00.154 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:00.154 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:07:00.154 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:07:00.154 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:07:00.155 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:07:00.155 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:07:00.155 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:00.155 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:07:00.155 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:07:00.155 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:07:00.155 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:00.155 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:00.155 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:00.155 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:07:00.155 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:00.155 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:07:00.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:07:00.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:07:00.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:07:00.263 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:07:00.263 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:07:00.263 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:07:00.263 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:07:00.263 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:07:00.263 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:07:00.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:07:00.625 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:07:00.626 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:07:00.626 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:07:01.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:07:01.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:07:01.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:07:01.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:07:01.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:07:01.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:07:01.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:07:01.122 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:07:01.123 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:07:01.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25336415 2024-09-18T15:07:01.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25336415 '!=' null 2024-09-18T15:07:01.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:07:01.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:01.758 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:07:01.758 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:01.832 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:07:01.832 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:07:01.832 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:01.832 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:01.832 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:01.832 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:07:01.833 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:07:01.833 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:07:01.833 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:07:01.833 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:07:01.833 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:01.833 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:07:01.833 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:07:01.833 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:07:01.833 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:01.833 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:01.833 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:01.833 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:07:01.833 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:01.834 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:07:01.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:07:01.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:07:01.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:07:01.941 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:07:01.941 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:07:01.941 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:07:01.941 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:07:01.942 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:07:01.942 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:07:02.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:07:02.308 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:07:02.308 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:07:02.308 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:07:02.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:07:02.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:07:02.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:07:02.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:07:02.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:07:02.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:07:02.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:07:02.831 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:07:02.831 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:07:03.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599533 2024-09-18T15:07:03.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599533 '!=' null 2024-09-18T15:07:03.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:07:03.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:03.534 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:07:03.534 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:03.608 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:07:03.609 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:07:03.609 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:03.609 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:03.609 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:03.609 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:07:03.609 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:07:03.609 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:07:03.609 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:07:03.609 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:07:03.609 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:03.609 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:07:03.609 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:07:03.610 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:07:03.610 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:03.610 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:03.610 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:03.610 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:07:03.610 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:03.610 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:07:03.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:07:03.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:07:03.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:07:03.717 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:07:03.718 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:07:03.718 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:07:03.718 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:07:03.718 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:07:03.718 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:07:04.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:07:04.082 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:07:04.082 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:07:04.082 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:07:04.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:07:04.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:07:04.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:07:04.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:07:04.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:07:04.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:07:04.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:07:04.729 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:07:04.730 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:07:05.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25336142 2024-09-18T15:07:05.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25336142 '!=' null 2024-09-18T15:07:05.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:07:05.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:05.340 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:07:05.341 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:05.415 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:07:05.416 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:07:05.416 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:05.416 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:05.416 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:05.416 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:07:05.416 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:07:05.416 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:07:05.416 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:07:05.416 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:07:05.416 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:05.416 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:07:05.416 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:07:05.417 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:07:05.417 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:05.417 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:05.417 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:05.417 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:07:05.417 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:05.417 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:07:05.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:07:05.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:07:05.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:07:05.525 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:07:05.525 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:07:05.525 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:07:05.525 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:07:05.525 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:07:05.525 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:07:05.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:07:05.887 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:07:05.887 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:07:05.887 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:07:06.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:07:06.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:07:06.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:07:06.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:07:06.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:07:06.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:07:06.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:07:06.388 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:07:06.388 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:07:07.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473181 2024-09-18T15:07:07.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473181 '!=' null 2024-09-18T15:07:07.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:07:07.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:07.036 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:07:07.036 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:07.110 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:07:07.110 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:07:07.110 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:07.110 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:07.110 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:07.110 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:07:07.110 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:07:07.110 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:07:07.111 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:07:07.111 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:07:07.111 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:07.111 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:07:07.111 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:07:07.111 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:07:07.111 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:07.111 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:07.111 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:07.111 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:07:07.111 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:07.111 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:07:07.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:07:07.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:07:07.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:07:07.219 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:07:07.219 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:07:07.219 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:07:07.219 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:07:07.219 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:07:07.219 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:07:07.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:07:07.580 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:07:07.581 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:07:07.581 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:07:08.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:07:08.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:07:08.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:07:08.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:07:08.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:07:08.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:07:08.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:07:08.159 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:07:08.160 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:07:08.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473268 2024-09-18T15:07:08.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473268 '!=' null 2024-09-18T15:07:08.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:07:08.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:08.736 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:07:08.736 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:08.812 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:07:08.812 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:07:08.812 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:08.812 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:08.812 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:08.812 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:07:08.812 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:07:08.812 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:07:08.813 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:07:08.813 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:07:08.813 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:08.813 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:07:08.813 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:07:08.813 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:07:08.813 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:08.813 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:08.813 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:08.813 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:07:08.813 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:08.813 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:07:08.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:07:08.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:07:08.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:07:08.921 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:07:08.921 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:07:08.921 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:07:08.921 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:07:08.921 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:07:08.921 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:07:09.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:07:09.287 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:07:09.287 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:07:09.287 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:07:09.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:07:09.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:07:09.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:07:09.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:07:09.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:07:09.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:07:09.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:07:09.789 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:07:09.789 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:07:10.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473268 2024-09-18T15:07:10.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473268 '!=' null 2024-09-18T15:07:10.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:07:10.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:10.468 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:07:10.468 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:10.542 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:07:10.542 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:07:10.543 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:10.543 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:10.543 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:10.543 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:07:10.543 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:07:10.543 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:07:10.543 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:07:10.543 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:07:10.543 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:10.543 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:07:10.543 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:07:10.543 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:07:10.543 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:10.543 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:10.544 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:10.544 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:07:10.544 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:10.544 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:07:10.549 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:07:10.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:07:10.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:07:10.651 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:07:10.651 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:07:10.651 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:07:10.651 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:07:10.652 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:07:10.652 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:07:11.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:07:11.057 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:07:11.057 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:07:11.057 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:07:11.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:07:11.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:07:11.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:07:11.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:07:11.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:07:11.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:07:11.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:07:11.559 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:07:11.559 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:07:12.079 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26210149 2024-09-18T15:07:12.079 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26210149 '!=' null 2024-09-18T15:07:12.079 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:07:12.080 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:12.080 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:07:12.080 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:12.155 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:07:12.155 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:07:12.156 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:12.156 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:12.156 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:12.156 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:07:12.156 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:07:12.156 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:07:12.156 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:07:12.156 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:07:12.156 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:12.156 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:07:12.156 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:07:12.156 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:07:12.156 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:12.157 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:12.157 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:12.157 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:07:12.157 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:12.157 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:07:12.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:07:12.265 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:07:12.265 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:07:12.265 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:07:12.265 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:07:12.265 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:07:12.265 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:07:12.265 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:07:12.265 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:07:12.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:07:12.641 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:07:12.641 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:07:12.641 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:07:13.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:07:13.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:07:13.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:07:13.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:07:13.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:07:13.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:07:13.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:07:13.139 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:07:13.139 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:07:13.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473414 2024-09-18T15:07:13.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473414 '!=' null 2024-09-18T15:07:13.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:07:13.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:13.756 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:07:13.756 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:13.831 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:07:13.831 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:07:13.832 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:13.832 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:13.832 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:13.832 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:07:13.832 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:07:13.832 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:07:13.832 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:07:13.832 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:07:13.832 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:13.832 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:07:13.832 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:07:13.832 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:07:13.832 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:13.832 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:13.833 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:13.833 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:07:13.833 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:13.833 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:07:13.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:07:13.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:07:13.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:07:13.940 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:07:13.940 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:07:13.940 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:07:13.940 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:07:13.941 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:07:13.941 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:07:14.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:07:14.306 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:07:14.306 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:07:14.306 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:07:14.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:07:14.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:07:14.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:07:14.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:07:14.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:07:14.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:07:14.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:07:14.984 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:07:14.984 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:07:15.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26210163 2024-09-18T15:07:15.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26210163 '!=' null 2024-09-18T15:07:15.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:07:15.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:15.560 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:07:15.560 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:15.635 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:07:15.635 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:07:15.635 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:15.635 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:15.635 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:15.635 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:07:15.635 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:07:15.635 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:07:15.635 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:07:15.635 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:07:15.635 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:15.636 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:07:15.636 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:07:15.636 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:07:15.636 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:15.636 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:15.636 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:15.636 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:07:15.636 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:15.636 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:07:15.642 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:07:15.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:07:15.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:07:15.744 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:07:15.744 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:07:15.744 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:07:15.744 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:07:15.744 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:07:15.744 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:07:16.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:07:16.105 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:07:16.105 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:07:16.106 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:07:16.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:07:16.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:07:16.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:07:16.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:07:16.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:07:16.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:07:16.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:07:16.632 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:07:16.633 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:07:17.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473363 2024-09-18T15:07:17.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473363 '!=' null 2024-09-18T15:07:17.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:07:17.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:17.174 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:07:17.174 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:17.249 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:07:17.249 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:07:17.250 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:17.250 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:17.250 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:17.250 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:07:17.250 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:07:17.250 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:07:17.250 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:07:17.250 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:07:17.250 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:17.250 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:07:17.250 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:07:17.250 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:07:17.250 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:17.251 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:17.251 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:17.251 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:07:17.251 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:17.251 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:07:17.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:07:17.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:07:17.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:07:17.358 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:07:17.358 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:07:17.358 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:07:17.358 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:07:17.358 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:07:17.359 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:07:17.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:07:17.717 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:07:17.717 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:07:17.717 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:07:18.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:07:18.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:07:18.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:07:18.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:07:18.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:07:18.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:07:18.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:07:18.294 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:07:18.294 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:07:18.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473327 2024-09-18T15:07:18.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473327 '!=' null 2024-09-18T15:07:18.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:07:18.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:18.798 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:07:18.798 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:18.873 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:07:18.873 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:07:18.873 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:18.873 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:18.873 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:18.873 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:07:18.874 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:07:18.874 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:07:18.874 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:07:18.874 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:07:18.874 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:18.874 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:07:18.874 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:07:18.874 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:07:18.874 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:18.874 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:18.874 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:18.874 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:07:18.874 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:18.874 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:07:18.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:07:18.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:07:18.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:07:18.982 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:07:18.982 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:07:18.982 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:07:18.982 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:07:18.982 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:07:18.982 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:07:19.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:07:19.347 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:07:19.347 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:07:19.347 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:07:19.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:07:19.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:07:19.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:07:19.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:07:19.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:07:19.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:07:19.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:07:19.873 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:07:19.874 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:07:20.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473327 2024-09-18T15:07:20.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473327 '!=' null 2024-09-18T15:07:20.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:07:20.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:20.634 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:07:20.634 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:20.710 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:07:20.710 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:07:20.710 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:20.710 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:20.710 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:20.710 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:07:20.710 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:07:20.710 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:07:20.710 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:07:20.710 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:07:20.710 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:20.710 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:07:20.711 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:07:20.711 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:07:20.711 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:20.711 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:20.711 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:20.711 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:07:20.711 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:20.711 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:07:20.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:07:20.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:07:20.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:07:20.818 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:07:20.818 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:07:20.819 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:07:20.819 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:07:20.819 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:07:20.819 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:07:21.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:07:21.178 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:07:21.179 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:07:21.179 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:07:21.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:07:21.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:07:21.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:07:21.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:07:21.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:07:21.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:07:21.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:07:21.696 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:07:21.696 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:07:22.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209776 2024-09-18T15:07:22.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209776 '!=' null 2024-09-18T15:07:22.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:07:22.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:22.373 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:07:22.373 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:22.448 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:07:22.448 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:07:22.448 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:22.448 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:22.448 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:22.448 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:07:22.448 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:07:22.448 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:07:22.448 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:07:22.448 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:07:22.449 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:22.449 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:07:22.449 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:07:22.449 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:07:22.449 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:22.449 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:22.449 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:22.449 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:07:22.449 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:22.449 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:07:22.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:07:22.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:07:22.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:07:22.557 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:07:22.557 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:07:22.557 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:07:22.557 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:07:22.557 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:07:22.557 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:07:22.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:07:22.982 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:07:22.982 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:07:22.983 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:07:23.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:07:23.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:07:23.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:07:23.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:07:23.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:07:23.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:07:23.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:07:23.482 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:07:23.482 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:07:24.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473132 2024-09-18T15:07:24.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473132 '!=' null 2024-09-18T15:07:24.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:07:24.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:24.140 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:07:24.140 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:24.214 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:07:24.215 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:07:24.215 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:24.215 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:24.215 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:24.215 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:07:24.215 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:07:24.215 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:07:24.215 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:07:24.215 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:07:24.215 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:24.215 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:07:24.215 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:07:24.215 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:07:24.215 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:24.215 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:24.216 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:24.216 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:07:24.216 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:24.216 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:07:24.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:07:24.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:07:24.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:07:24.324 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:07:24.324 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:07:24.324 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:07:24.324 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:07:24.324 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:07:24.324 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:07:24.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:07:24.684 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:07:24.684 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:07:24.684 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:07:25.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:07:25.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:07:25.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:07:25.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:07:25.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:07:25.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:07:25.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:07:25.180 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:07:25.180 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:07:25.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209698 2024-09-18T15:07:25.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209698 '!=' null 2024-09-18T15:07:25.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:07:25.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:25.736 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:07:25.736 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:25.810 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:07:25.810 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:07:25.810 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:25.810 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:25.811 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:25.811 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:07:25.811 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:07:25.811 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:07:25.811 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:07:25.811 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:07:25.811 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:25.811 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:07:25.811 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:07:25.811 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:07:25.811 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:25.811 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:25.811 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:25.811 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:07:25.812 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:25.812 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:07:25.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:07:25.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:07:25.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:07:25.919 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:07:25.919 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:07:25.919 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:07:25.919 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:07:25.919 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:07:25.919 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:07:26.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:07:26.297 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:07:26.297 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:07:26.298 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:07:26.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:07:26.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:07:26.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:07:26.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:07:26.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:07:26.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:07:26.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:07:26.817 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:07:26.817 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:07:27.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473259 2024-09-18T15:07:27.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473259 '!=' null 2024-09-18T15:07:27.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:07:27.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:27.311 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:07:27.311 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:27.386 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:07:27.386 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:07:27.386 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:27.387 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:27.387 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:27.387 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:07:27.387 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:07:27.387 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:07:27.387 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:07:27.387 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:07:27.387 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:27.387 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:07:27.387 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:07:27.387 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:07:27.387 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:27.387 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:27.387 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:27.388 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:07:27.388 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:27.388 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:07:27.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:07:27.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:07:27.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:07:27.495 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:07:27.495 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:07:27.495 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:07:27.495 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:07:27.495 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:07:27.495 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:07:27.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:07:27.856 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:07:27.856 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:07:27.856 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:07:28.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:07:28.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:07:28.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:07:28.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:07:28.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:07:28.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:07:28.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:07:28.352 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:07:28.352 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:07:28.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473356 2024-09-18T15:07:28.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473356 '!=' null 2024-09-18T15:07:28.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:07:28.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:28.915 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:07:28.915 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:28.990 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:07:28.990 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:07:28.990 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:28.990 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:28.990 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:28.990 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:07:28.990 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:07:28.990 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:07:28.990 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:07:28.990 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:07:28.990 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:28.991 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:07:28.991 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:07:28.991 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:07:28.991 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:28.991 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:28.991 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:28.991 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:07:28.991 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:28.991 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:07:28.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:07:29.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:07:29.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:07:29.099 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:07:29.099 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:07:29.099 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:07:29.099 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:07:29.099 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:07:29.099 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:07:29.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:07:29.458 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:07:29.459 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:07:29.459 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:07:30.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:07:30.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:07:30.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:07:30.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:07:30.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:07:30.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:07:30.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:07:30.072 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:07:30.072 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:07:30.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209952 2024-09-18T15:07:30.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209952 '!=' null 2024-09-18T15:07:30.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:07:30.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:30.674 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:07:30.674 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:30.749 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:07:30.749 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:07:30.749 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:30.749 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:30.749 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:30.749 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:07:30.750 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:07:30.750 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:07:30.750 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:07:30.750 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:07:30.750 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:30.750 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:07:30.750 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:07:30.750 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:07:30.752 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:30.752 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:30.752 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:30.752 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:07:30.752 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:30.752 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:07:30.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:07:30.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:07:30.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:07:30.859 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:07:30.859 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:07:30.860 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:07:30.860 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:07:30.860 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:07:30.860 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:07:31.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:07:31.237 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:07:31.237 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:07:31.237 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:07:31.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:07:31.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:07:31.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:07:31.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:07:31.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:07:31.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:07:31.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:07:31.790 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:07:31.790 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:07:32.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209952 2024-09-18T15:07:32.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209952 '!=' null 2024-09-18T15:07:32.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:07:32.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:32.352 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:07:32.352 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:32.427 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:07:32.427 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:07:32.427 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:32.427 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:32.427 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:32.428 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:07:32.428 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:07:32.428 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:07:32.428 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:07:32.428 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:07:32.428 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:32.428 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:07:32.428 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:07:32.428 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:07:32.428 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:32.428 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:32.428 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:32.428 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:07:32.428 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:32.429 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:07:32.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:07:32.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:07:32.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:07:32.536 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:07:32.536 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:07:32.536 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:07:32.536 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:07:32.536 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:07:32.537 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:07:33.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:07:33.029 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:07:33.029 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:07:33.029 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:07:33.648 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:07:33.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:07:33.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:07:33.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:07:33.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:07:33.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:07:33.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:07:33.649 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:07:33.649 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:07:34.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473184 2024-09-18T15:07:34.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473184 '!=' null 2024-09-18T15:07:34.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:07:34.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:34.187 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:07:34.187 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:34.261 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:07:34.261 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:07:34.262 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:34.262 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:34.262 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:34.262 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:07:34.262 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:07:34.262 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:07:34.262 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:07:34.262 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:07:34.262 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:34.262 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:07:34.262 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:07:34.262 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:07:34.262 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:34.262 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:34.262 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:34.263 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:07:34.263 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:34.263 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:07:34.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:07:34.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:07:34.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:07:34.370 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:07:34.370 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:07:34.370 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:07:34.370 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:07:34.370 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:07:34.371 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:07:34.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:07:34.795 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:07:34.795 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:07:34.795 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:07:35.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:07:35.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:07:35.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:07:35.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:07:35.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:07:35.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:07:35.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:07:35.464 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:07:35.464 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:07:36.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209597 2024-09-18T15:07:36.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209597 '!=' null 2024-09-18T15:07:36.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:07:36.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:36.225 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:07:36.226 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:36.300 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:07:36.300 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:07:36.300 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:36.300 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:36.300 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:36.300 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:07:36.300 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:07:36.300 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:07:36.301 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:07:36.301 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:07:36.301 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:36.301 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:07:36.301 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:07:36.301 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:07:36.301 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:36.301 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:36.301 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:36.301 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:07:36.301 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:36.301 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:07:36.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:07:36.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:07:36.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:07:36.409 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:07:36.409 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:07:36.409 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:07:36.409 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:07:36.409 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:07:36.409 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:07:36.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:07:36.771 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:07:36.771 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:07:36.771 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:07:37.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:07:37.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:07:37.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:07:37.406 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:07:37.406 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:07:37.406 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:07:37.406 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:07:37.406 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:07:37.406 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:07:38.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17472846 2024-09-18T15:07:38.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17472846 '!=' null 2024-09-18T15:07:38.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:07:38.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:38.058 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:07:38.058 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:38.132 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:07:38.132 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:07:38.132 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:38.133 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:38.133 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:38.133 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:07:38.133 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:07:38.133 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:07:38.133 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:07:38.133 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:07:38.133 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:38.133 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:07:38.133 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:07:38.133 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:07:38.133 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:38.133 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:38.133 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:38.134 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:07:38.134 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:38.134 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:07:38.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:07:38.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:07:38.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:07:38.241 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:07:38.241 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:07:38.241 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:07:38.241 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:07:38.241 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:07:38.241 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:07:38.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:07:38.610 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:07:38.610 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:07:38.610 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:07:39.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:07:39.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:07:39.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:07:39.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:07:39.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:07:39.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:07:39.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:07:39.213 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:07:39.213 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:07:39.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17472770 2024-09-18T15:07:39.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17472770 '!=' null 2024-09-18T15:07:39.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:07:39.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:39.905 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:07:39.905 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:39.980 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:07:39.981 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:07:39.981 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:39.981 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:39.981 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:39.981 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:07:39.981 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:07:39.981 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:07:39.981 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:07:39.981 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:07:39.981 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:39.981 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:07:39.981 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:07:39.981 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:07:39.981 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:39.982 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:39.982 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:39.982 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:07:39.982 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:39.982 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:07:39.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:07:40.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:07:40.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:07:40.090 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:07:40.090 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:07:40.090 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:07:40.090 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:07:40.090 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:07:40.090 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:07:40.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:07:40.455 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:07:40.456 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:07:40.456 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:07:40.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:07:40.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:07:40.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:07:40.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:07:40.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:07:40.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:07:40.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:07:40.977 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:07:40.977 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:07:41.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25335378 2024-09-18T15:07:41.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25335378 '!=' null 2024-09-18T15:07:41.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:07:41.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:41.486 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:07:41.486 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:41.562 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:07:41.562 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:07:41.562 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:41.562 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:41.562 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:41.562 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:07:41.562 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:07:41.562 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:07:41.562 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:07:41.562 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:07:41.562 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:41.563 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:07:41.563 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:07:41.563 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:07:41.563 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:41.563 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:41.563 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:41.563 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:07:41.563 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:41.563 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:07:41.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:07:41.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:07:41.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:07:41.671 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:07:41.671 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:07:41.671 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:07:41.671 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:07:41.671 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:07:41.671 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:07:42.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:07:42.040 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:07:42.040 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:07:42.040 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:07:42.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:07:42.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:07:42.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:07:42.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:07:42.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:07:42.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:07:42.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:07:42.608 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:07:42.608 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:07:43.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599120 2024-09-18T15:07:43.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599120 '!=' null 2024-09-18T15:07:43.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:07:43.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:43.219 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:07:43.219 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:43.293 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:07:43.294 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:07:43.294 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:43.294 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:43.294 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:43.294 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:07:43.294 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:07:43.294 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:07:43.294 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:07:43.294 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:07:43.294 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:43.294 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:07:43.294 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:07:43.294 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:07:43.294 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:43.295 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:43.295 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:43.295 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:07:43.295 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:43.295 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:07:43.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:07:43.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:07:43.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:07:43.402 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:07:43.403 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:07:43.403 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:07:43.403 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:07:43.403 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:07:43.403 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:07:43.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:07:43.770 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:07:43.770 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:07:43.771 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:07:44.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:07:44.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:07:44.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:07:44.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:07:44.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:07:44.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:07:44.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:07:44.393 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:07:44.393 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:07:45.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25335879 2024-09-18T15:07:45.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25335879 '!=' null 2024-09-18T15:07:45.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:07:45.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:45.195 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:07:45.195 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:45.270 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:07:45.270 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:07:45.271 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:45.271 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:45.271 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:45.271 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:07:45.271 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:07:45.272 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:07:45.272 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:07:45.272 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:07:45.272 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:45.272 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:07:45.273 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:07:45.273 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:07:45.273 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:45.273 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:45.273 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:45.273 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:07:45.274 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:45.274 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:07:45.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:07:45.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:07:45.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:07:45.379 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:07:45.379 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:07:45.379 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:07:45.379 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:07:45.379 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:07:45.380 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:07:45.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:07:45.739 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:07:45.739 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:07:45.739 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:07:46.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:07:46.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:07:46.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:07:46.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:07:46.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:07:46.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:07:46.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:07:46.290 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:07:46.290 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:07:46.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599410 2024-09-18T15:07:46.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599410 '!=' null 2024-09-18T15:07:46.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:07:46.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:46.800 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:07:46.800 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:46.874 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:07:46.874 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:07:46.874 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:46.875 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:46.875 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:46.875 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:07:46.875 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:07:46.875 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:07:46.875 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:07:46.875 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:07:46.875 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:46.875 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:07:46.875 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:07:46.875 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:07:46.875 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:46.875 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:46.876 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:46.876 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:07:46.876 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:46.876 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:07:46.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:07:46.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:07:46.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:07:46.984 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:07:46.984 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:07:46.984 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:07:46.984 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:07:46.984 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:07:46.984 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:07:47.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:07:47.430 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:07:47.430 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:07:47.430 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:07:48.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:07:48.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:07:48.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:07:48.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:07:48.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:07:48.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:07:48.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:07:48.024 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:07:48.025 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:07:48.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599655 2024-09-18T15:07:48.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599655 '!=' null 2024-09-18T15:07:48.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:07:48.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:48.739 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:07:48.739 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:48.814 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:07:48.814 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:07:48.814 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:48.815 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:48.815 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:48.815 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:07:48.815 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:07:48.815 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:07:48.815 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:07:48.815 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:07:48.815 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:48.815 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:07:48.815 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:07:48.816 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:07:48.816 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:48.816 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:48.816 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:48.816 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:07:48.816 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:48.817 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:07:48.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:07:48.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:07:48.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:07:48.924 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:07:48.924 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:07:48.924 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:07:48.924 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:07:48.924 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:07:48.924 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:07:49.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:07:49.289 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:07:49.290 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:07:49.290 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:07:49.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:07:49.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:07:49.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:07:49.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:07:49.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:07:49.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:07:49.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:07:49.836 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:07:49.836 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:07:50.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599655 2024-09-18T15:07:50.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599655 '!=' null 2024-09-18T15:07:50.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:07:50.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:50.356 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:07:50.357 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:50.430 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:07:50.431 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:07:50.431 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:50.431 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:50.431 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:50.431 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:07:50.431 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:07:50.431 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:07:50.432 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:07:50.432 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:07:50.432 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:50.432 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:07:50.432 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:07:50.432 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:07:50.432 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:50.432 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:50.432 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:50.432 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:07:50.432 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:50.432 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:07:50.437 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:07:50.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:07:50.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:07:50.539 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:07:50.539 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:07:50.540 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:07:50.540 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:07:50.540 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:07:50.540 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:07:50.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:07:50.925 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:07:50.925 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:07:50.926 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:07:51.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:07:51.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:07:51.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:07:51.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:07:51.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:07:51.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:07:51.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:07:51.428 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:07:51.428 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:07:52.079 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25336446 2024-09-18T15:07:52.079 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25336446 '!=' null 2024-09-18T15:07:52.079 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:07:52.079 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:52.079 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:07:52.079 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:52.154 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:07:52.155 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:07:52.155 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:52.155 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:52.155 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:52.155 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:07:52.155 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:07:52.155 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:07:52.155 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:07:52.155 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:07:52.156 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:52.156 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:07:52.156 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:07:52.156 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:07:52.156 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:52.156 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:52.156 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:52.156 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:07:52.156 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:52.156 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:07:52.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:07:52.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:07:52.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:07:52.263 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:07:52.263 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:07:52.264 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:07:52.264 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:07:52.264 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:07:52.264 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:07:52.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:07:52.627 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:07:52.627 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:07:52.627 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:07:53.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:07:53.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:07:53.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:07:53.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:07:53.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:07:53.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:07:53.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:07:53.163 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:07:53.163 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:07:53.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473486 2024-09-18T15:07:53.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473486 '!=' null 2024-09-18T15:07:53.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:07:53.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:53.810 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:07:53.810 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:53.885 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:07:53.885 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:07:53.885 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:53.885 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:53.885 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:53.885 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:07:53.885 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:07:53.886 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:07:53.886 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:07:53.886 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:07:53.886 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:53.886 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:07:53.886 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:07:53.886 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:07:53.886 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:53.886 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:53.886 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:53.887 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:07:53.887 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:53.887 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:07:53.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:07:53.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:07:53.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:07:53.994 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:07:53.994 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:07:53.994 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:07:53.994 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:07:53.994 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:07:53.994 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:07:54.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:07:54.354 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:07:54.355 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:07:54.355 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:07:54.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:07:54.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:07:54.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:07:54.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:07:54.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:07:54.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:07:54.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:07:54.859 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:07:54.859 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:07:55.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26210274 2024-09-18T15:07:55.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26210274 '!=' null 2024-09-18T15:07:55.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:07:55.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:55.426 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:07:55.426 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:55.501 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:07:55.501 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:07:55.501 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:55.501 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:55.501 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:55.501 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:07:55.501 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:07:55.502 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:07:55.502 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:07:55.502 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:07:55.502 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:55.502 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:07:55.502 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:07:55.502 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:07:55.502 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:55.502 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:55.502 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:55.502 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:07:55.503 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:55.503 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:07:55.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:07:55.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:07:55.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:07:55.610 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:07:55.610 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:07:55.610 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:07:55.610 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:07:55.610 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:07:55.610 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:07:55.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:07:55.971 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:07:55.971 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:07:55.971 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:07:56.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:07:56.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:07:56.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:07:56.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:07:56.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:07:56.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:07:56.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:07:56.468 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:07:56.468 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:07:57.052 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473439 2024-09-18T15:07:57.052 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473439 '!=' null 2024-09-18T15:07:57.052 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:07:57.052 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:57.053 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:07:57.053 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:57.127 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:07:57.127 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:07:57.128 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:57.128 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:57.128 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:57.128 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:07:57.128 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:07:57.128 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:07:57.128 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:07:57.128 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:07:57.128 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:57.128 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:07:57.128 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:07:57.128 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:07:57.129 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:57.129 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:57.129 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:57.129 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:07:57.129 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:57.129 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:07:57.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:07:57.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:07:57.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:07:57.236 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:07:57.236 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:07:57.236 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:07:57.237 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:07:57.237 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:07:57.237 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:07:57.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:07:57.598 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:07:57.598 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:07:57.599 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:07:58.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:07:58.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:07:58.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:07:58.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:07:58.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:07:58.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:07:58.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:07:58.101 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:07:58.101 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:07:58.660 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473571 2024-09-18T15:07:58.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473571 '!=' null 2024-09-18T15:07:58.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:07:58.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:58.750 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:07:58.751 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:07:58.751 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:07:58.751 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:07:58.751 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:58.751 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:58.752 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:58.752 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:07:58.752 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:07:58.752 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:07:58.752 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:07:58.752 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:07:58.752 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:58.752 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:07:58.752 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:07:58.752 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:07:58.752 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:07:58.752 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:07:58.752 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:07:58.753 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:07:58.753 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:07:58.753 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:07:58.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:07:58.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:07:58.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:07:58.844 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:07:58.844 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:07:58.844 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:07:58.844 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:07:58.844 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:07:58.844 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:07:59.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:07:59.314 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:07:59.314 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:07:59.315 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:07:59.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:07:59.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:07:59.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:07:59.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:07:59.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:07:59.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:07:59.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:07:59.923 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:07:59.923 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:08:00.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473571 2024-09-18T15:08:00.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473571 '!=' null 2024-09-18T15:08:00.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:08:00.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:08:00.517 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:08:00.517 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:08:00.592 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:08:00.592 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:08:00.592 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:08:00.592 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:08:00.592 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:08:00.592 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:08:00.592 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:00.592 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:08:00.592 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:08:00.592 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:08:00.593 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:00.593 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:00.593 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:00.593 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:08:00.593 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:08:00.593 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:08:00.593 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:08:00.593 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:08:00.593 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:00.593 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:08:00.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:08:00.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:08:00.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:08:00.701 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:08:00.701 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:08:00.701 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:08:00.701 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:08:00.701 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:08:00.701 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:08:01.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:08:01.114 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:08:01.115 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:08:01.115 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:08:01.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:08:01.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:08:01.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:08:01.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:08:01.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:08:01.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:08:01.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:08:01.730 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:08:01.730 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:08:02.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26210407 2024-09-18T15:08:02.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26210407 '!=' null 2024-09-18T15:08:02.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:08:02.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:08:02.274 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:08:02.274 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:08:02.348 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:08:02.349 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:08:02.349 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:08:02.349 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:08:02.349 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:08:02.349 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:08:02.349 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:02.349 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:08:02.349 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:08:02.349 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:08:02.349 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:02.349 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:02.350 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:02.350 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:08:02.350 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:08:02.350 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:08:02.350 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:08:02.350 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:08:02.350 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:02.350 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:08:02.355 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:08:02.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:08:02.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:08:02.457 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:08:02.458 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:08:02.458 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:08:02.458 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:08:02.458 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:08:02.458 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:08:02.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:08:02.821 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:08:02.821 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:08:02.821 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:08:03.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:08:03.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:08:03.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:08:03.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:08:03.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:08:03.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:08:03.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:08:03.452 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:08:03.452 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:08:04.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473536 2024-09-18T15:08:04.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473536 '!=' null 2024-09-18T15:08:04.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:08:04.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:08:04.013 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:08:04.013 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:08:04.087 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:08:04.087 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:08:04.088 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:08:04.088 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:08:04.088 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:08:04.088 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:08:04.088 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:04.088 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:08:04.088 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:08:04.088 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:08:04.088 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:04.088 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:04.088 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:04.088 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:08:04.088 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:08:04.088 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:08:04.089 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:08:04.089 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:08:04.089 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:04.089 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:08:04.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:08:04.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:08:04.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:08:04.196 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:08:04.196 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:08:04.196 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:08:04.196 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:08:04.196 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:08:04.197 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:08:04.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:08:04.560 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:08:04.560 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:08:04.560 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:08:05.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:08:05.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:08:05.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:08:05.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:08:05.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:08:05.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:08:05.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:08:05.061 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:08:05.061 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:08:05.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26210337 2024-09-18T15:08:05.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26210337 '!=' null 2024-09-18T15:08:05.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:08:05.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:08:05.619 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:08:05.619 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:08:05.694 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:08:05.694 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:08:05.695 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:08:05.695 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:08:05.695 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:08:05.695 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:08:05.695 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:05.695 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:08:05.695 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:08:05.695 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:08:05.695 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:05.695 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:05.695 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:05.695 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:08:05.695 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:08:05.695 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:08:05.695 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:08:05.696 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:08:05.696 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:05.696 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:08:05.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:08:05.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:08:05.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:08:05.804 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:08:05.804 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:08:05.804 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:08:05.804 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:08:05.804 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:08:05.804 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:08:06.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:08:06.193 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:08:06.193 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:08:06.194 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:08:06.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:08:06.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:08:06.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:08:06.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:08:06.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:08:06.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:08:06.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:08:06.825 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:08:06.825 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:08:07.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473522 2024-09-18T15:08:07.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473522 '!=' null 2024-09-18T15:08:07.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:08:07.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:08:07.362 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:08:07.362 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:08:07.437 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:08:07.437 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:08:07.437 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:08:07.437 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:08:07.437 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:08:07.437 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:08:07.438 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:07.438 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:08:07.438 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:08:07.438 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:08:07.438 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:07.438 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:07.438 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:07.438 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:08:07.438 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:08:07.438 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:08:07.438 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:08:07.438 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:08:07.438 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:07.439 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:08:07.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:08:07.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:08:07.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:08:07.546 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:08:07.547 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:08:07.547 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:08:07.547 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:08:07.547 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:08:07.547 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:08:07.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:08:07.918 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:08:07.918 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:08:07.918 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:08:08.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:08:08.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:08:08.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:08:08.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:08:08.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:08:08.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:08:08.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:08:08.410 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:08:08.410 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:08:09.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473591 2024-09-18T15:08:09.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473591 '!=' null 2024-09-18T15:08:09.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:08:09.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:08:09.106 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:08:09.106 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:08:09.180 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:08:09.180 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:08:09.180 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:08:09.180 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:08:09.181 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:08:09.181 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:08:09.181 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:09.181 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:08:09.181 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:08:09.181 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:08:09.181 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:09.181 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:09.181 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:09.181 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:08:09.181 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:08:09.181 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:08:09.181 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:08:09.182 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:08:09.182 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:09.182 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:08:09.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:08:09.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:08:09.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:08:09.289 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:08:09.289 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:08:09.289 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:08:09.289 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:08:09.290 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:08:09.290 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:08:09.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:08:09.697 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:08:09.697 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:08:09.697 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:08:10.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:08:10.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:08:10.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:08:10.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:08:10.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:08:10.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:08:10.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:08:10.350 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:08:10.350 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:08:10.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26210037 2024-09-18T15:08:10.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26210037 '!=' null 2024-09-18T15:08:10.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:08:10.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:08:10.898 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:08:10.898 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:08:10.972 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:08:10.972 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:08:10.972 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:08:10.972 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:08:10.973 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:08:10.973 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:08:10.973 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:10.973 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:08:10.973 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:08:10.973 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:08:10.973 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:10.973 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:10.973 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:10.973 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:08:10.973 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:08:10.973 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:08:10.973 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:08:10.973 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:08:10.974 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:10.974 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:08:10.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:08:11.080 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:08:11.080 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:08:11.081 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:08:11.081 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:08:11.081 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:08:11.081 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:08:11.081 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:08:11.081 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:08:11.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:08:11.446 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:08:11.447 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:08:11.447 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:08:12.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:08:12.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:08:12.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:08:12.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:08:12.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:08:12.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:08:12.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:08:12.024 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:08:12.024 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:08:12.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473089 2024-09-18T15:08:12.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473089 '!=' null 2024-09-18T15:08:12.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:08:12.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:08:12.533 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:08:12.533 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:08:12.607 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:08:12.608 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:08:12.608 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:08:12.608 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:08:12.608 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:08:12.608 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:08:12.608 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:12.608 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:08:12.608 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:08:12.609 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:08:12.609 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:12.609 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:12.609 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:12.609 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:08:12.609 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:08:12.609 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:08:12.609 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:08:12.609 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:08:12.609 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:12.609 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:08:12.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:08:12.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:08:12.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:08:12.717 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:08:12.717 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:08:12.717 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:08:12.717 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:08:12.717 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:08:12.717 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:08:13.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:08:13.162 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:08:13.162 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:08:13.163 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:08:13.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:08:13.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:08:13.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:08:13.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:08:13.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:08:13.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:08:13.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:08:13.662 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:08:13.662 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:08:14.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473089 2024-09-18T15:08:14.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473089 '!=' null 2024-09-18T15:08:14.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:08:14.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:08:14.348 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:08:14.348 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:08:14.423 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:08:14.423 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:08:14.423 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:08:14.423 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:08:14.424 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:08:14.424 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:08:14.424 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:14.424 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:08:14.424 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:08:14.424 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:08:14.424 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:14.424 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:14.424 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:14.424 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:08:14.424 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:08:14.424 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:08:14.424 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:08:14.424 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:08:14.424 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:14.425 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:08:14.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:08:14.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:08:14.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:08:14.532 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:08:14.532 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:08:14.532 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:08:14.532 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:08:14.532 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:08:14.532 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:08:14.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:08:14.897 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:08:14.898 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:08:14.898 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:08:15.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:08:15.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:08:15.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:08:15.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:08:15.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:08:15.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:08:15.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:08:15.530 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:08:15.530 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:08:16.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209617 2024-09-18T15:08:16.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209617 '!=' null 2024-09-18T15:08:16.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:08:16.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:08:16.067 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:08:16.068 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:08:16.143 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:08:16.143 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:08:16.143 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:08:16.143 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:08:16.143 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:08:16.143 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:08:16.143 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:16.143 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:08:16.144 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:08:16.144 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:08:16.144 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:16.144 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:16.144 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:16.144 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:08:16.144 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:08:16.144 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:08:16.144 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:08:16.144 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:08:16.144 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:16.144 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:08:16.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:08:16.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:08:16.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:08:16.252 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:08:16.252 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:08:16.252 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:08:16.252 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:08:16.252 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:08:16.253 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:08:16.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:08:16.616 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:08:16.616 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:08:16.616 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:08:17.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:08:17.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:08:17.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:08:17.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:08:17.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:08:17.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:08:17.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:08:17.130 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:08:17.130 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:08:17.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473028 2024-09-18T15:08:17.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473028 '!=' null 2024-09-18T15:08:17.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:08:17.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:08:17.763 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:08:17.763 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:08:17.839 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:08:17.839 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:08:17.839 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:08:17.839 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:08:17.839 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:08:17.839 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [ 2024-09-18T15:08:17.839 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:17.839 INFO:tasks.workunit.client.0.smithi047.stdout: "item": "3.0", 2024-09-18T15:08:17.839 INFO:tasks.workunit.client.0.smithi047.stdout: "prio": 190, 2024-09-18T15:08:17.839 INFO:tasks.workunit.client.0.smithi047.stdout: "can_preempt": true 2024-09-18T15:08:17.839 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:17.840 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:17.840 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:17.840 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:08:17.840 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:08:17.840 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:08:17.840 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:08:17.840 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:08:17.840 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:17.840 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:08:17.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:08:17.947 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:08:17.947 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:08:17.948 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:08:17.948 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:08:17.948 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:08:17.948 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:08:17.949 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:08:17.949 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:08:18.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:08:18.404 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:08:18.405 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:08:18.405 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:08:19.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:08:19.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:08:19.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:08:19.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:08:19.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:08:19.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:08:19.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:08:19.006 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:08:19.006 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:08:19.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473049 2024-09-18T15:08:19.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473049 '!=' null 2024-09-18T15:08:19.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:08:19.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:08:19.576 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:08:19.576 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:08:19.651 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:08:19.651 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:08:19.651 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:08:19.651 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:08:19.651 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:08:19.651 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:08:19.651 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:19.651 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:08:19.651 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:08:19.651 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:08:19.652 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:08:19.652 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:08:19.652 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:19.652 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:08:19.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:08:19.760 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:08:19.760 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:08:19.760 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:08:19.760 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:08:19.760 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:08:19.760 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:08:19.761 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:08:19.761 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:08:20.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:08:20.125 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:08:20.125 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:08:20.125 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:08:20.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:08:20.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:08:20.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:08:20.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:08:20.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:08:20.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:08:20.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:08:20.631 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:08:20.631 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:08:21.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24899723 2024-09-18T15:08:21.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24899723 '!=' null 2024-09-18T15:08:21.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:08:21.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:08:21.145 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:08:21.145 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:08:21.220 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:08:21.220 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:08:21.220 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:08:21.220 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:08:21.220 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:08:21.220 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:08:21.220 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:21.221 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:08:21.221 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:08:21.221 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:08:21.221 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:08:21.221 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:08:21.221 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:21.221 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:08:21.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:08:21.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:08:21.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:08:21.329 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:08:21.329 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:08:21.329 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:08:21.329 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:08:21.329 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:08:21.329 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:08:21.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:08:21.689 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:08:21.689 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:08:21.690 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:08:22.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:08:22.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:08:22.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:08:22.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:08:22.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:08:22.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:08:22.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:08:22.187 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:08:22.187 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:08:22.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11794698 2024-09-18T15:08:22.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11794698 '!=' null 2024-09-18T15:08:22.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:08:22.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:08:22.719 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:08:22.719 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:08:22.793 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:08:22.793 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:08:22.793 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:08:22.793 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:08:22.793 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:08:22.793 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:08:22.793 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:22.794 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:08:22.794 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:08:22.794 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:08:22.794 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:08:22.794 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:08:22.794 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:22.794 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:08:22.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:08:22.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:08:22.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:08:22.902 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:08:22.902 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:08:22.902 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:08:22.902 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:08:22.902 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:08:22.903 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:08:23.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:08:23.259 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:08:23.260 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:08:23.260 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:08:23.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-18T15:08:23.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-18T15:08:23.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:08:23.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:08:23.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:08:23.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:08:23.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:08:23.754 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:08:23.754 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:08:24.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11794698 2024-09-18T15:08:24.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11794698 '!=' null 2024-09-18T15:08:24.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:08:24.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:08:24.293 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-18T15:08:24.293 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_recovery_reservations 2024-09-18T15:08:24.368 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:08:24.369 INFO:tasks.workunit.client.0.smithi047.stdout: "local_reservations": { 2024-09-18T15:08:24.369 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:08:24.369 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:08:24.369 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:08:24.369 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:08:24.369 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:24.369 INFO:tasks.workunit.client.0.smithi047.stdout: "remote_reservations": { 2024-09-18T15:08:24.369 INFO:tasks.workunit.client.0.smithi047.stdout: "max_allowed": 1, 2024-09-18T15:08:24.369 INFO:tasks.workunit.client.0.smithi047.stdout: "min_priority": 0, 2024-09-18T15:08:24.369 INFO:tasks.workunit.client.0.smithi047.stdout: "queues": [], 2024-09-18T15:08:24.369 INFO:tasks.workunit.client.0.smithi047.stdout: "in_progress": [] 2024-09-18T15:08:24.369 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:24.369 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:08:24.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:08:24.477 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:08:24.477 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:08:24.477 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:08:24.477 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:08:24.477 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:08:24.477 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:08:24.478 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:08:24.478 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:08:24.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-09-18T15:08:24.839 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:08:24.840 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:08:24.840 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:08:25.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 3 2024-09-18T15:08:25.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:08:25.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:08:25.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:298: TEST_recovery_priority: ceph pg dump pgs 2024-09-18T15:08:25.683 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T15:08:25.684 INFO:tasks.workunit.client.0.smithi047.stdout:7.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-09-18T15:04:57.814947+0000 94'600 119:1097 [1,4] 1 [1,4] 1 0'0 2024-09-18T14:57:07.342435+0000 0'0 2024-09-18T14:57:07.342435+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:55:00.267491+0000 0 0 2024-09-18T15:08:25.684 INFO:tasks.workunit.client.0.smithi047.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-09-18T15:08:19.164283+0000 94'600 120:1111 [1,2] 1 [1,2] 1 0'0 2024-09-18T14:56:56.247413+0000 0'0 2024-09-18T14:56:56.247413+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:01:13.267916+0000 0 0 2024-09-18T15:08:25.684 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-09-18T15:06:39.055628+0000 94'600 119:1105 [1,2] 1 [1,2] 1 0'0 2024-09-18T14:56:50.949946+0000 0'0 2024-09-18T14:56:50.949946+0000 0 0 periodic scrub scheduled @ 2024-09-19T19:56:33.644892+0000 0 0 2024-09-18T15:08:25.684 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T15:08:25.684 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T15:08:25.684 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T15:08:25.696 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:300: TEST_recovery_priority: get_asok_path osd.1 2024-09-18T15:08:25.696 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:08:25.696 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:08:25.696 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:08:25.696 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:08:25.696 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215238 2024-09-18T15:08:25.696 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.215238/ceph-osd.1.asok 2024-09-18T15:08:25.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:300: TEST_recovery_priority: CEPH_ARGS= 2024-09-18T15:08:25.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:300: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.215238/ceph-osd.1.asok dump_pgstate_history 2024-09-18T15:08:25.773 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:08:25.773 INFO:tasks.workunit.client.0.smithi047.stdout: "pgs": [ 2024-09-18T15:08:25.773 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.773 INFO:tasks.workunit.client.0.smithi047.stdout: "pg": "7.0", 2024-09-18T15:08:25.773 INFO:tasks.workunit.client.0.smithi047.stdout: "currently": "Started/Primary/Active/Clean", 2024-09-18T15:08:25.773 INFO:tasks.workunit.client.0.smithi047.stdout: "history": [ 2024-09-18T15:08:25.773 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.773 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": "62", 2024-09-18T15:08:25.774 INFO:tasks.workunit.client.0.smithi047.stdout: "states": [ 2024-09-18T15:08:25.774 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.774 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Start", 2024-09-18T15:08:25.774 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:07.887169+0000", 2024-09-18T15:08:25.774 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:07.887177+0000" 2024-09-18T15:08:25.774 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.774 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.774 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-18T15:08:25.774 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:07.887182+0000", 2024-09-18T15:08:25.774 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:08.013228+0000" 2024-09-18T15:08:25.774 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.774 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.775 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-18T15:08:25.775 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:08.013229+0000", 2024-09-18T15:08:25.775 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:08.013275+0000" 2024-09-18T15:08:25.775 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.775 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.775 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-18T15:08:25.775 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:08.013276+0000", 2024-09-18T15:08:25.775 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:08.013284+0000" 2024-09-18T15:08:25.775 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.775 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.775 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-18T15:08:25.775 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:08.013284+0000", 2024-09-18T15:08:25.776 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:08.197490+0000" 2024-09-18T15:08:25.776 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.776 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.776 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering", 2024-09-18T15:08:25.776 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:07.887179+0000", 2024-09-18T15:08:25.776 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:08.197498+0000" 2024-09-18T15:08:25.776 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.776 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.776 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/Activating", 2024-09-18T15:08:25.776 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:08.197648+0000", 2024-09-18T15:08:25.776 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:09.060014+0000" 2024-09-18T15:08:25.776 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.777 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.777 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-18T15:08:25.777 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:09.060015+0000", 2024-09-18T15:08:25.777 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:09.060081+0000" 2024-09-18T15:08:25.777 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.777 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.777 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/Clean", 2024-09-18T15:08:25.777 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:09.060081+0000", 2024-09-18T15:08:25.777 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:09.470522+0000" 2024-09-18T15:08:25.777 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.778 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.778 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active", 2024-09-18T15:08:25.778 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:08.197499+0000", 2024-09-18T15:08:25.778 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:09.470551+0000" 2024-09-18T15:08:25.778 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.778 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.778 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary", 2024-09-18T15:08:25.778 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:07.887177+0000", 2024-09-18T15:08:25.778 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:09.470586+0000" 2024-09-18T15:08:25.778 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.778 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.778 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started", 2024-09-18T15:08:25.778 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:07.887167+0000", 2024-09-18T15:08:25.778 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:09.470591+0000" 2024-09-18T15:08:25.779 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:25.779 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:25.779 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.779 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.779 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": "63", 2024-09-18T15:08:25.779 INFO:tasks.workunit.client.0.smithi047.stdout: "states": [ 2024-09-18T15:08:25.779 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.779 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Reset", 2024-09-18T15:08:25.779 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:09.470591+0000", 2024-09-18T15:08:25.779 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:09.471197+0000" 2024-09-18T15:08:25.779 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:25.779 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:25.780 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.780 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.780 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": "91", 2024-09-18T15:08:25.780 INFO:tasks.workunit.client.0.smithi047.stdout: "states": [ 2024-09-18T15:08:25.780 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.780 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Start", 2024-09-18T15:08:25.780 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:09.471206+0000", 2024-09-18T15:08:25.780 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:09.471227+0000" 2024-09-18T15:08:25.780 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.780 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.780 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-18T15:08:25.780 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:09.471239+0000", 2024-09-18T15:08:25.780 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:09.589208+0000" 2024-09-18T15:08:25.780 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.781 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.781 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-18T15:08:25.781 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:09.589209+0000", 2024-09-18T15:08:25.781 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:09.589312+0000" 2024-09-18T15:08:25.781 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.781 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.786 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-18T15:08:25.786 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:09.589315+0000", 2024-09-18T15:08:25.786 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:09.589337+0000" 2024-09-18T15:08:25.786 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.786 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.786 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-18T15:08:25.786 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:09.589338+0000", 2024-09-18T15:08:25.787 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:10.093999+0000" 2024-09-18T15:08:25.787 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.787 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.787 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering", 2024-09-18T15:08:25.787 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:09.471233+0000", 2024-09-18T15:08:25.787 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:10.094009+0000" 2024-09-18T15:08:25.787 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.787 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.787 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/Activating", 2024-09-18T15:08:25.787 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:10.094144+0000", 2024-09-18T15:08:25.787 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:10.803720+0000" 2024-09-18T15:08:25.787 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.787 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.787 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-18T15:08:25.788 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:10.803721+0000", 2024-09-18T15:08:25.788 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:10.803780+0000" 2024-09-18T15:08:25.788 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.788 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.788 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/Clean", 2024-09-18T15:08:25.788 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:10.803781+0000", 2024-09-18T15:08:25.788 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:38.116270+0000" 2024-09-18T15:08:25.788 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.788 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.788 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active", 2024-09-18T15:08:25.788 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:10.094010+0000", 2024-09-18T15:08:25.788 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:38.116288+0000" 2024-09-18T15:08:25.788 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.788 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.789 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary", 2024-09-18T15:08:25.789 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:09.471228+0000", 2024-09-18T15:08:25.789 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:38.116308+0000" 2024-09-18T15:08:25.789 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.789 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.789 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started", 2024-09-18T15:08:25.789 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:09.471198+0000", 2024-09-18T15:08:25.789 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:38.116311+0000" 2024-09-18T15:08:25.789 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:25.789 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:25.789 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.789 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.789 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": "92", 2024-09-18T15:08:25.789 INFO:tasks.workunit.client.0.smithi047.stdout: "states": [ 2024-09-18T15:08:25.790 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.790 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Reset", 2024-09-18T15:08:25.790 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:38.116312+0000", 2024-09-18T15:08:25.790 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:38.116600+0000" 2024-09-18T15:08:25.790 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:25.790 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:25.790 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.790 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.790 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": "99", 2024-09-18T15:08:25.790 INFO:tasks.workunit.client.0.smithi047.stdout: "states": [ 2024-09-18T15:08:25.790 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.790 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Start", 2024-09-18T15:08:25.790 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:38.116604+0000", 2024-09-18T15:08:25.791 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:38.116613+0000" 2024-09-18T15:08:25.791 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.791 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.791 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-18T15:08:25.791 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:38.116621+0000", 2024-09-18T15:08:25.791 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:38.226090+0000" 2024-09-18T15:08:25.791 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.791 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.791 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-18T15:08:25.791 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:38.226091+0000", 2024-09-18T15:08:25.791 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:38.226142+0000" 2024-09-18T15:08:25.792 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.792 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.792 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-18T15:08:25.792 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:38.226143+0000", 2024-09-18T15:08:25.792 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:38.226155+0000" 2024-09-18T15:08:25.792 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.792 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.792 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-18T15:08:25.792 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:38.226155+0000", 2024-09-18T15:08:25.792 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:38.857217+0000" 2024-09-18T15:08:25.792 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.792 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.792 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering", 2024-09-18T15:08:25.793 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:38.116618+0000", 2024-09-18T15:08:25.793 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:38.857225+0000" 2024-09-18T15:08:25.793 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.793 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.793 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/Activating", 2024-09-18T15:08:25.793 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:38.857280+0000", 2024-09-18T15:08:25.793 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:38.955293+0000" 2024-09-18T15:08:25.793 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.793 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.793 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-18T15:08:25.793 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:38.955294+0000", 2024-09-18T15:08:25.793 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:38.955321+0000" 2024-09-18T15:08:25.793 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.793 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.794 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/Clean", 2024-09-18T15:08:25.794 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:38.955321+0000", 2024-09-18T15:08:25.794 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:30.656330+0000" 2024-09-18T15:08:25.794 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.794 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.794 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active", 2024-09-18T15:08:25.794 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:38.857225+0000", 2024-09-18T15:08:25.794 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:30.656348+0000" 2024-09-18T15:08:25.794 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.794 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.794 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary", 2024-09-18T15:08:25.794 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:38.116613+0000", 2024-09-18T15:08:25.794 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:30.656520+0000" 2024-09-18T15:08:25.794 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.795 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.795 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started", 2024-09-18T15:08:25.795 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:38.116600+0000", 2024-09-18T15:08:25.795 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:30.656524+0000" 2024-09-18T15:08:25.795 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:25.795 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:25.795 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.795 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.795 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": "100", 2024-09-18T15:08:25.795 INFO:tasks.workunit.client.0.smithi047.stdout: "states": [ 2024-09-18T15:08:25.795 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.795 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Reset", 2024-09-18T15:08:25.795 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:30.656524+0000", 2024-09-18T15:08:25.795 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:30.657042+0000" 2024-09-18T15:08:25.795 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:25.796 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:25.796 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.796 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.796 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": "102", 2024-09-18T15:08:25.796 INFO:tasks.workunit.client.0.smithi047.stdout: "states": [ 2024-09-18T15:08:25.796 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.796 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Start", 2024-09-18T15:08:25.796 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:30.657046+0000", 2024-09-18T15:08:25.796 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:30.657059+0000" 2024-09-18T15:08:25.796 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.796 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.796 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-18T15:08:25.796 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:30.657066+0000", 2024-09-18T15:08:25.797 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:30.776204+0000" 2024-09-18T15:08:25.797 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.797 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.797 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-18T15:08:25.797 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:30.776205+0000", 2024-09-18T15:08:25.797 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:30.776267+0000" 2024-09-18T15:08:25.797 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.797 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.797 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering", 2024-09-18T15:08:25.797 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:30.657063+0000", 2024-09-18T15:08:25.797 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:30.776274+0000" 2024-09-18T15:08:25.797 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.797 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.797 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/WaitActingChange", 2024-09-18T15:08:25.798 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:30.776275+0000", 2024-09-18T15:08:25.798 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:31.287386+0000" 2024-09-18T15:08:25.798 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.798 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.798 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary", 2024-09-18T15:08:25.798 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:30.657059+0000", 2024-09-18T15:08:25.798 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:31.287414+0000" 2024-09-18T15:08:25.798 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.798 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.798 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started", 2024-09-18T15:08:25.798 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:30.657042+0000", 2024-09-18T15:08:25.798 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:31.287418+0000" 2024-09-18T15:08:25.798 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:25.798 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:25.798 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.799 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.799 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": "102", 2024-09-18T15:08:25.799 INFO:tasks.workunit.client.0.smithi047.stdout: "states": [ 2024-09-18T15:08:25.799 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.799 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Reset", 2024-09-18T15:08:25.799 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:31.287419+0000", 2024-09-18T15:08:25.799 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:31.287798+0000" 2024-09-18T15:08:25.799 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:25.799 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:25.799 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.799 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.799 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": "115", 2024-09-18T15:08:25.799 INFO:tasks.workunit.client.0.smithi047.stdout: "states": [ 2024-09-18T15:08:25.799 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.800 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Start", 2024-09-18T15:08:25.800 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:31.287803+0000", 2024-09-18T15:08:25.800 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:31.287816+0000" 2024-09-18T15:08:25.800 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.800 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.800 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-18T15:08:25.800 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:31.287824+0000", 2024-09-18T15:08:25.800 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:31.421407+0000" 2024-09-18T15:08:25.800 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.800 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.800 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-18T15:08:25.800 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:31.421408+0000", 2024-09-18T15:08:25.800 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:31.421491+0000" 2024-09-18T15:08:25.800 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.801 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.801 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-18T15:08:25.801 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:31.421492+0000", 2024-09-18T15:08:25.801 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:31.421507+0000" 2024-09-18T15:08:25.801 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.801 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.801 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-18T15:08:25.801 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:31.421507+0000", 2024-09-18T15:08:25.801 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:31.697674+0000" 2024-09-18T15:08:25.801 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.801 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.801 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering", 2024-09-18T15:08:25.801 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:31.287820+0000", 2024-09-18T15:08:25.801 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:31.697684+0000" 2024-09-18T15:08:25.802 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.802 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.802 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/Activating", 2024-09-18T15:08:25.802 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:31.699316+0000", 2024-09-18T15:08:25.802 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:31.857800+0000" 2024-09-18T15:08:25.802 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.802 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.802 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2024-09-18T15:08:25.802 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:31.857801+0000", 2024-09-18T15:08:25.802 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:31.857975+0000" 2024-09-18T15:08:25.802 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.802 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.802 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2024-09-18T15:08:25.802 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:31.857976+0000", 2024-09-18T15:08:25.803 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:31.858308+0000" 2024-09-18T15:08:25.803 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.803 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.803 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/Recovering", 2024-09-18T15:08:25.803 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:31.858309+0000", 2024-09-18T15:08:25.803 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:58.501133+0000" 2024-09-18T15:08:25.803 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.803 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.803 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/NotRecovering", 2024-09-18T15:08:25.803 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:58.501134+0000", 2024-09-18T15:08:25.803 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:58.501301+0000" 2024-09-18T15:08:25.803 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.803 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.803 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2024-09-18T15:08:25.804 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:58.501302+0000", 2024-09-18T15:08:25.804 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:03:10.756038+0000" 2024-09-18T15:08:25.804 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.804 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.804 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2024-09-18T15:08:25.804 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:03:10.756038+0000", 2024-09-18T15:08:25.804 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:03:10.756380+0000" 2024-09-18T15:08:25.804 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.804 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.804 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/Recovering", 2024-09-18T15:08:25.804 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:03:10.756380+0000", 2024-09-18T15:08:25.804 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:04:56.245769+0000" 2024-09-18T15:08:25.804 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.804 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.805 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-18T15:08:25.805 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:04:56.245769+0000", 2024-09-18T15:08:25.805 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:04:56.631046+0000" 2024-09-18T15:08:25.805 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.805 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.805 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active", 2024-09-18T15:08:25.805 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:31.697685+0000", 2024-09-18T15:08:25.805 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:04:56.631075+0000" 2024-09-18T15:08:25.805 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.805 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.805 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary", 2024-09-18T15:08:25.805 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:31.287816+0000", 2024-09-18T15:08:25.805 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:04:56.631100+0000" 2024-09-18T15:08:25.805 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.806 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.806 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started", 2024-09-18T15:08:25.806 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:31.287798+0000", 2024-09-18T15:08:25.806 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:04:56.631136+0000" 2024-09-18T15:08:25.806 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:25.806 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:25.806 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.806 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.806 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": "115", 2024-09-18T15:08:25.806 INFO:tasks.workunit.client.0.smithi047.stdout: "states": [ 2024-09-18T15:08:25.806 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.806 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Reset", 2024-09-18T15:08:25.806 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:04:56.631137+0000", 2024-09-18T15:08:25.806 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:04:56.631554+0000" 2024-09-18T15:08:25.806 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:25.807 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:25.807 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:25.807 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:25.807 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.807 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.807 INFO:tasks.workunit.client.0.smithi047.stdout: "pg": "3.0", 2024-09-18T15:08:25.807 INFO:tasks.workunit.client.0.smithi047.stdout: "currently": "Started/Primary/Active/Clean", 2024-09-18T15:08:25.807 INFO:tasks.workunit.client.0.smithi047.stdout: "history": [ 2024-09-18T15:08:25.807 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.807 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": "42", 2024-09-18T15:08:25.807 INFO:tasks.workunit.client.0.smithi047.stdout: "states": [ 2024-09-18T15:08:25.807 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.807 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Start", 2024-09-18T15:08:25.807 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:56.801392+0000", 2024-09-18T15:08:25.807 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:56.801399+0000" 2024-09-18T15:08:25.808 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.808 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.808 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-18T15:08:25.808 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:56.801404+0000", 2024-09-18T15:08:25.808 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:56.923451+0000" 2024-09-18T15:08:25.808 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.808 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.808 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-18T15:08:25.808 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:56.923455+0000", 2024-09-18T15:08:25.808 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:56.923505+0000" 2024-09-18T15:08:25.808 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.808 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.808 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-18T15:08:25.808 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:56.923505+0000", 2024-09-18T15:08:25.809 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:56.923517+0000" 2024-09-18T15:08:25.809 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.809 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.809 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-18T15:08:25.809 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:56.923517+0000", 2024-09-18T15:08:25.809 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:57.290951+0000" 2024-09-18T15:08:25.809 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.809 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.809 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering", 2024-09-18T15:08:25.809 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:56.801402+0000", 2024-09-18T15:08:25.809 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:57.290962+0000" 2024-09-18T15:08:25.809 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.809 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.809 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/Activating", 2024-09-18T15:08:25.810 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:57.291161+0000", 2024-09-18T15:08:25.810 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:57.740525+0000" 2024-09-18T15:08:25.810 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.810 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.810 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-18T15:08:25.810 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:57.740526+0000", 2024-09-18T15:08:25.810 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:57.740570+0000" 2024-09-18T15:08:25.810 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.810 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.810 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/Clean", 2024-09-18T15:08:25.810 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:57.740570+0000", 2024-09-18T15:08:25.810 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:58.393641+0000" 2024-09-18T15:08:25.810 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.810 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.811 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active", 2024-09-18T15:08:25.811 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:57.290963+0000", 2024-09-18T15:08:25.811 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:58.393670+0000" 2024-09-18T15:08:25.811 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.811 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.811 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary", 2024-09-18T15:08:25.811 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:56.801399+0000", 2024-09-18T15:08:25.811 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:58.393704+0000" 2024-09-18T15:08:25.811 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.811 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.811 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started", 2024-09-18T15:08:25.811 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:56.801390+0000", 2024-09-18T15:08:25.811 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:58.393710+0000" 2024-09-18T15:08:25.811 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:25.811 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:25.812 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.812 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.812 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": "42", 2024-09-18T15:08:25.812 INFO:tasks.workunit.client.0.smithi047.stdout: "states": [ 2024-09-18T15:08:25.812 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.812 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Reset", 2024-09-18T15:08:25.812 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:58.393710+0000", 2024-09-18T15:08:25.812 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:58.394272+0000" 2024-09-18T15:08:25.812 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:25.812 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:25.812 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.812 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.812 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": "89", 2024-09-18T15:08:25.812 INFO:tasks.workunit.client.0.smithi047.stdout: "states": [ 2024-09-18T15:08:25.813 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.813 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Start", 2024-09-18T15:08:25.813 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:58.394278+0000", 2024-09-18T15:08:25.813 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:58.394297+0000" 2024-09-18T15:08:25.813 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.813 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.813 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-18T15:08:25.813 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:58.394310+0000", 2024-09-18T15:08:25.813 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:58.503400+0000" 2024-09-18T15:08:25.813 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.813 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.813 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-18T15:08:25.813 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:58.503400+0000", 2024-09-18T15:08:25.813 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:58.503457+0000" 2024-09-18T15:08:25.813 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.814 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.814 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-18T15:08:25.814 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:58.503457+0000", 2024-09-18T15:08:25.814 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:58.503468+0000" 2024-09-18T15:08:25.814 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.814 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.814 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-18T15:08:25.814 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:58.503468+0000", 2024-09-18T15:08:25.814 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:58.754287+0000" 2024-09-18T15:08:25.814 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.814 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.814 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering", 2024-09-18T15:08:25.814 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:58.394304+0000", 2024-09-18T15:08:25.814 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:58.754299+0000" 2024-09-18T15:08:25.815 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.815 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.815 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/Activating", 2024-09-18T15:08:25.815 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:58.754466+0000", 2024-09-18T15:08:25.815 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:59.366301+0000" 2024-09-18T15:08:25.815 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.815 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.815 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-18T15:08:25.815 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:59.366302+0000", 2024-09-18T15:08:25.815 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:59.366332+0000" 2024-09-18T15:08:25.815 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.815 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.815 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/Clean", 2024-09-18T15:08:25.815 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:59.366333+0000", 2024-09-18T15:08:25.816 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:37.798471+0000" 2024-09-18T15:08:25.816 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.816 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.816 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active", 2024-09-18T15:08:25.816 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:58.754300+0000", 2024-09-18T15:08:25.816 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:37.798490+0000" 2024-09-18T15:08:25.816 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.816 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.816 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary", 2024-09-18T15:08:25.816 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:58.394298+0000", 2024-09-18T15:08:25.816 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:37.798510+0000" 2024-09-18T15:08:25.816 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.816 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.816 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started", 2024-09-18T15:08:25.817 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:58.394273+0000", 2024-09-18T15:08:25.817 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:37.798512+0000" 2024-09-18T15:08:25.817 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:25.817 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:25.817 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.817 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.817 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": "89", 2024-09-18T15:08:25.817 INFO:tasks.workunit.client.0.smithi047.stdout: "states": [ 2024-09-18T15:08:25.817 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.817 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Reset", 2024-09-18T15:08:25.817 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:37.798513+0000", 2024-09-18T15:08:25.817 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:37.798796+0000" 2024-09-18T15:08:25.817 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:25.817 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:25.818 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.818 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.818 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": "107", 2024-09-18T15:08:25.818 INFO:tasks.workunit.client.0.smithi047.stdout: "states": [ 2024-09-18T15:08:25.818 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.818 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Start", 2024-09-18T15:08:25.818 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:37.798799+0000", 2024-09-18T15:08:25.818 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:37.798810+0000" 2024-09-18T15:08:25.818 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.818 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.818 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-18T15:08:25.818 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:37.798816+0000", 2024-09-18T15:08:25.818 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:37.826556+0000" 2024-09-18T15:08:25.818 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.819 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.819 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-18T15:08:25.819 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:37.826557+0000", 2024-09-18T15:08:25.819 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:37.826638+0000" 2024-09-18T15:08:25.819 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.819 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.819 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-18T15:08:25.819 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:37.826639+0000", 2024-09-18T15:08:25.819 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:37.826659+0000" 2024-09-18T15:08:25.819 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.819 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.819 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-18T15:08:25.819 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:37.826660+0000", 2024-09-18T15:08:25.819 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:38.116846+0000" 2024-09-18T15:08:25.819 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.820 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.820 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering", 2024-09-18T15:08:25.820 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:37.798814+0000", 2024-09-18T15:08:25.820 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:38.116855+0000" 2024-09-18T15:08:25.820 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.820 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.820 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/Activating", 2024-09-18T15:08:25.820 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:38.116933+0000", 2024-09-18T15:08:25.820 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:38.300686+0000" 2024-09-18T15:08:25.820 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.820 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.820 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-18T15:08:25.820 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:38.300686+0000", 2024-09-18T15:08:25.820 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:57:38.300711+0000" 2024-09-18T15:08:25.821 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.821 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.821 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/Clean", 2024-09-18T15:08:25.821 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:38.300711+0000", 2024-09-18T15:08:25.821 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:46.976300+0000" 2024-09-18T15:08:25.821 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.821 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.821 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active", 2024-09-18T15:08:25.821 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:38.116856+0000", 2024-09-18T15:08:25.821 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:46.976329+0000" 2024-09-18T15:08:25.821 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.821 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.821 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary", 2024-09-18T15:08:25.821 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:37.798811+0000", 2024-09-18T15:08:25.822 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:46.976512+0000" 2024-09-18T15:08:25.822 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.822 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.822 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started", 2024-09-18T15:08:25.822 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:57:37.798796+0000", 2024-09-18T15:08:25.822 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:46.976516+0000" 2024-09-18T15:08:25.822 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:25.822 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:25.822 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.822 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.822 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": "107", 2024-09-18T15:08:25.822 INFO:tasks.workunit.client.0.smithi047.stdout: "states": [ 2024-09-18T15:08:25.822 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.822 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Reset", 2024-09-18T15:08:25.823 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:46.976517+0000", 2024-09-18T15:08:25.823 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:46.976985+0000" 2024-09-18T15:08:25.823 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:25.823 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:25.823 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.823 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.823 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": "109", 2024-09-18T15:08:25.823 INFO:tasks.workunit.client.0.smithi047.stdout: "states": [ 2024-09-18T15:08:25.823 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.823 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Start", 2024-09-18T15:08:25.823 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:46.976989+0000", 2024-09-18T15:08:25.823 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:46.977002+0000" 2024-09-18T15:08:25.823 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.823 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.824 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-18T15:08:25.824 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:46.977009+0000", 2024-09-18T15:08:25.824 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:46.977720+0000" 2024-09-18T15:08:25.824 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.824 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.824 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-18T15:08:25.824 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:46.977721+0000", 2024-09-18T15:08:25.824 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:46.977784+0000" 2024-09-18T15:08:25.824 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.824 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.824 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering", 2024-09-18T15:08:25.824 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:46.977006+0000", 2024-09-18T15:08:25.824 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:46.977791+0000" 2024-09-18T15:08:25.824 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.825 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.825 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/WaitActingChange", 2024-09-18T15:08:25.825 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:46.977792+0000", 2024-09-18T15:08:25.825 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:47.370102+0000" 2024-09-18T15:08:25.825 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.825 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.825 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary", 2024-09-18T15:08:25.825 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:46.977003+0000", 2024-09-18T15:08:25.825 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:47.370134+0000" 2024-09-18T15:08:25.825 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.825 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.825 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started", 2024-09-18T15:08:25.825 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:46.976986+0000", 2024-09-18T15:08:25.826 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:47.370138+0000" 2024-09-18T15:08:25.826 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:25.826 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:25.826 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.826 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.826 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": "109", 2024-09-18T15:08:25.826 INFO:tasks.workunit.client.0.smithi047.stdout: "states": [ 2024-09-18T15:08:25.826 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.826 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Reset", 2024-09-18T15:08:25.826 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:47.370138+0000", 2024-09-18T15:08:25.826 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:47.370490+0000" 2024-09-18T15:08:25.826 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:25.826 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:25.826 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.826 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.827 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": "119", 2024-09-18T15:08:25.827 INFO:tasks.workunit.client.0.smithi047.stdout: "states": [ 2024-09-18T15:08:25.827 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.827 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Start", 2024-09-18T15:08:25.827 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:47.370494+0000", 2024-09-18T15:08:25.827 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:47.370507+0000" 2024-09-18T15:08:25.827 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.827 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.827 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-18T15:08:25.827 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:47.370515+0000", 2024-09-18T15:08:25.827 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:47.371108+0000" 2024-09-18T15:08:25.827 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.827 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.827 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-18T15:08:25.828 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:47.371108+0000", 2024-09-18T15:08:25.828 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:47.371181+0000" 2024-09-18T15:08:25.828 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.828 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.828 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-18T15:08:25.828 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:47.371182+0000", 2024-09-18T15:08:25.828 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:47.371197+0000" 2024-09-18T15:08:25.828 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.828 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.828 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-18T15:08:25.828 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:47.371198+0000", 2024-09-18T15:08:25.828 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:47.596347+0000" 2024-09-18T15:08:25.828 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.828 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.829 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering", 2024-09-18T15:08:25.829 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:47.370511+0000", 2024-09-18T15:08:25.829 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:47.596355+0000" 2024-09-18T15:08:25.829 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.829 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.829 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/Activating", 2024-09-18T15:08:25.829 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:47.597728+0000", 2024-09-18T15:08:25.829 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:48.006810+0000" 2024-09-18T15:08:25.829 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.829 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.829 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2024-09-18T15:08:25.829 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:48.006810+0000", 2024-09-18T15:08:25.829 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:58.501242+0000" 2024-09-18T15:08:25.829 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.830 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.830 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2024-09-18T15:08:25.830 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:58.501243+0000", 2024-09-18T15:08:25.830 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:58.501557+0000" 2024-09-18T15:08:25.830 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.830 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.830 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/Recovering", 2024-09-18T15:08:25.830 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:58.501561+0000", 2024-09-18T15:08:25.830 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:03:10.755929+0000" 2024-09-18T15:08:25.830 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.830 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.830 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/NotRecovering", 2024-09-18T15:08:25.830 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:03:10.755930+0000", 2024-09-18T15:08:25.830 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:03:10.756091+0000" 2024-09-18T15:08:25.830 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.831 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.831 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2024-09-18T15:08:25.831 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:03:10.756091+0000", 2024-09-18T15:08:25.831 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:06:37.314481+0000" 2024-09-18T15:08:25.831 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.831 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.831 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2024-09-18T15:08:25.831 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:06:37.314481+0000", 2024-09-18T15:08:25.831 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:06:37.314765+0000" 2024-09-18T15:08:25.831 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.831 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.831 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/Recovering", 2024-09-18T15:08:25.831 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:06:37.314765+0000", 2024-09-18T15:08:25.831 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:08:18.301159+0000" 2024-09-18T15:08:25.832 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.832 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.832 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-18T15:08:25.832 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:08:18.301160+0000", 2024-09-18T15:08:25.832 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:08:18.493209+0000" 2024-09-18T15:08:25.832 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.832 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.832 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active", 2024-09-18T15:08:25.832 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:47.596355+0000", 2024-09-18T15:08:25.832 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:08:18.493228+0000" 2024-09-18T15:08:25.832 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.832 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.832 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary", 2024-09-18T15:08:25.832 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:47.370508+0000", 2024-09-18T15:08:25.833 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:08:18.493252+0000" 2024-09-18T15:08:25.833 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.833 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.833 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started", 2024-09-18T15:08:25.833 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:47.370490+0000", 2024-09-18T15:08:25.833 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:08:18.493256+0000" 2024-09-18T15:08:25.833 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:25.833 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:25.833 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.833 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.833 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": "119", 2024-09-18T15:08:25.833 INFO:tasks.workunit.client.0.smithi047.stdout: "states": [ 2024-09-18T15:08:25.833 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.833 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Reset", 2024-09-18T15:08:25.833 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:08:18.493256+0000", 2024-09-18T15:08:25.834 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:08:18.493701+0000" 2024-09-18T15:08:25.834 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:25.834 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:25.834 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:25.834 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:25.834 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.834 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.834 INFO:tasks.workunit.client.0.smithi047.stdout: "pg": "1.0", 2024-09-18T15:08:25.834 INFO:tasks.workunit.client.0.smithi047.stdout: "currently": "Started/Primary/Active/Clean", 2024-09-18T15:08:25.834 INFO:tasks.workunit.client.0.smithi047.stdout: "history": [ 2024-09-18T15:08:25.834 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.834 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": "27", 2024-09-18T15:08:25.834 INFO:tasks.workunit.client.0.smithi047.stdout: "states": [ 2024-09-18T15:08:25.834 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.835 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Initial", 2024-09-18T15:08:25.835 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:51.814433+0000", 2024-09-18T15:08:25.835 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:51.814683+0000" 2024-09-18T15:08:25.835 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:25.835 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:25.835 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.835 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.835 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": "27", 2024-09-18T15:08:25.835 INFO:tasks.workunit.client.0.smithi047.stdout: "states": [ 2024-09-18T15:08:25.835 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.835 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Reset", 2024-09-18T15:08:25.835 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:51.814684+0000", 2024-09-18T15:08:25.835 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:51.814730+0000" 2024-09-18T15:08:25.835 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:25.835 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:25.836 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.836 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.836 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": "30", 2024-09-18T15:08:25.836 INFO:tasks.workunit.client.0.smithi047.stdout: "states": [ 2024-09-18T15:08:25.836 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.836 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Start", 2024-09-18T15:08:25.836 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:51.814736+0000", 2024-09-18T15:08:25.836 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:51.814750+0000" 2024-09-18T15:08:25.836 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.836 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.836 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-18T15:08:25.836 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:51.814774+0000", 2024-09-18T15:08:25.836 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:51.940238+0000" 2024-09-18T15:08:25.836 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.837 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.837 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-18T15:08:25.837 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:51.940239+0000", 2024-09-18T15:08:25.837 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:51.940310+0000" 2024-09-18T15:08:25.837 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.837 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.837 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-18T15:08:25.837 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:51.940311+0000", 2024-09-18T15:08:25.837 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:51.940321+0000" 2024-09-18T15:08:25.837 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.837 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.837 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-18T15:08:25.837 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:51.940321+0000", 2024-09-18T15:08:25.837 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:52.116058+0000" 2024-09-18T15:08:25.838 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.838 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.838 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering", 2024-09-18T15:08:25.838 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:51.814765+0000", 2024-09-18T15:08:25.838 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:52.116065+0000" 2024-09-18T15:08:25.838 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.838 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.838 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/Activating", 2024-09-18T15:08:25.838 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:52.116276+0000", 2024-09-18T15:08:25.838 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:52.368313+0000" 2024-09-18T15:08:25.838 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.838 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.838 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-18T15:08:25.838 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:52.368314+0000", 2024-09-18T15:08:25.838 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:52.368372+0000" 2024-09-18T15:08:25.839 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.839 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.839 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/Clean", 2024-09-18T15:08:25.839 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:52.368372+0000", 2024-09-18T15:08:25.839 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:53.139441+0000" 2024-09-18T15:08:25.839 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.839 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.839 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active", 2024-09-18T15:08:25.839 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:52.116066+0000", 2024-09-18T15:08:25.839 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:53.139469+0000" 2024-09-18T15:08:25.839 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.839 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.839 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary", 2024-09-18T15:08:25.839 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:51.814751+0000", 2024-09-18T15:08:25.840 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:53.139494+0000" 2024-09-18T15:08:25.840 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.840 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.840 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started", 2024-09-18T15:08:25.840 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:51.814731+0000", 2024-09-18T15:08:25.840 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:53.139499+0000" 2024-09-18T15:08:25.840 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:25.840 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:25.840 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.840 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.840 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": "30", 2024-09-18T15:08:25.840 INFO:tasks.workunit.client.0.smithi047.stdout: "states": [ 2024-09-18T15:08:25.840 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.840 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Reset", 2024-09-18T15:08:25.841 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:53.139499+0000", 2024-09-18T15:08:25.841 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:53.140012+0000" 2024-09-18T15:08:25.841 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:25.841 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:25.841 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.841 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.841 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": "104", 2024-09-18T15:08:25.841 INFO:tasks.workunit.client.0.smithi047.stdout: "states": [ 2024-09-18T15:08:25.841 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.841 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Start", 2024-09-18T15:08:25.841 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:53.140019+0000", 2024-09-18T15:08:25.841 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:53.140045+0000" 2024-09-18T15:08:25.841 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.841 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.842 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-18T15:08:25.842 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:53.140057+0000", 2024-09-18T15:08:25.842 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:53.249119+0000" 2024-09-18T15:08:25.842 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.842 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.842 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-18T15:08:25.842 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:53.249120+0000", 2024-09-18T15:08:25.842 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:53.249203+0000" 2024-09-18T15:08:25.842 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.842 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.842 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-18T15:08:25.842 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:53.249203+0000", 2024-09-18T15:08:25.842 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:53.249223+0000" 2024-09-18T15:08:25.842 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.842 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.843 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-18T15:08:25.843 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:53.249224+0000", 2024-09-18T15:08:25.843 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:53.734475+0000" 2024-09-18T15:08:25.843 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.843 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.843 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering", 2024-09-18T15:08:25.843 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:53.140051+0000", 2024-09-18T15:08:25.843 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:53.734483+0000" 2024-09-18T15:08:25.843 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.843 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.843 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/Activating", 2024-09-18T15:08:25.843 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:53.734590+0000", 2024-09-18T15:08:25.843 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:54.246727+0000" 2024-09-18T15:08:25.843 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.844 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.844 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-18T15:08:25.844 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:54.246728+0000", 2024-09-18T15:08:25.844 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T14:56:54.246787+0000" 2024-09-18T15:08:25.844 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.844 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.844 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/Clean", 2024-09-18T15:08:25.844 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:54.246787+0000", 2024-09-18T15:08:25.844 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:39.250728+0000" 2024-09-18T15:08:25.844 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.844 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.844 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active", 2024-09-18T15:08:25.844 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:53.734484+0000", 2024-09-18T15:08:25.844 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:39.250753+0000" 2024-09-18T15:08:25.845 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.845 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.845 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary", 2024-09-18T15:08:25.845 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:53.140046+0000", 2024-09-18T15:08:25.845 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:39.250947+0000" 2024-09-18T15:08:25.845 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.845 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.845 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started", 2024-09-18T15:08:25.845 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T14:56:53.140013+0000", 2024-09-18T15:08:25.845 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:39.250952+0000" 2024-09-18T15:08:25.845 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:25.845 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:25.845 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.845 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.845 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": "104", 2024-09-18T15:08:25.846 INFO:tasks.workunit.client.0.smithi047.stdout: "states": [ 2024-09-18T15:08:25.846 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.846 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Reset", 2024-09-18T15:08:25.846 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:39.250953+0000", 2024-09-18T15:08:25.846 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:39.251432+0000" 2024-09-18T15:08:25.846 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:25.846 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:25.846 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.846 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.846 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": "105", 2024-09-18T15:08:25.846 INFO:tasks.workunit.client.0.smithi047.stdout: "states": [ 2024-09-18T15:08:25.846 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.846 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Start", 2024-09-18T15:08:25.846 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:39.251436+0000", 2024-09-18T15:08:25.847 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:39.251449+0000" 2024-09-18T15:08:25.847 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.847 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.847 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-18T15:08:25.847 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:39.251455+0000", 2024-09-18T15:08:25.847 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:39.252043+0000" 2024-09-18T15:08:25.847 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.847 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.847 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-18T15:08:25.847 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:39.252044+0000", 2024-09-18T15:08:25.847 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:39.252113+0000" 2024-09-18T15:08:25.847 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.847 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.847 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering", 2024-09-18T15:08:25.847 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:39.251452+0000", 2024-09-18T15:08:25.848 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:39.252121+0000" 2024-09-18T15:08:25.848 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.848 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.848 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/WaitActingChange", 2024-09-18T15:08:25.848 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:39.252122+0000", 2024-09-18T15:08:25.848 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:39.577183+0000" 2024-09-18T15:08:25.848 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.848 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.848 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary", 2024-09-18T15:08:25.848 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:39.251449+0000", 2024-09-18T15:08:25.848 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:39.577206+0000" 2024-09-18T15:08:25.848 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.848 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.848 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started", 2024-09-18T15:08:25.849 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:39.251432+0000", 2024-09-18T15:08:25.849 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:39.577209+0000" 2024-09-18T15:08:25.849 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:25.849 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:25.849 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.849 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.849 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": "105", 2024-09-18T15:08:25.849 INFO:tasks.workunit.client.0.smithi047.stdout: "states": [ 2024-09-18T15:08:25.849 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.849 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Reset", 2024-09-18T15:08:25.849 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:39.577210+0000", 2024-09-18T15:08:25.849 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:39.577543+0000" 2024-09-18T15:08:25.849 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:25.849 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:25.849 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.850 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.850 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": "117", 2024-09-18T15:08:25.850 INFO:tasks.workunit.client.0.smithi047.stdout: "states": [ 2024-09-18T15:08:25.850 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.850 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Start", 2024-09-18T15:08:25.850 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:39.577547+0000", 2024-09-18T15:08:25.850 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:39.577559+0000" 2024-09-18T15:08:25.850 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.850 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.850 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-18T15:08:25.850 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:39.577567+0000", 2024-09-18T15:08:25.850 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:39.694958+0000" 2024-09-18T15:08:25.850 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.850 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.851 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-18T15:08:25.851 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:39.694959+0000", 2024-09-18T15:08:25.851 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:39.695051+0000" 2024-09-18T15:08:25.851 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.851 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.851 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-18T15:08:25.851 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:39.695051+0000", 2024-09-18T15:08:25.851 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:39.695066+0000" 2024-09-18T15:08:25.851 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.851 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.851 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-18T15:08:25.851 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:39.695066+0000", 2024-09-18T15:08:25.851 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:39.929515+0000" 2024-09-18T15:08:25.851 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.851 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.852 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Peering", 2024-09-18T15:08:25.852 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:39.577563+0000", 2024-09-18T15:08:25.852 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:39.929523+0000" 2024-09-18T15:08:25.852 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.852 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.852 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/Activating", 2024-09-18T15:08:25.852 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:39.930953+0000", 2024-09-18T15:08:25.852 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:02:40.508084+0000" 2024-09-18T15:08:25.852 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.852 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.852 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2024-09-18T15:08:25.852 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:40.508085+0000", 2024-09-18T15:08:25.852 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:04:56.245950+0000" 2024-09-18T15:08:25.852 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.853 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.853 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2024-09-18T15:08:25.853 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:04:56.245951+0000", 2024-09-18T15:08:25.853 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:04:56.246257+0000" 2024-09-18T15:08:25.853 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.853 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.853 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/Recovering", 2024-09-18T15:08:25.853 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:04:56.246261+0000", 2024-09-18T15:08:25.853 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:06:37.314329+0000" 2024-09-18T15:08:25.854 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.854 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.854 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-18T15:08:25.854 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:06:37.314330+0000", 2024-09-18T15:08:25.854 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:06:37.766468+0000" 2024-09-18T15:08:25.854 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.854 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.854 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary/Active", 2024-09-18T15:08:25.854 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:39.929524+0000", 2024-09-18T15:08:25.854 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:06:37.766491+0000" 2024-09-18T15:08:25.854 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.854 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.854 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started/Primary", 2024-09-18T15:08:25.854 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:39.577559+0000", 2024-09-18T15:08:25.854 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:06:37.766515+0000" 2024-09-18T15:08:25.855 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.855 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.855 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Started", 2024-09-18T15:08:25.855 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:02:39.577543+0000", 2024-09-18T15:08:25.855 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:06:37.766518+0000" 2024-09-18T15:08:25.855 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:25.855 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:25.855 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:08:25.855 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.856 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": "117", 2024-09-18T15:08:25.856 INFO:tasks.workunit.client.0.smithi047.stdout: "states": [ 2024-09-18T15:08:25.856 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:08:25.856 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "Reset", 2024-09-18T15:08:25.856 INFO:tasks.workunit.client.0.smithi047.stdout: "enter": "2024-09-18T15:06:37.766518+0000", 2024-09-18T15:08:25.856 INFO:tasks.workunit.client.0.smithi047.stdout: "exit": "2024-09-18T15:06:37.766940+0000" 2024-09-18T15:08:25.856 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:25.856 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:25.856 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:25.856 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:25.856 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:08:25.856 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:08:25.856 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:08:25.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:302: TEST_recovery_priority: '[' 0 '!=' 0 ']' 2024-09-18T15:08:25.857 INFO:tasks.workunit.client.0.smithi047.stdout:TEST PASSED 2024-09-18T15:08:25.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:306: TEST_recovery_priority: echo TEST PASSED 2024-09-18T15:08:25.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:309: TEST_recovery_priority: delete_pool test1 2024-09-18T15:08:25.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test1 2024-09-18T15:08:25.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2024-09-18T15:08:26.266 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test1' does not exist 2024-09-18T15:08:26.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:310: TEST_recovery_priority: delete_pool test3 2024-09-18T15:08:26.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test3 2024-09-18T15:08:26.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2024-09-18T15:08:26.817 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test3' does not exist 2024-09-18T15:08:26.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:311: TEST_recovery_priority: delete_pool test7 2024-09-18T15:08:26.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test7 2024-09-18T15:08:26.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test7 test7 --yes-i-really-really-mean-it 2024-09-18T15:08:27.386 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test7' does not exist 2024-09-18T15:08:27.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:312: TEST_recovery_priority: kill_daemons td/osd-recovery-prio 2024-09-18T15:08:27.398 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:08:27.398 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:08:27.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:08:27.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:08:27.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:08:32.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:08:32.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:313: TEST_recovery_priority: return 0 2024-09-18T15:08:32.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:41: run: teardown td/osd-recovery-prio 2024-09-18T15:08:32.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2024-09-18T15:08:32.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:08:32.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2024-09-18T15:08:32.715 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:08:32.715 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:08:32.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:08:32.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:08:32.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:08:32.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:08:32.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:08:32.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:08:32.725 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:08:32.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:08:32.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:08:32.727 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:08:32.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:08:32.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:08:32.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:08:32.728 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:08:32.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:08:32.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:08:32.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2024-09-18T15:08:32.808 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:08:32.808 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:08:32.808 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215238 2024-09-18T15:08:32.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.215238 2024-09-18T15:08:32.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:08:32.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:08:32.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T15:08:32.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/osd-recovery-prio 0 2024-09-18T15:08:32.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2024-09-18T15:08:32.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T15:08:32.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2024-09-18T15:08:32.811 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:08:32.811 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:08:32.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:08:32.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:08:32.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:08:32.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:08:32.812 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:08:32.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:08:32.813 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:08:32.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:08:32.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:08:32.815 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:08:32.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:08:32.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:08:32.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:08:32.816 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:08:32.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:08:32.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T15:08:32.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2024-09-18T15:08:32.819 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:08:32.819 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:08:32.819 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215238 2024-09-18T15:08:32.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.215238 2024-09-18T15:08:32.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:08:32.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:08:32.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T15:08:32.821 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T15:08:32.822 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T15:08:32.849 INFO:tasks.workunit:Running workunit osd/osd-recovery-space.sh... 2024-09-18T15:08:32.849 DEBUG:teuthology.orchestra.run.smithi047:workunit test osd/osd-recovery-space.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh 2024-09-18T15:08:32.912 INFO:tasks.workunit.client.0.smithi047.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T15:08:32.918 INFO:tasks.workunit.client.0.smithi047.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T15:08:32.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T15:08:32.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T15:08:32.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T15:08:32.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T15:08:32.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T15:08:32.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T15:08:32.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T15:08:32.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T15:08:32.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/osd-recovery-space 2024-09-18T15:08:32.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:21: run: local dir=td/osd-recovery-space 2024-09-18T15:08:32.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:22: run: shift 2024-09-18T15:08:32.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:24: run: export CEPH_MON=127.0.0.1:7221 2024-09-18T15:08:32.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:24: run: CEPH_MON=127.0.0.1:7221 2024-09-18T15:08:32.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:25: run: export CEPH_ARGS 2024-09-18T15:08:32.920 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:26: run: uuidgen 2024-09-18T15:08:32.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:26: run: CEPH_ARGS+='--fsid=961d5ebb-7a59-4c27-b062-d9452168f211 --auth-supported=none ' 2024-09-18T15:08:32.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7221 ' 2024-09-18T15:08:32.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:28: run: CEPH_ARGS+='--osd_max_backfills=10 ' 2024-09-18T15:08:32.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:29: run: CEPH_ARGS+='--osd_mclock_override_recovery_settings=true ' 2024-09-18T15:08:32.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:30: run: export objects=600 2024-09-18T15:08:32.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:30: run: objects=600 2024-09-18T15:08:32.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:31: run: export poolprefix=test 2024-09-18T15:08:32.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:31: run: poolprefix=test 2024-09-18T15:08:32.921 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:33: run: set 2024-09-18T15:08:32.921 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:33: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-18T15:08:32.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:33: run: local funcs=TEST_recovery_test_simple 2024-09-18T15:08:32.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:34: run: for func in $funcs 2024-09-18T15:08:32.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:35: run: setup td/osd-recovery-space 2024-09-18T15:08:32.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-space 2024-09-18T15:08:32.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-space 2024-09-18T15:08:32.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-space 2024-09-18T15:08:32.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:08:32.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-space KILL 2024-09-18T15:08:32.923 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:08:32.923 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:08:32.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:08:32.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:08:32.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:08:32.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:08:32.924 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:08:32.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:08:32.925 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:08:32.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:08:32.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:08:32.925 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:08:32.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:08:32.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:08:32.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:08:32.926 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:08:32.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:08:32.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:08:32.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-space 2024-09-18T15:08:32.929 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:08:32.929 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:08:32.929 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.271403 2024-09-18T15:08:32.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.271403 2024-09-18T15:08:32.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:08:32.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:08:32.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-space 2024-09-18T15:08:32.931 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T15:08:32.931 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:08:32.931 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.271403 2024-09-18T15:08:32.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.271403 2024-09-18T15:08:32.934 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T15:08:32.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T15:08:32.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T15:08:32.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T15:08:32.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-space 1' TERM HUP INT 2024-09-18T15:08:32.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:36: run: TEST_recovery_test_simple td/osd-recovery-space 2024-09-18T15:08:32.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:93: TEST_recovery_test_simple: local dir=td/osd-recovery-space 2024-09-18T15:08:32.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:94: TEST_recovery_test_simple: local pools=1 2024-09-18T15:08:32.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:95: TEST_recovery_test_simple: local OSDS=2 2024-09-18T15:08:32.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:97: TEST_recovery_test_simple: run_mon td/osd-recovery-space a 2024-09-18T15:08:32.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-space 2024-09-18T15:08:32.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T15:08:32.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T15:08:32.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T15:08:32.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-space/a 2024-09-18T15:08:32.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-space/a --run-dir=td/osd-recovery-space 2024-09-18T15:08:33.302 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T15:08:33.302 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:08:33.302 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:08:33.302 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:08:33.302 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:08:33.302 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.271403 2024-09-18T15:08:33.303 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.271403/$cluster-$name.asok' 2024-09-18T15:08:33.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-space/a '--log-file=td/osd-recovery-space/$name.log' '--admin-socket=/tmp/ceph-asok.271403/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-space/log --run-dir=td/osd-recovery-space '--pid-file=td/osd-recovery-space/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T15:08:33.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T15:08:33.555 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T15:08:33.555 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:08:33.555 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:08:33.555 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T15:08:33.555 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T15:08:33.555 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:08:33.555 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:08:33.555 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:08:33.555 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:08:33.555 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:08:33.555 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.271403 2024-09-18T15:08:33.556 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.271403/ceph-mon.a.asok 2024-09-18T15:08:33.556 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:08:33.556 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.271403/ceph-mon.a.asok config get fsid 2024-09-18T15:08:33.637 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T15:08:33.637 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:08:33.637 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:08:33.637 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T15:08:33.638 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T15:08:33.638 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:08:33.638 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:08:33.638 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:08:33.638 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:08:33.638 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:08:33.638 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.271403 2024-09-18T15:08:33.638 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.271403/ceph-mon.a.asok 2024-09-18T15:08:33.638 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:08:33.638 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.271403/ceph-mon.a.asok config get mon_host 2024-09-18T15:08:33.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:98: TEST_recovery_test_simple: run_mgr td/osd-recovery-space x 2024-09-18T15:08:33.714 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-space 2024-09-18T15:08:33.714 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T15:08:33.714 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T15:08:33.714 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T15:08:33.714 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-space/x 2024-09-18T15:08:33.714 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T15:08:34.559 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T15:08:34.559 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:08:34.559 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:08:34.559 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:08:34.559 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:08:34.559 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.271403 2024-09-18T15:08:34.559 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.271403/$cluster-$name.asok' 2024-09-18T15:08:34.560 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:08:34.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-space/x '--log-file=td/osd-recovery-space/$name.log' '--admin-socket=/tmp/ceph-asok.271403/$cluster-$name.asok' --run-dir=td/osd-recovery-space '--pid-file=td/osd-recovery-space/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:08:34.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:99: TEST_recovery_test_simple: export CEPH_ARGS 2024-09-18T15:08:34.585 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:101: TEST_recovery_test_simple: expr 2 - 1 2024-09-18T15:08:34.586 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:101: TEST_recovery_test_simple: seq 0 1 2024-09-18T15:08:34.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:101: TEST_recovery_test_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T15:08:34.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:103: TEST_recovery_test_simple: run_osd td/osd-recovery-space 0 2024-09-18T15:08:34.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-space 2024-09-18T15:08:34.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:08:34.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T15:08:34.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:08:34.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-space/0 2024-09-18T15:08:34.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=961d5ebb-7a59-4c27-b062-d9452168f211 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true ' 2024-09-18T15:08:34.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:08:34.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:08:34.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:08:34.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-space/0' 2024-09-18T15:08:34.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-space/0/journal' 2024-09-18T15:08:34.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:08:34.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:08:34.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-space' 2024-09-18T15:08:34.591 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:08:34.591 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:08:34.591 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:08:34.592 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:08:34.592 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:08:34.592 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.271403 2024-09-18T15:08:34.592 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.271403/$cluster-$name.asok' 2024-09-18T15:08:34.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.271403/$cluster-$name.asok' 2024-09-18T15:08:34.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:08:34.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:08:34.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:08:34.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-space/$name.log' 2024-09-18T15:08:34.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-space/$name.pid' 2024-09-18T15:08:34.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:08:34.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:08:34.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:08:34.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:08:34.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:08:34.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:08:34.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-space/0 2024-09-18T15:08:34.593 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:08:34.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ae9f9cd4-3b3d-4842-aca6-b2b6407dea9d 2024-09-18T15:08:34.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ae9f9cd4-3b3d-4842-aca6-b2b6407dea9d' 2024-09-18T15:08:34.594 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 ae9f9cd4-3b3d-4842-aca6-b2b6407dea9d 2024-09-18T15:08:34.594 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:08:34.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDy7OpmBGIcJBAAk7k2nqQ0FaJ/sJwVJiYQdQ== 2024-09-18T15:08:34.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDy7OpmBGIcJBAAk7k2nqQ0FaJ/sJwVJiYQdQ=="}' 2024-09-18T15:08:34.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ae9f9cd4-3b3d-4842-aca6-b2b6407dea9d -i td/osd-recovery-space/0/new.json 2024-09-18T15:08:34.901 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:08:34.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-space/0/new.json 2024-09-18T15:08:34.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=961d5ebb-7a59-4c27-b062-d9452168f211 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/0 --osd-journal=td/osd-recovery-space/0/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.271403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDy7OpmBGIcJBAAk7k2nqQ0FaJ/sJwVJiYQdQ== --osd-uuid ae9f9cd4-3b3d-4842-aca6-b2b6407dea9d 2024-09-18T15:08:34.938 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:08:34.937+0000 7f705f20a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:08:34.946 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:08:34.945+0000 7f705f20a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:08:34.947 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:08:34.946+0000 7f705f20a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:08:34.947 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:08:34.946+0000 7f705f20a440 -1 bluestore(td/osd-recovery-space/0/block) _read_bdev_label failed to open td/osd-recovery-space/0/block: (2) No such file or directory 2024-09-18T15:08:34.948 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:08:34.946+0000 7f705f20a440 -1 bluestore(td/osd-recovery-space/0/block) _read_bdev_label failed to open td/osd-recovery-space/0/block: (2) No such file or directory 2024-09-18T15:08:34.948 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:08:34.946+0000 7f705f20a440 -1 bluestore(td/osd-recovery-space/0/block) _read_bdev_label failed to open td/osd-recovery-space/0/block: (2) No such file or directory 2024-09-18T15:08:34.986 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:08:34.985+0000 7f705f20a440 -1 bluestore(td/osd-recovery-space/0) _read_fsid unparsable uuid 2024-09-18T15:08:40.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-space/0/keyring 2024-09-18T15:08:40.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:08:40.329 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T15:08:40.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T15:08:40.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-space/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:08:40.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T15:08:40.855 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T15:08:40.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=961d5ebb-7a59-4c27-b062-d9452168f211 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/0 --osd-journal=td/osd-recovery-space/0/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.271403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:08:40.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:08:40.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:08:40.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:08:40.880 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:08:40.879+0000 7fb279823440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:08:40.882 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:08:40.881+0000 7fb279823440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:08:40.883 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:08:40.882+0000 7fb279823440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:08:41.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T15:08:41.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:08:41.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:08:41.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:08:41.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:08:41.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:08:41.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:08:41.221 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:08:41.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:08:41.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:08:41.401 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:08:41.400+0000 7fb279823440 -1 Falling back to public interface 2024-09-18T15:08:41.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:08:42.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:08:42.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:08:42.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:08:42.634 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:08:42.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:08:42.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:08:43.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:08:43.873 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:08:43.871+0000 7fb279823440 -1 osd.0 0 log_to_monitors true 2024-09-18T15:08:44.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:08:44.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:08:44.013 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:08:44.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:08:44.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:08:44.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:08:44.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:08:45.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:08:45.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:08:45.441 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:08:45.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:08:45.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:08:45.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:08:45.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:08:46.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:08:46.870 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:08:46.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:08:46.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:08:46.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:08:46.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:08:47.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:08:48.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:08:48.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:08:48.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:08:48.256 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:08:48.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:08:48.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:08:48.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:08:49.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:08:49.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:08:49.682 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:08:49.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:08:49.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:08:49.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:08:50.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:08:51.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:08:51.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:08:51.071 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:08:51.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:08:51.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:08:51.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:08:51.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:08:52.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:08:52.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:08:52.586 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:08:52.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:08:52.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:08:52.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:08:52.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:08:53.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:08:53.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:08:53.972 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:08:53.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:08:53.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:08:53.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:08:54.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:08:55.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:08:55.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:08:55.362 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:08:55.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:08:55.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:08:55.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:08:56.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:08:57.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:08:57.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:08:57.259 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:08:57.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:08:57.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:08:57.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:08:57.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:08:58.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:08:58.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:08:58.738 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:08:58.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:08:58.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:08:58.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:08:59.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:09:00.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:09:00.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:09:00.635 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:09:00.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:09:00.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:09:00.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:09:01.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:09:02.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:09:02.589 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:09:02.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:09:02.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:09:02.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:09:02.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:09:03.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:09:04.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:09:04.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:09:04.398 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:09:04.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:09:04.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:09:04.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:09:05.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:09:06.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:09:06.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:09:06.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:09:06.209 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:09:06.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:09:06.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:09:06.587 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/272496,v1:127.0.0.1:6803/272496] [v2:127.0.0.1:6804/272496,v1:127.0.0.1:6805/272496] exists,up ae9f9cd4-3b3d-4842-aca6-b2b6407dea9d 2024-09-18T15:09:06.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:09:06.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:09:06.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:09:06.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:101: TEST_recovery_test_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T15:09:06.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:103: TEST_recovery_test_simple: run_osd td/osd-recovery-space 1 2024-09-18T15:09:06.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-space 2024-09-18T15:09:06.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:09:06.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T15:09:06.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:09:06.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-space/1 2024-09-18T15:09:06.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=961d5ebb-7a59-4c27-b062-d9452168f211 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true ' 2024-09-18T15:09:06.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:09:06.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:09:06.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:09:06.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-space/1' 2024-09-18T15:09:06.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-space/1/journal' 2024-09-18T15:09:06.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:09:06.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:09:06.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-space' 2024-09-18T15:09:06.593 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:09:06.593 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:09:06.593 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:09:06.593 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:09:06.593 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:09:06.593 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.271403 2024-09-18T15:09:06.593 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.271403/$cluster-$name.asok' 2024-09-18T15:09:06.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.271403/$cluster-$name.asok' 2024-09-18T15:09:06.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:09:06.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:09:06.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:09:06.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-space/$name.log' 2024-09-18T15:09:06.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-space/$name.pid' 2024-09-18T15:09:06.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:09:06.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:09:06.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:09:06.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:09:06.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:09:06.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:09:06.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-space/1 2024-09-18T15:09:06.594 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:09:06.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7132dd11-4630-44ed-9dd0-c86e3ef796a7 2024-09-18T15:09:06.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 7132dd11-4630-44ed-9dd0-c86e3ef796a7' 2024-09-18T15:09:06.594 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 7132dd11-4630-44ed-9dd0-c86e3ef796a7 2024-09-18T15:09:06.595 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:09:06.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAS7epmkJQkJBAApGKAhdZa1Y4/aBRSNrmr2w== 2024-09-18T15:09:06.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAS7epmkJQkJBAApGKAhdZa1Y4/aBRSNrmr2w=="}' 2024-09-18T15:09:06.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7132dd11-4630-44ed-9dd0-c86e3ef796a7 -i td/osd-recovery-space/1/new.json 2024-09-18T15:09:07.126 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:09:07.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-space/1/new.json 2024-09-18T15:09:07.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=961d5ebb-7a59-4c27-b062-d9452168f211 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/1 --osd-journal=td/osd-recovery-space/1/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.271403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAS7epmkJQkJBAApGKAhdZa1Y4/aBRSNrmr2w== --osd-uuid 7132dd11-4630-44ed-9dd0-c86e3ef796a7 2024-09-18T15:09:07.166 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:09:07.164+0000 7feee2471440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:09:07.167 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:09:07.166+0000 7feee2471440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:09:07.168 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:09:07.167+0000 7feee2471440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:09:07.169 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:09:07.167+0000 7feee2471440 -1 bluestore(td/osd-recovery-space/1/block) _read_bdev_label failed to open td/osd-recovery-space/1/block: (2) No such file or directory 2024-09-18T15:09:07.169 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:09:07.167+0000 7feee2471440 -1 bluestore(td/osd-recovery-space/1/block) _read_bdev_label failed to open td/osd-recovery-space/1/block: (2) No such file or directory 2024-09-18T15:09:07.169 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:09:07.168+0000 7feee2471440 -1 bluestore(td/osd-recovery-space/1/block) _read_bdev_label failed to open td/osd-recovery-space/1/block: (2) No such file or directory 2024-09-18T15:09:07.215 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:09:07.214+0000 7feee2471440 -1 bluestore(td/osd-recovery-space/1) _read_fsid unparsable uuid 2024-09-18T15:09:13.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-space/1/keyring 2024-09-18T15:09:13.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:09:13.046 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T15:09:13.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T15:09:13.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-space/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:09:13.556 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T15:09:13.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T15:09:13.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=961d5ebb-7a59-4c27-b062-d9452168f211 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/1 --osd-journal=td/osd-recovery-space/1/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.271403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:09:13.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:09:13.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:09:13.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:09:13.581 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:09:13.580+0000 7f703494c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:09:13.605 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:09:13.603+0000 7f703494c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:09:13.606 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:09:13.604+0000 7f703494c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:09:14.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T15:09:14.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:09:14.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:09:14.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:09:14.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:09:14.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:09:14.002 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:09:14.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:09:14.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:09:14.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:09:14.150 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:09:14.149+0000 7f703494c440 -1 Falling back to public interface 2024-09-18T15:09:14.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:09:15.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:09:15.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:09:15.377 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:09:15.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:09:15.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:09:15.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:09:15.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:09:16.684 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:09:16.683+0000 7f703494c440 -1 osd.1 0 log_to_monitors true 2024-09-18T15:09:16.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:09:16.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:09:16.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:09:16.757 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:09:16.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:09:16.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:09:17.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:09:18.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:09:18.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:09:18.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:09:18.228 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:09:18.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:09:18.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:09:18.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:09:19.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:09:19.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:09:19.727 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:09:19.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:09:19.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:09:19.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:09:20.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:09:21.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:09:21.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:09:21.108 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:09:21.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:09:21.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:09:21.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:09:21.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:09:22.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:09:22.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:09:22.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:09:22.558 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:09:22.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:09:22.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:09:23.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:09:24.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:09:24.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:09:24.087 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:09:24.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:09:24.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:09:24.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:09:24.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:09:25.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:09:25.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:09:25.616 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:09:25.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:09:25.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:09:25.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:09:26.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:09:27.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:09:27.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:09:27.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:09:27.028 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:09:27.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:09:27.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:09:27.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:09:28.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:09:28.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:09:28.528 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:09:28.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:09:28.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:09:28.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:09:29.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:09:30.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:09:30.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:09:30.359 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:09:30.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:09:30.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:09:30.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:09:31.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:09:32.419 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:09:32.419 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:09:32.419 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:09:32.419 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:09:32.420 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:09:32.420 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:09:33.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:09:34.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:09:34.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:09:34.222 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:09:34.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:09:34.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:09:34.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:09:34.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:09:35.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:09:35.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:09:35.694 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:09:35.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:09:35.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:09:35.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:09:36.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:09:37.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:09:37.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:09:37.369 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:09:37.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:09:37.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:09:37.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:09:38.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:09:39.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:09:39.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:09:39.490 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:09:39.491 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:09:39.491 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:09:39.491 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:09:39.944 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/274225,v1:127.0.0.1:6811/274225] [v2:127.0.0.1:6812/274225,v1:127.0.0.1:6813/274225] exists,up 7132dd11-4630-44ed-9dd0-c86e3ef796a7 2024-09-18T15:09:39.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:09:39.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:09:39.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:09:39.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:106: TEST_recovery_test_simple: ceph osd set-nearfull-ratio .40 2024-09-18T15:09:40.608 INFO:tasks.workunit.client.0.smithi047.stderr:osd set-nearfull-ratio 0.4 2024-09-18T15:09:40.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:107: TEST_recovery_test_simple: ceph osd set-backfillfull-ratio .45 2024-09-18T15:09:41.344 INFO:tasks.workunit.client.0.smithi047.stderr:osd set-backfillfull-ratio 0.45 2024-09-18T15:09:41.355 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:108: TEST_recovery_test_simple: ceph osd set-full-ratio .50 2024-09-18T15:09:42.081 INFO:tasks.workunit.client.0.smithi047.stderr:osd set-full-ratio 0.5 2024-09-18T15:09:42.092 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:110: TEST_recovery_test_simple: seq 1 1 2024-09-18T15:09:42.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:110: TEST_recovery_test_simple: for p in $(seq 1 $pools) 2024-09-18T15:09:42.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:112: TEST_recovery_test_simple: create_pool test1 1 1 2024-09-18T15:09:42.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2024-09-18T15:09:42.753 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test1' already exists 2024-09-18T15:09:42.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:09:43.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:113: TEST_recovery_test_simple: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2024-09-18T15:09:44.634 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 1 size to 1 2024-09-18T15:09:44.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: wait_for_clean 2024-09-18T15:09:44.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:09:44.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:09:44.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:09:44.647 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:09:44.647 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:09:44.647 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:09:44.647 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:09:44.647 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:09:44.647 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:09:44.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:09:44.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:09:44.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:09:44.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:09:44.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:09:44.737 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:09:45.197 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:09:45.197 INFO:tasks.workunit.client.0.smithi047.stderr:1' 2024-09-18T15:09:45.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:09:45.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:09:45.198 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:09:45.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836494 2024-09-18T15:09:45.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836494 2024-09-18T15:09:45.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836494' 2024-09-18T15:09:45.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:09:45.308 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:09:45.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=38654705674 2024-09-18T15:09:45.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 38654705674 2024-09-18T15:09:45.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836494 1-38654705674' 2024-09-18T15:09:45.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:09:45.419 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836494 2024-09-18T15:09:45.419 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:09:45.419 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:09:45.420 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836494 2024-09-18T15:09:45.420 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:09:45.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836494 2024-09-18T15:09:45.422 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836494 2024-09-18T15:09:45.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-09-18T15:09:45.422 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:09:45.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836492 -lt 21474836494 2024-09-18T15:09:45.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:09:46.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:09:46.794 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:09:47.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836495 -lt 21474836494 2024-09-18T15:09:47.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:09:47.218 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-38654705674 2024-09-18T15:09:47.218 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:09:47.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:09:47.219 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-38654705674 2024-09-18T15:09:47.219 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:09:47.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=38654705674 2024-09-18T15:09:47.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 38654705674' 2024-09-18T15:09:47.221 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 38654705674 2024-09-18T15:09:47.221 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:09:47.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 38654705674 -lt 38654705674 2024-09-18T15:09:47.591 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:09:47.591 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:09:47.591 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:09:48.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:09:48.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:09:48.087 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:09:48.087 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:09:48.087 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:09:48.087 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:09:48.087 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:09:48.087 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:09:48.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:09:48.488 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:09:48.488 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:09:48.488 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:09:48.988 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:09:48.988 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:09:48.988 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:09:48.989 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:118: TEST_recovery_test_simple: dd if=/dev/urandom of=td/osd-recovery-space/datafile bs=1024 count=5 2024-09-18T15:09:48.989 INFO:tasks.workunit.client.0.smithi047.stderr:5+0 records in 2024-09-18T15:09:48.989 INFO:tasks.workunit.client.0.smithi047.stderr:5+0 records out 2024-09-18T15:09:48.989 INFO:tasks.workunit.client.0.smithi047.stderr:5120 bytes (5.1 kB, 5.0 KiB) copied, 7.0382e-05 s, 72.7 MB/s 2024-09-18T15:09:48.989 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: seq 1 600 2024-09-18T15:09:48.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:48.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj1 td/osd-recovery-space/datafile 2024-09-18T15:09:49.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:49.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj2 td/osd-recovery-space/datafile 2024-09-18T15:09:49.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:49.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj3 td/osd-recovery-space/datafile 2024-09-18T15:09:49.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:49.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj4 td/osd-recovery-space/datafile 2024-09-18T15:09:49.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:49.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj5 td/osd-recovery-space/datafile 2024-09-18T15:09:49.432 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:49.432 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj6 td/osd-recovery-space/datafile 2024-09-18T15:09:49.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:49.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj7 td/osd-recovery-space/datafile 2024-09-18T15:09:49.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:49.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj8 td/osd-recovery-space/datafile 2024-09-18T15:09:49.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:49.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj9 td/osd-recovery-space/datafile 2024-09-18T15:09:49.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:49.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj10 td/osd-recovery-space/datafile 2024-09-18T15:09:49.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:49.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj11 td/osd-recovery-space/datafile 2024-09-18T15:09:49.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:49.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj12 td/osd-recovery-space/datafile 2024-09-18T15:09:49.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:49.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj13 td/osd-recovery-space/datafile 2024-09-18T15:09:50.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:50.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj14 td/osd-recovery-space/datafile 2024-09-18T15:09:50.135 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:50.135 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj15 td/osd-recovery-space/datafile 2024-09-18T15:09:50.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:50.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj16 td/osd-recovery-space/datafile 2024-09-18T15:09:50.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:50.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj17 td/osd-recovery-space/datafile 2024-09-18T15:09:50.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:50.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj18 td/osd-recovery-space/datafile 2024-09-18T15:09:50.469 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:50.470 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj19 td/osd-recovery-space/datafile 2024-09-18T15:09:50.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:50.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj20 td/osd-recovery-space/datafile 2024-09-18T15:09:50.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:50.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj21 td/osd-recovery-space/datafile 2024-09-18T15:09:50.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:50.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj22 td/osd-recovery-space/datafile 2024-09-18T15:09:50.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:50.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj23 td/osd-recovery-space/datafile 2024-09-18T15:09:50.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:50.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj24 td/osd-recovery-space/datafile 2024-09-18T15:09:50.988 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:50.989 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj25 td/osd-recovery-space/datafile 2024-09-18T15:09:51.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:51.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj26 td/osd-recovery-space/datafile 2024-09-18T15:09:51.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:51.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj27 td/osd-recovery-space/datafile 2024-09-18T15:09:51.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:51.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj28 td/osd-recovery-space/datafile 2024-09-18T15:09:51.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:51.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj29 td/osd-recovery-space/datafile 2024-09-18T15:09:51.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:51.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj30 td/osd-recovery-space/datafile 2024-09-18T15:09:51.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:51.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj31 td/osd-recovery-space/datafile 2024-09-18T15:09:51.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:51.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj32 td/osd-recovery-space/datafile 2024-09-18T15:09:51.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:51.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj33 td/osd-recovery-space/datafile 2024-09-18T15:09:51.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:51.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj34 td/osd-recovery-space/datafile 2024-09-18T15:09:51.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:51.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj35 td/osd-recovery-space/datafile 2024-09-18T15:09:51.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:51.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj36 td/osd-recovery-space/datafile 2024-09-18T15:09:51.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:51.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj37 td/osd-recovery-space/datafile 2024-09-18T15:09:51.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:51.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj38 td/osd-recovery-space/datafile 2024-09-18T15:09:52.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:52.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj39 td/osd-recovery-space/datafile 2024-09-18T15:09:52.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:52.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj40 td/osd-recovery-space/datafile 2024-09-18T15:09:52.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:52.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj41 td/osd-recovery-space/datafile 2024-09-18T15:09:52.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:52.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj42 td/osd-recovery-space/datafile 2024-09-18T15:09:52.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:52.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj43 td/osd-recovery-space/datafile 2024-09-18T15:09:52.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:52.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj44 td/osd-recovery-space/datafile 2024-09-18T15:09:52.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:52.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj45 td/osd-recovery-space/datafile 2024-09-18T15:09:52.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:52.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj46 td/osd-recovery-space/datafile 2024-09-18T15:09:52.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:52.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj47 td/osd-recovery-space/datafile 2024-09-18T15:09:52.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:52.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj48 td/osd-recovery-space/datafile 2024-09-18T15:09:52.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:52.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj49 td/osd-recovery-space/datafile 2024-09-18T15:09:52.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:52.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj50 td/osd-recovery-space/datafile 2024-09-18T15:09:53.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:53.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj51 td/osd-recovery-space/datafile 2024-09-18T15:09:53.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:53.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj52 td/osd-recovery-space/datafile 2024-09-18T15:09:53.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:53.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj53 td/osd-recovery-space/datafile 2024-09-18T15:09:53.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:53.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj54 td/osd-recovery-space/datafile 2024-09-18T15:09:53.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:53.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj55 td/osd-recovery-space/datafile 2024-09-18T15:09:53.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:53.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj56 td/osd-recovery-space/datafile 2024-09-18T15:09:53.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:53.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj57 td/osd-recovery-space/datafile 2024-09-18T15:09:53.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:53.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj58 td/osd-recovery-space/datafile 2024-09-18T15:09:53.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:53.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj59 td/osd-recovery-space/datafile 2024-09-18T15:09:53.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:53.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj60 td/osd-recovery-space/datafile 2024-09-18T15:09:53.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:53.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj61 td/osd-recovery-space/datafile 2024-09-18T15:09:53.826 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:53.826 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj62 td/osd-recovery-space/datafile 2024-09-18T15:09:53.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:53.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj63 td/osd-recovery-space/datafile 2024-09-18T15:09:53.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:53.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj64 td/osd-recovery-space/datafile 2024-09-18T15:09:54.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:54.052 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj65 td/osd-recovery-space/datafile 2024-09-18T15:09:54.135 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:54.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj66 td/osd-recovery-space/datafile 2024-09-18T15:09:54.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:54.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj67 td/osd-recovery-space/datafile 2024-09-18T15:09:54.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:54.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj68 td/osd-recovery-space/datafile 2024-09-18T15:09:54.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:54.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj69 td/osd-recovery-space/datafile 2024-09-18T15:09:54.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:54.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj70 td/osd-recovery-space/datafile 2024-09-18T15:09:54.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:54.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj71 td/osd-recovery-space/datafile 2024-09-18T15:09:54.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:54.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj72 td/osd-recovery-space/datafile 2024-09-18T15:09:54.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:54.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj73 td/osd-recovery-space/datafile 2024-09-18T15:09:54.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:54.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj74 td/osd-recovery-space/datafile 2024-09-18T15:09:54.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:54.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj75 td/osd-recovery-space/datafile 2024-09-18T15:09:55.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:55.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj76 td/osd-recovery-space/datafile 2024-09-18T15:09:55.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:55.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj77 td/osd-recovery-space/datafile 2024-09-18T15:09:55.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:55.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj78 td/osd-recovery-space/datafile 2024-09-18T15:09:55.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:55.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj79 td/osd-recovery-space/datafile 2024-09-18T15:09:55.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:55.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj80 td/osd-recovery-space/datafile 2024-09-18T15:09:55.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:55.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj81 td/osd-recovery-space/datafile 2024-09-18T15:09:55.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:55.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj82 td/osd-recovery-space/datafile 2024-09-18T15:09:55.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:55.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj83 td/osd-recovery-space/datafile 2024-09-18T15:09:55.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:55.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj84 td/osd-recovery-space/datafile 2024-09-18T15:09:55.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:55.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj85 td/osd-recovery-space/datafile 2024-09-18T15:09:55.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:55.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj86 td/osd-recovery-space/datafile 2024-09-18T15:09:55.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:55.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj87 td/osd-recovery-space/datafile 2024-09-18T15:09:55.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:55.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj88 td/osd-recovery-space/datafile 2024-09-18T15:09:56.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:56.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj89 td/osd-recovery-space/datafile 2024-09-18T15:09:56.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:56.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj90 td/osd-recovery-space/datafile 2024-09-18T15:09:56.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:56.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj91 td/osd-recovery-space/datafile 2024-09-18T15:09:56.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:56.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj92 td/osd-recovery-space/datafile 2024-09-18T15:09:56.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:56.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj93 td/osd-recovery-space/datafile 2024-09-18T15:09:56.470 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:56.470 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj94 td/osd-recovery-space/datafile 2024-09-18T15:09:56.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:56.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj95 td/osd-recovery-space/datafile 2024-09-18T15:09:56.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:56.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj96 td/osd-recovery-space/datafile 2024-09-18T15:09:56.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:56.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj97 td/osd-recovery-space/datafile 2024-09-18T15:09:56.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:56.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj98 td/osd-recovery-space/datafile 2024-09-18T15:09:56.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:56.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj99 td/osd-recovery-space/datafile 2024-09-18T15:09:56.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:56.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj100 td/osd-recovery-space/datafile 2024-09-18T15:09:57.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:57.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj101 td/osd-recovery-space/datafile 2024-09-18T15:09:57.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:57.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj102 td/osd-recovery-space/datafile 2024-09-18T15:09:57.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:57.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj103 td/osd-recovery-space/datafile 2024-09-18T15:09:57.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:57.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj104 td/osd-recovery-space/datafile 2024-09-18T15:09:57.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:57.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj105 td/osd-recovery-space/datafile 2024-09-18T15:09:57.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:57.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj106 td/osd-recovery-space/datafile 2024-09-18T15:09:57.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:57.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj107 td/osd-recovery-space/datafile 2024-09-18T15:09:57.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:57.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj108 td/osd-recovery-space/datafile 2024-09-18T15:09:57.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:57.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj109 td/osd-recovery-space/datafile 2024-09-18T15:09:57.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:57.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj110 td/osd-recovery-space/datafile 2024-09-18T15:09:57.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:57.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj111 td/osd-recovery-space/datafile 2024-09-18T15:09:57.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:57.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj112 td/osd-recovery-space/datafile 2024-09-18T15:09:57.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:57.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj113 td/osd-recovery-space/datafile 2024-09-18T15:09:58.052 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:58.052 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj114 td/osd-recovery-space/datafile 2024-09-18T15:09:58.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:58.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj115 td/osd-recovery-space/datafile 2024-09-18T15:09:58.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:58.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj116 td/osd-recovery-space/datafile 2024-09-18T15:09:58.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:58.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj117 td/osd-recovery-space/datafile 2024-09-18T15:09:58.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:58.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj118 td/osd-recovery-space/datafile 2024-09-18T15:09:58.470 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:58.470 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj119 td/osd-recovery-space/datafile 2024-09-18T15:09:58.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:58.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj120 td/osd-recovery-space/datafile 2024-09-18T15:09:58.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:58.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj121 td/osd-recovery-space/datafile 2024-09-18T15:09:58.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:58.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj122 td/osd-recovery-space/datafile 2024-09-18T15:09:58.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:58.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj123 td/osd-recovery-space/datafile 2024-09-18T15:09:58.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:58.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj124 td/osd-recovery-space/datafile 2024-09-18T15:09:58.989 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:58.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj125 td/osd-recovery-space/datafile 2024-09-18T15:09:59.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:59.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj126 td/osd-recovery-space/datafile 2024-09-18T15:09:59.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:59.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj127 td/osd-recovery-space/datafile 2024-09-18T15:09:59.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:59.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj128 td/osd-recovery-space/datafile 2024-09-18T15:09:59.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:59.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj129 td/osd-recovery-space/datafile 2024-09-18T15:09:59.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:59.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj130 td/osd-recovery-space/datafile 2024-09-18T15:09:59.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:59.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj131 td/osd-recovery-space/datafile 2024-09-18T15:09:59.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:59.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj132 td/osd-recovery-space/datafile 2024-09-18T15:09:59.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:59.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj133 td/osd-recovery-space/datafile 2024-09-18T15:09:59.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:59.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj134 td/osd-recovery-space/datafile 2024-09-18T15:09:59.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:59.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj135 td/osd-recovery-space/datafile 2024-09-18T15:09:59.826 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:59.826 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj136 td/osd-recovery-space/datafile 2024-09-18T15:09:59.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:59.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj137 td/osd-recovery-space/datafile 2024-09-18T15:09:59.985 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:09:59.985 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj138 td/osd-recovery-space/datafile 2024-09-18T15:10:00.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:00.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj139 td/osd-recovery-space/datafile 2024-09-18T15:10:00.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:00.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj140 td/osd-recovery-space/datafile 2024-09-18T15:10:00.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:00.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj141 td/osd-recovery-space/datafile 2024-09-18T15:10:00.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:00.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj142 td/osd-recovery-space/datafile 2024-09-18T15:10:00.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:00.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj143 td/osd-recovery-space/datafile 2024-09-18T15:10:00.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:00.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj144 td/osd-recovery-space/datafile 2024-09-18T15:10:00.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:00.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj145 td/osd-recovery-space/datafile 2024-09-18T15:10:00.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:00.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj146 td/osd-recovery-space/datafile 2024-09-18T15:10:00.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:00.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj147 td/osd-recovery-space/datafile 2024-09-18T15:10:00.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:00.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj148 td/osd-recovery-space/datafile 2024-09-18T15:10:00.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:00.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj149 td/osd-recovery-space/datafile 2024-09-18T15:10:01.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:01.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj150 td/osd-recovery-space/datafile 2024-09-18T15:10:01.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:01.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj151 td/osd-recovery-space/datafile 2024-09-18T15:10:01.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:01.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj152 td/osd-recovery-space/datafile 2024-09-18T15:10:01.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:01.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj153 td/osd-recovery-space/datafile 2024-09-18T15:10:01.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:01.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj154 td/osd-recovery-space/datafile 2024-09-18T15:10:01.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:01.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj155 td/osd-recovery-space/datafile 2024-09-18T15:10:01.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:01.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj156 td/osd-recovery-space/datafile 2024-09-18T15:10:01.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:01.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj157 td/osd-recovery-space/datafile 2024-09-18T15:10:01.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:01.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj158 td/osd-recovery-space/datafile 2024-09-18T15:10:01.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:01.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj159 td/osd-recovery-space/datafile 2024-09-18T15:10:01.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:01.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj160 td/osd-recovery-space/datafile 2024-09-18T15:10:01.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:01.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj161 td/osd-recovery-space/datafile 2024-09-18T15:10:01.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:01.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj162 td/osd-recovery-space/datafile 2024-09-18T15:10:02.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:02.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj163 td/osd-recovery-space/datafile 2024-09-18T15:10:02.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:02.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj164 td/osd-recovery-space/datafile 2024-09-18T15:10:02.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:02.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj165 td/osd-recovery-space/datafile 2024-09-18T15:10:02.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:02.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj166 td/osd-recovery-space/datafile 2024-09-18T15:10:02.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:02.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj167 td/osd-recovery-space/datafile 2024-09-18T15:10:02.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:02.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj168 td/osd-recovery-space/datafile 2024-09-18T15:10:02.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:02.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj169 td/osd-recovery-space/datafile 2024-09-18T15:10:02.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:02.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj170 td/osd-recovery-space/datafile 2024-09-18T15:10:02.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:02.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj171 td/osd-recovery-space/datafile 2024-09-18T15:10:02.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:02.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj172 td/osd-recovery-space/datafile 2024-09-18T15:10:02.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:02.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj173 td/osd-recovery-space/datafile 2024-09-18T15:10:02.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:02.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj174 td/osd-recovery-space/datafile 2024-09-18T15:10:02.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:02.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj175 td/osd-recovery-space/datafile 2024-09-18T15:10:03.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:03.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj176 td/osd-recovery-space/datafile 2024-09-18T15:10:03.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:03.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj177 td/osd-recovery-space/datafile 2024-09-18T15:10:03.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:03.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj178 td/osd-recovery-space/datafile 2024-09-18T15:10:03.299 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:03.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj179 td/osd-recovery-space/datafile 2024-09-18T15:10:03.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:03.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj180 td/osd-recovery-space/datafile 2024-09-18T15:10:03.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:03.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj181 td/osd-recovery-space/datafile 2024-09-18T15:10:03.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:03.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj182 td/osd-recovery-space/datafile 2024-09-18T15:10:03.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:03.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj183 td/osd-recovery-space/datafile 2024-09-18T15:10:03.676 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:03.676 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj184 td/osd-recovery-space/datafile 2024-09-18T15:10:03.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:03.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj185 td/osd-recovery-space/datafile 2024-09-18T15:10:03.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:03.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj186 td/osd-recovery-space/datafile 2024-09-18T15:10:03.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:03.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj187 td/osd-recovery-space/datafile 2024-09-18T15:10:03.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:03.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj188 td/osd-recovery-space/datafile 2024-09-18T15:10:04.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:04.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj189 td/osd-recovery-space/datafile 2024-09-18T15:10:04.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:04.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj190 td/osd-recovery-space/datafile 2024-09-18T15:10:04.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:04.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj191 td/osd-recovery-space/datafile 2024-09-18T15:10:04.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:04.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj192 td/osd-recovery-space/datafile 2024-09-18T15:10:04.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:04.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj193 td/osd-recovery-space/datafile 2024-09-18T15:10:04.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:04.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj194 td/osd-recovery-space/datafile 2024-09-18T15:10:04.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:04.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj195 td/osd-recovery-space/datafile 2024-09-18T15:10:04.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:04.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj196 td/osd-recovery-space/datafile 2024-09-18T15:10:04.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:04.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj197 td/osd-recovery-space/datafile 2024-09-18T15:10:04.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:04.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj198 td/osd-recovery-space/datafile 2024-09-18T15:10:04.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:04.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj199 td/osd-recovery-space/datafile 2024-09-18T15:10:04.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:04.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj200 td/osd-recovery-space/datafile 2024-09-18T15:10:05.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:05.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj201 td/osd-recovery-space/datafile 2024-09-18T15:10:05.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:05.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj202 td/osd-recovery-space/datafile 2024-09-18T15:10:05.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:05.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj203 td/osd-recovery-space/datafile 2024-09-18T15:10:05.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:05.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj204 td/osd-recovery-space/datafile 2024-09-18T15:10:05.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:05.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj205 td/osd-recovery-space/datafile 2024-09-18T15:10:05.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:05.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj206 td/osd-recovery-space/datafile 2024-09-18T15:10:05.492 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:05.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj207 td/osd-recovery-space/datafile 2024-09-18T15:10:05.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:05.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj208 td/osd-recovery-space/datafile 2024-09-18T15:10:05.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:05.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj209 td/osd-recovery-space/datafile 2024-09-18T15:10:05.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:05.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj210 td/osd-recovery-space/datafile 2024-09-18T15:10:06.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:06.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj211 td/osd-recovery-space/datafile 2024-09-18T15:10:06.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:06.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj212 td/osd-recovery-space/datafile 2024-09-18T15:10:06.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:06.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj213 td/osd-recovery-space/datafile 2024-09-18T15:10:06.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:06.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj214 td/osd-recovery-space/datafile 2024-09-18T15:10:06.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:06.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj215 td/osd-recovery-space/datafile 2024-09-18T15:10:06.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:06.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj216 td/osd-recovery-space/datafile 2024-09-18T15:10:06.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:06.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj217 td/osd-recovery-space/datafile 2024-09-18T15:10:06.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:06.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj218 td/osd-recovery-space/datafile 2024-09-18T15:10:06.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:06.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj219 td/osd-recovery-space/datafile 2024-09-18T15:10:06.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:06.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj220 td/osd-recovery-space/datafile 2024-09-18T15:10:06.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:06.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj221 td/osd-recovery-space/datafile 2024-09-18T15:10:06.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:06.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj222 td/osd-recovery-space/datafile 2024-09-18T15:10:07.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:07.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj223 td/osd-recovery-space/datafile 2024-09-18T15:10:07.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:07.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj224 td/osd-recovery-space/datafile 2024-09-18T15:10:07.157 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:07.157 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj225 td/osd-recovery-space/datafile 2024-09-18T15:10:07.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:07.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj226 td/osd-recovery-space/datafile 2024-09-18T15:10:07.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:07.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj227 td/osd-recovery-space/datafile 2024-09-18T15:10:07.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:07.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj228 td/osd-recovery-space/datafile 2024-09-18T15:10:07.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:07.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj229 td/osd-recovery-space/datafile 2024-09-18T15:10:07.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:07.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj230 td/osd-recovery-space/datafile 2024-09-18T15:10:07.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:07.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj231 td/osd-recovery-space/datafile 2024-09-18T15:10:07.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:07.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj232 td/osd-recovery-space/datafile 2024-09-18T15:10:07.760 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:07.760 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj233 td/osd-recovery-space/datafile 2024-09-18T15:10:07.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:07.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj234 td/osd-recovery-space/datafile 2024-09-18T15:10:07.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:07.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj235 td/osd-recovery-space/datafile 2024-09-18T15:10:07.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:07.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj236 td/osd-recovery-space/datafile 2024-09-18T15:10:08.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:08.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj237 td/osd-recovery-space/datafile 2024-09-18T15:10:08.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:08.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj238 td/osd-recovery-space/datafile 2024-09-18T15:10:08.212 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:08.212 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj239 td/osd-recovery-space/datafile 2024-09-18T15:10:08.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:08.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj240 td/osd-recovery-space/datafile 2024-09-18T15:10:08.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:08.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj241 td/osd-recovery-space/datafile 2024-09-18T15:10:08.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:08.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj242 td/osd-recovery-space/datafile 2024-09-18T15:10:08.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:08.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj243 td/osd-recovery-space/datafile 2024-09-18T15:10:08.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:08.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj244 td/osd-recovery-space/datafile 2024-09-18T15:10:08.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:08.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj245 td/osd-recovery-space/datafile 2024-09-18T15:10:08.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:08.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj246 td/osd-recovery-space/datafile 2024-09-18T15:10:08.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:08.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj247 td/osd-recovery-space/datafile 2024-09-18T15:10:09.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:09.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj248 td/osd-recovery-space/datafile 2024-09-18T15:10:09.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:09.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj249 td/osd-recovery-space/datafile 2024-09-18T15:10:09.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:09.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj250 td/osd-recovery-space/datafile 2024-09-18T15:10:09.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:09.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj251 td/osd-recovery-space/datafile 2024-09-18T15:10:09.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:09.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj252 td/osd-recovery-space/datafile 2024-09-18T15:10:09.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:09.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj253 td/osd-recovery-space/datafile 2024-09-18T15:10:09.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:09.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj254 td/osd-recovery-space/datafile 2024-09-18T15:10:09.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:09.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj255 td/osd-recovery-space/datafile 2024-09-18T15:10:09.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:09.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj256 td/osd-recovery-space/datafile 2024-09-18T15:10:09.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:09.760 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj257 td/osd-recovery-space/datafile 2024-09-18T15:10:09.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:09.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj258 td/osd-recovery-space/datafile 2024-09-18T15:10:09.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:09.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj259 td/osd-recovery-space/datafile 2024-09-18T15:10:10.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:10.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj260 td/osd-recovery-space/datafile 2024-09-18T15:10:10.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:10.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj261 td/osd-recovery-space/datafile 2024-09-18T15:10:10.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:10.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj262 td/osd-recovery-space/datafile 2024-09-18T15:10:10.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:10.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj263 td/osd-recovery-space/datafile 2024-09-18T15:10:10.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:10.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj264 td/osd-recovery-space/datafile 2024-09-18T15:10:10.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:10.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj265 td/osd-recovery-space/datafile 2024-09-18T15:10:10.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:10.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj266 td/osd-recovery-space/datafile 2024-09-18T15:10:10.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:10.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj267 td/osd-recovery-space/datafile 2024-09-18T15:10:10.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:10.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj268 td/osd-recovery-space/datafile 2024-09-18T15:10:10.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:10.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj269 td/osd-recovery-space/datafile 2024-09-18T15:10:10.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:10.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj270 td/osd-recovery-space/datafile 2024-09-18T15:10:10.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:10.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj271 td/osd-recovery-space/datafile 2024-09-18T15:10:10.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:10.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj272 td/osd-recovery-space/datafile 2024-09-18T15:10:11.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:11.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj273 td/osd-recovery-space/datafile 2024-09-18T15:10:11.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:11.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj274 td/osd-recovery-space/datafile 2024-09-18T15:10:11.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:11.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj275 td/osd-recovery-space/datafile 2024-09-18T15:10:11.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:11.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj276 td/osd-recovery-space/datafile 2024-09-18T15:10:11.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:11.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj277 td/osd-recovery-space/datafile 2024-09-18T15:10:11.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:11.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj278 td/osd-recovery-space/datafile 2024-09-18T15:10:11.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:11.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj279 td/osd-recovery-space/datafile 2024-09-18T15:10:11.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:11.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj280 td/osd-recovery-space/datafile 2024-09-18T15:10:11.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:11.760 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj281 td/osd-recovery-space/datafile 2024-09-18T15:10:11.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:11.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj282 td/osd-recovery-space/datafile 2024-09-18T15:10:11.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:11.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj283 td/osd-recovery-space/datafile 2024-09-18T15:10:11.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:11.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj284 td/osd-recovery-space/datafile 2024-09-18T15:10:12.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:12.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj285 td/osd-recovery-space/datafile 2024-09-18T15:10:12.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:12.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj286 td/osd-recovery-space/datafile 2024-09-18T15:10:12.212 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:12.212 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj287 td/osd-recovery-space/datafile 2024-09-18T15:10:12.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:12.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj288 td/osd-recovery-space/datafile 2024-09-18T15:10:12.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:12.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj289 td/osd-recovery-space/datafile 2024-09-18T15:10:12.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:12.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj290 td/osd-recovery-space/datafile 2024-09-18T15:10:12.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:12.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj291 td/osd-recovery-space/datafile 2024-09-18T15:10:12.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:12.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj292 td/osd-recovery-space/datafile 2024-09-18T15:10:12.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:12.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj293 td/osd-recovery-space/datafile 2024-09-18T15:10:12.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:12.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj294 td/osd-recovery-space/datafile 2024-09-18T15:10:12.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:12.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj295 td/osd-recovery-space/datafile 2024-09-18T15:10:12.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:12.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj296 td/osd-recovery-space/datafile 2024-09-18T15:10:13.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:13.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj297 td/osd-recovery-space/datafile 2024-09-18T15:10:13.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:13.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj298 td/osd-recovery-space/datafile 2024-09-18T15:10:13.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:13.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj299 td/osd-recovery-space/datafile 2024-09-18T15:10:13.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:13.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj300 td/osd-recovery-space/datafile 2024-09-18T15:10:13.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:13.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj301 td/osd-recovery-space/datafile 2024-09-18T15:10:13.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:13.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj302 td/osd-recovery-space/datafile 2024-09-18T15:10:13.492 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:13.492 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj303 td/osd-recovery-space/datafile 2024-09-18T15:10:13.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:13.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj304 td/osd-recovery-space/datafile 2024-09-18T15:10:13.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:13.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj305 td/osd-recovery-space/datafile 2024-09-18T15:10:13.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:13.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj306 td/osd-recovery-space/datafile 2024-09-18T15:10:13.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:13.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj307 td/osd-recovery-space/datafile 2024-09-18T15:10:13.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:13.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj308 td/osd-recovery-space/datafile 2024-09-18T15:10:13.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:13.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj309 td/osd-recovery-space/datafile 2024-09-18T15:10:14.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:14.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj310 td/osd-recovery-space/datafile 2024-09-18T15:10:14.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:14.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj311 td/osd-recovery-space/datafile 2024-09-18T15:10:14.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:14.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj312 td/osd-recovery-space/datafile 2024-09-18T15:10:14.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:14.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj313 td/osd-recovery-space/datafile 2024-09-18T15:10:14.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:14.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj314 td/osd-recovery-space/datafile 2024-09-18T15:10:14.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:14.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj315 td/osd-recovery-space/datafile 2024-09-18T15:10:14.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:14.514 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj316 td/osd-recovery-space/datafile 2024-09-18T15:10:14.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:14.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj317 td/osd-recovery-space/datafile 2024-09-18T15:10:14.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:14.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj318 td/osd-recovery-space/datafile 2024-09-18T15:10:14.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:14.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj319 td/osd-recovery-space/datafile 2024-09-18T15:10:14.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:14.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj320 td/osd-recovery-space/datafile 2024-09-18T15:10:14.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:14.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj321 td/osd-recovery-space/datafile 2024-09-18T15:10:14.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:14.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj322 td/osd-recovery-space/datafile 2024-09-18T15:10:15.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:15.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj323 td/osd-recovery-space/datafile 2024-09-18T15:10:15.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:15.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj324 td/osd-recovery-space/datafile 2024-09-18T15:10:15.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:15.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj325 td/osd-recovery-space/datafile 2024-09-18T15:10:15.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:15.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj326 td/osd-recovery-space/datafile 2024-09-18T15:10:15.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:15.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj327 td/osd-recovery-space/datafile 2024-09-18T15:10:15.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:15.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj328 td/osd-recovery-space/datafile 2024-09-18T15:10:15.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:15.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj329 td/osd-recovery-space/datafile 2024-09-18T15:10:15.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:15.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj330 td/osd-recovery-space/datafile 2024-09-18T15:10:15.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:15.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj331 td/osd-recovery-space/datafile 2024-09-18T15:10:15.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:15.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj332 td/osd-recovery-space/datafile 2024-09-18T15:10:15.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:15.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj333 td/osd-recovery-space/datafile 2024-09-18T15:10:15.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:15.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj334 td/osd-recovery-space/datafile 2024-09-18T15:10:15.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:15.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj335 td/osd-recovery-space/datafile 2024-09-18T15:10:16.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:16.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj336 td/osd-recovery-space/datafile 2024-09-18T15:10:16.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:16.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj337 td/osd-recovery-space/datafile 2024-09-18T15:10:16.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:16.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj338 td/osd-recovery-space/datafile 2024-09-18T15:10:16.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:16.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj339 td/osd-recovery-space/datafile 2024-09-18T15:10:16.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:16.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj340 td/osd-recovery-space/datafile 2024-09-18T15:10:16.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:16.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj341 td/osd-recovery-space/datafile 2024-09-18T15:10:16.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:16.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj342 td/osd-recovery-space/datafile 2024-09-18T15:10:16.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:16.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj343 td/osd-recovery-space/datafile 2024-09-18T15:10:16.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:16.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj344 td/osd-recovery-space/datafile 2024-09-18T15:10:16.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:16.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj345 td/osd-recovery-space/datafile 2024-09-18T15:10:16.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:16.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj346 td/osd-recovery-space/datafile 2024-09-18T15:10:16.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:16.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj347 td/osd-recovery-space/datafile 2024-09-18T15:10:17.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:17.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj348 td/osd-recovery-space/datafile 2024-09-18T15:10:17.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:17.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj349 td/osd-recovery-space/datafile 2024-09-18T15:10:17.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:17.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj350 td/osd-recovery-space/datafile 2024-09-18T15:10:17.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:17.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj351 td/osd-recovery-space/datafile 2024-09-18T15:10:17.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:17.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj352 td/osd-recovery-space/datafile 2024-09-18T15:10:17.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:17.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj353 td/osd-recovery-space/datafile 2024-09-18T15:10:17.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:17.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj354 td/osd-recovery-space/datafile 2024-09-18T15:10:17.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:17.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj355 td/osd-recovery-space/datafile 2024-09-18T15:10:17.660 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:17.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj356 td/osd-recovery-space/datafile 2024-09-18T15:10:17.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:17.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj357 td/osd-recovery-space/datafile 2024-09-18T15:10:17.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:17.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj358 td/osd-recovery-space/datafile 2024-09-18T15:10:17.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:17.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj359 td/osd-recovery-space/datafile 2024-09-18T15:10:17.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:17.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj360 td/osd-recovery-space/datafile 2024-09-18T15:10:18.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:18.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj361 td/osd-recovery-space/datafile 2024-09-18T15:10:18.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:18.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj362 td/osd-recovery-space/datafile 2024-09-18T15:10:18.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:18.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj363 td/osd-recovery-space/datafile 2024-09-18T15:10:18.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:18.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj364 td/osd-recovery-space/datafile 2024-09-18T15:10:18.389 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:18.389 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj365 td/osd-recovery-space/datafile 2024-09-18T15:10:18.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:18.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj366 td/osd-recovery-space/datafile 2024-09-18T15:10:18.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:18.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj367 td/osd-recovery-space/datafile 2024-09-18T15:10:18.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:18.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj368 td/osd-recovery-space/datafile 2024-09-18T15:10:18.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:18.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj369 td/osd-recovery-space/datafile 2024-09-18T15:10:18.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:18.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj370 td/osd-recovery-space/datafile 2024-09-18T15:10:18.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:18.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj371 td/osd-recovery-space/datafile 2024-09-18T15:10:19.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:19.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj372 td/osd-recovery-space/datafile 2024-09-18T15:10:19.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:19.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj373 td/osd-recovery-space/datafile 2024-09-18T15:10:19.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:19.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj374 td/osd-recovery-space/datafile 2024-09-18T15:10:19.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:19.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj375 td/osd-recovery-space/datafile 2024-09-18T15:10:19.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:19.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj376 td/osd-recovery-space/datafile 2024-09-18T15:10:19.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:19.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj377 td/osd-recovery-space/datafile 2024-09-18T15:10:19.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:19.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj378 td/osd-recovery-space/datafile 2024-09-18T15:10:19.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:19.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj379 td/osd-recovery-space/datafile 2024-09-18T15:10:19.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:19.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj380 td/osd-recovery-space/datafile 2024-09-18T15:10:19.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:19.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj381 td/osd-recovery-space/datafile 2024-09-18T15:10:19.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:19.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj382 td/osd-recovery-space/datafile 2024-09-18T15:10:19.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:19.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj383 td/osd-recovery-space/datafile 2024-09-18T15:10:19.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:19.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj384 td/osd-recovery-space/datafile 2024-09-18T15:10:19.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:19.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj385 td/osd-recovery-space/datafile 2024-09-18T15:10:20.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:20.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj386 td/osd-recovery-space/datafile 2024-09-18T15:10:20.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:20.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj387 td/osd-recovery-space/datafile 2024-09-18T15:10:20.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:20.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj388 td/osd-recovery-space/datafile 2024-09-18T15:10:20.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:20.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj389 td/osd-recovery-space/datafile 2024-09-18T15:10:20.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:20.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj390 td/osd-recovery-space/datafile 2024-09-18T15:10:20.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:20.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj391 td/osd-recovery-space/datafile 2024-09-18T15:10:20.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:20.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj392 td/osd-recovery-space/datafile 2024-09-18T15:10:20.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:20.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj393 td/osd-recovery-space/datafile 2024-09-18T15:10:20.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:20.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj394 td/osd-recovery-space/datafile 2024-09-18T15:10:20.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:20.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj395 td/osd-recovery-space/datafile 2024-09-18T15:10:20.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:20.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj396 td/osd-recovery-space/datafile 2024-09-18T15:10:20.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:20.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj397 td/osd-recovery-space/datafile 2024-09-18T15:10:21.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:21.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj398 td/osd-recovery-space/datafile 2024-09-18T15:10:21.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:21.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj399 td/osd-recovery-space/datafile 2024-09-18T15:10:21.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:21.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj400 td/osd-recovery-space/datafile 2024-09-18T15:10:21.285 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:21.285 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj401 td/osd-recovery-space/datafile 2024-09-18T15:10:21.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:21.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj402 td/osd-recovery-space/datafile 2024-09-18T15:10:21.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:21.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj403 td/osd-recovery-space/datafile 2024-09-18T15:10:21.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:21.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj404 td/osd-recovery-space/datafile 2024-09-18T15:10:21.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:21.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj405 td/osd-recovery-space/datafile 2024-09-18T15:10:21.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:21.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj406 td/osd-recovery-space/datafile 2024-09-18T15:10:21.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:21.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj407 td/osd-recovery-space/datafile 2024-09-18T15:10:21.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:21.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj408 td/osd-recovery-space/datafile 2024-09-18T15:10:21.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:21.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj409 td/osd-recovery-space/datafile 2024-09-18T15:10:21.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:21.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj410 td/osd-recovery-space/datafile 2024-09-18T15:10:22.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:22.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj411 td/osd-recovery-space/datafile 2024-09-18T15:10:22.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:22.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj412 td/osd-recovery-space/datafile 2024-09-18T15:10:22.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:22.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj413 td/osd-recovery-space/datafile 2024-09-18T15:10:22.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:22.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj414 td/osd-recovery-space/datafile 2024-09-18T15:10:22.389 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:22.390 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj415 td/osd-recovery-space/datafile 2024-09-18T15:10:22.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:22.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj416 td/osd-recovery-space/datafile 2024-09-18T15:10:22.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:22.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj417 td/osd-recovery-space/datafile 2024-09-18T15:10:22.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:22.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj418 td/osd-recovery-space/datafile 2024-09-18T15:10:22.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:22.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj419 td/osd-recovery-space/datafile 2024-09-18T15:10:22.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:22.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj420 td/osd-recovery-space/datafile 2024-09-18T15:10:22.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:22.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj421 td/osd-recovery-space/datafile 2024-09-18T15:10:22.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:22.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj422 td/osd-recovery-space/datafile 2024-09-18T15:10:23.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:23.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj423 td/osd-recovery-space/datafile 2024-09-18T15:10:23.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:23.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj424 td/osd-recovery-space/datafile 2024-09-18T15:10:23.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:23.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj425 td/osd-recovery-space/datafile 2024-09-18T15:10:23.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:23.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj426 td/osd-recovery-space/datafile 2024-09-18T15:10:23.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:23.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj427 td/osd-recovery-space/datafile 2024-09-18T15:10:23.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:23.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj428 td/osd-recovery-space/datafile 2024-09-18T15:10:23.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:23.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj429 td/osd-recovery-space/datafile 2024-09-18T15:10:23.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:23.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj430 td/osd-recovery-space/datafile 2024-09-18T15:10:23.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:23.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj431 td/osd-recovery-space/datafile 2024-09-18T15:10:23.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:23.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj432 td/osd-recovery-space/datafile 2024-09-18T15:10:23.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:23.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj433 td/osd-recovery-space/datafile 2024-09-18T15:10:23.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:23.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj434 td/osd-recovery-space/datafile 2024-09-18T15:10:23.988 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:23.988 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj435 td/osd-recovery-space/datafile 2024-09-18T15:10:24.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:24.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj436 td/osd-recovery-space/datafile 2024-09-18T15:10:24.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:24.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj437 td/osd-recovery-space/datafile 2024-09-18T15:10:24.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:24.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj438 td/osd-recovery-space/datafile 2024-09-18T15:10:24.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:24.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj439 td/osd-recovery-space/datafile 2024-09-18T15:10:24.406 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:24.406 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj440 td/osd-recovery-space/datafile 2024-09-18T15:10:24.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:24.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj441 td/osd-recovery-space/datafile 2024-09-18T15:10:24.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:24.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj442 td/osd-recovery-space/datafile 2024-09-18T15:10:24.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:24.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj443 td/osd-recovery-space/datafile 2024-09-18T15:10:24.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:24.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj444 td/osd-recovery-space/datafile 2024-09-18T15:10:24.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:24.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj445 td/osd-recovery-space/datafile 2024-09-18T15:10:24.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:24.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj446 td/osd-recovery-space/datafile 2024-09-18T15:10:25.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:25.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj447 td/osd-recovery-space/datafile 2024-09-18T15:10:25.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:25.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj448 td/osd-recovery-space/datafile 2024-09-18T15:10:25.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:25.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj449 td/osd-recovery-space/datafile 2024-09-18T15:10:25.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:25.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj450 td/osd-recovery-space/datafile 2024-09-18T15:10:25.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:25.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj451 td/osd-recovery-space/datafile 2024-09-18T15:10:25.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:25.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj452 td/osd-recovery-space/datafile 2024-09-18T15:10:25.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:25.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj453 td/osd-recovery-space/datafile 2024-09-18T15:10:25.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:25.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj454 td/osd-recovery-space/datafile 2024-09-18T15:10:25.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:25.612 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj455 td/osd-recovery-space/datafile 2024-09-18T15:10:25.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:25.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj456 td/osd-recovery-space/datafile 2024-09-18T15:10:25.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:25.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj457 td/osd-recovery-space/datafile 2024-09-18T15:10:25.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:25.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj458 td/osd-recovery-space/datafile 2024-09-18T15:10:25.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:25.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj459 td/osd-recovery-space/datafile 2024-09-18T15:10:25.988 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:25.988 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj460 td/osd-recovery-space/datafile 2024-09-18T15:10:26.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:26.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj461 td/osd-recovery-space/datafile 2024-09-18T15:10:26.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:26.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj462 td/osd-recovery-space/datafile 2024-09-18T15:10:26.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:26.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj463 td/osd-recovery-space/datafile 2024-09-18T15:10:26.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:26.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj464 td/osd-recovery-space/datafile 2024-09-18T15:10:26.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:26.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj465 td/osd-recovery-space/datafile 2024-09-18T15:10:26.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:26.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj466 td/osd-recovery-space/datafile 2024-09-18T15:10:26.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:26.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj467 td/osd-recovery-space/datafile 2024-09-18T15:10:26.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:26.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj468 td/osd-recovery-space/datafile 2024-09-18T15:10:26.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:26.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj469 td/osd-recovery-space/datafile 2024-09-18T15:10:26.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:26.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj470 td/osd-recovery-space/datafile 2024-09-18T15:10:26.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:26.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj471 td/osd-recovery-space/datafile 2024-09-18T15:10:27.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:27.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj472 td/osd-recovery-space/datafile 2024-09-18T15:10:27.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:27.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj473 td/osd-recovery-space/datafile 2024-09-18T15:10:27.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:27.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj474 td/osd-recovery-space/datafile 2024-09-18T15:10:27.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:27.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj475 td/osd-recovery-space/datafile 2024-09-18T15:10:27.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:27.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj476 td/osd-recovery-space/datafile 2024-09-18T15:10:27.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:27.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj477 td/osd-recovery-space/datafile 2024-09-18T15:10:27.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:27.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj478 td/osd-recovery-space/datafile 2024-09-18T15:10:27.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:27.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj479 td/osd-recovery-space/datafile 2024-09-18T15:10:27.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:27.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj480 td/osd-recovery-space/datafile 2024-09-18T15:10:27.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:27.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj481 td/osd-recovery-space/datafile 2024-09-18T15:10:27.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:27.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj482 td/osd-recovery-space/datafile 2024-09-18T15:10:27.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:27.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj483 td/osd-recovery-space/datafile 2024-09-18T15:10:27.988 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:27.988 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj484 td/osd-recovery-space/datafile 2024-09-18T15:10:28.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:28.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj485 td/osd-recovery-space/datafile 2024-09-18T15:10:28.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:28.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj486 td/osd-recovery-space/datafile 2024-09-18T15:10:28.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:28.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj487 td/osd-recovery-space/datafile 2024-09-18T15:10:28.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:28.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj488 td/osd-recovery-space/datafile 2024-09-18T15:10:28.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:28.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj489 td/osd-recovery-space/datafile 2024-09-18T15:10:28.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:28.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj490 td/osd-recovery-space/datafile 2024-09-18T15:10:28.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:28.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj491 td/osd-recovery-space/datafile 2024-09-18T15:10:28.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:28.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj492 td/osd-recovery-space/datafile 2024-09-18T15:10:28.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:28.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj493 td/osd-recovery-space/datafile 2024-09-18T15:10:28.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:28.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj494 td/osd-recovery-space/datafile 2024-09-18T15:10:28.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:28.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj495 td/osd-recovery-space/datafile 2024-09-18T15:10:29.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:29.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj496 td/osd-recovery-space/datafile 2024-09-18T15:10:29.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:29.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj497 td/osd-recovery-space/datafile 2024-09-18T15:10:29.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:29.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj498 td/osd-recovery-space/datafile 2024-09-18T15:10:29.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:29.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj499 td/osd-recovery-space/datafile 2024-09-18T15:10:29.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:29.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj500 td/osd-recovery-space/datafile 2024-09-18T15:10:29.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:29.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj501 td/osd-recovery-space/datafile 2024-09-18T15:10:29.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:29.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj502 td/osd-recovery-space/datafile 2024-09-18T15:10:29.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:29.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj503 td/osd-recovery-space/datafile 2024-09-18T15:10:29.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:29.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj504 td/osd-recovery-space/datafile 2024-09-18T15:10:29.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:29.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj505 td/osd-recovery-space/datafile 2024-09-18T15:10:29.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:29.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj506 td/osd-recovery-space/datafile 2024-09-18T15:10:29.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:29.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj507 td/osd-recovery-space/datafile 2024-09-18T15:10:29.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:29.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj508 td/osd-recovery-space/datafile 2024-09-18T15:10:29.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:29.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj509 td/osd-recovery-space/datafile 2024-09-18T15:10:30.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:30.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj510 td/osd-recovery-space/datafile 2024-09-18T15:10:30.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:30.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj511 td/osd-recovery-space/datafile 2024-09-18T15:10:30.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:30.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj512 td/osd-recovery-space/datafile 2024-09-18T15:10:30.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:30.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj513 td/osd-recovery-space/datafile 2024-09-18T15:10:30.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:30.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj514 td/osd-recovery-space/datafile 2024-09-18T15:10:30.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:30.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj515 td/osd-recovery-space/datafile 2024-09-18T15:10:30.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:30.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj516 td/osd-recovery-space/datafile 2024-09-18T15:10:30.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:30.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj517 td/osd-recovery-space/datafile 2024-09-18T15:10:30.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:30.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj518 td/osd-recovery-space/datafile 2024-09-18T15:10:30.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:30.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj519 td/osd-recovery-space/datafile 2024-09-18T15:10:30.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:30.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj520 td/osd-recovery-space/datafile 2024-09-18T15:10:30.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:30.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj521 td/osd-recovery-space/datafile 2024-09-18T15:10:31.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:31.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj522 td/osd-recovery-space/datafile 2024-09-18T15:10:31.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:31.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj523 td/osd-recovery-space/datafile 2024-09-18T15:10:31.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:31.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj524 td/osd-recovery-space/datafile 2024-09-18T15:10:31.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:31.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj525 td/osd-recovery-space/datafile 2024-09-18T15:10:31.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:31.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj526 td/osd-recovery-space/datafile 2024-09-18T15:10:31.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:31.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj527 td/osd-recovery-space/datafile 2024-09-18T15:10:31.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:31.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj528 td/osd-recovery-space/datafile 2024-09-18T15:10:31.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:31.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj529 td/osd-recovery-space/datafile 2024-09-18T15:10:31.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:31.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj530 td/osd-recovery-space/datafile 2024-09-18T15:10:31.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:31.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj531 td/osd-recovery-space/datafile 2024-09-18T15:10:31.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:31.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj532 td/osd-recovery-space/datafile 2024-09-18T15:10:31.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:31.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj533 td/osd-recovery-space/datafile 2024-09-18T15:10:31.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:31.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj534 td/osd-recovery-space/datafile 2024-09-18T15:10:32.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:32.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj535 td/osd-recovery-space/datafile 2024-09-18T15:10:32.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:32.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj536 td/osd-recovery-space/datafile 2024-09-18T15:10:32.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:32.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj537 td/osd-recovery-space/datafile 2024-09-18T15:10:32.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:32.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj538 td/osd-recovery-space/datafile 2024-09-18T15:10:32.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:32.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj539 td/osd-recovery-space/datafile 2024-09-18T15:10:32.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:32.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj540 td/osd-recovery-space/datafile 2024-09-18T15:10:32.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:32.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj541 td/osd-recovery-space/datafile 2024-09-18T15:10:32.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:32.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj542 td/osd-recovery-space/datafile 2024-09-18T15:10:32.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:32.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj543 td/osd-recovery-space/datafile 2024-09-18T15:10:32.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:32.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj544 td/osd-recovery-space/datafile 2024-09-18T15:10:32.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:32.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj545 td/osd-recovery-space/datafile 2024-09-18T15:10:32.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:32.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj546 td/osd-recovery-space/datafile 2024-09-18T15:10:33.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:33.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj547 td/osd-recovery-space/datafile 2024-09-18T15:10:33.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:33.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj548 td/osd-recovery-space/datafile 2024-09-18T15:10:33.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:33.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj549 td/osd-recovery-space/datafile 2024-09-18T15:10:33.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:33.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj550 td/osd-recovery-space/datafile 2024-09-18T15:10:33.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:33.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj551 td/osd-recovery-space/datafile 2024-09-18T15:10:33.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:33.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj552 td/osd-recovery-space/datafile 2024-09-18T15:10:33.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:33.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj553 td/osd-recovery-space/datafile 2024-09-18T15:10:33.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:33.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj554 td/osd-recovery-space/datafile 2024-09-18T15:10:33.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:33.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj555 td/osd-recovery-space/datafile 2024-09-18T15:10:33.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:33.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj556 td/osd-recovery-space/datafile 2024-09-18T15:10:33.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:33.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj557 td/osd-recovery-space/datafile 2024-09-18T15:10:33.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:33.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj558 td/osd-recovery-space/datafile 2024-09-18T15:10:33.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:33.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj559 td/osd-recovery-space/datafile 2024-09-18T15:10:34.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:34.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj560 td/osd-recovery-space/datafile 2024-09-18T15:10:34.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:34.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj561 td/osd-recovery-space/datafile 2024-09-18T15:10:34.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:34.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj562 td/osd-recovery-space/datafile 2024-09-18T15:10:34.265 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:34.265 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj563 td/osd-recovery-space/datafile 2024-09-18T15:10:34.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:34.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj564 td/osd-recovery-space/datafile 2024-09-18T15:10:34.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:34.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj565 td/osd-recovery-space/datafile 2024-09-18T15:10:34.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:34.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj566 td/osd-recovery-space/datafile 2024-09-18T15:10:34.599 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:34.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj567 td/osd-recovery-space/datafile 2024-09-18T15:10:34.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:34.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj568 td/osd-recovery-space/datafile 2024-09-18T15:10:34.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:34.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj569 td/osd-recovery-space/datafile 2024-09-18T15:10:34.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:34.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj570 td/osd-recovery-space/datafile 2024-09-18T15:10:34.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:34.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj571 td/osd-recovery-space/datafile 2024-09-18T15:10:35.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:35.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj572 td/osd-recovery-space/datafile 2024-09-18T15:10:35.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:35.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj573 td/osd-recovery-space/datafile 2024-09-18T15:10:35.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:35.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj574 td/osd-recovery-space/datafile 2024-09-18T15:10:35.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:35.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj575 td/osd-recovery-space/datafile 2024-09-18T15:10:35.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:35.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj576 td/osd-recovery-space/datafile 2024-09-18T15:10:35.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:35.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj577 td/osd-recovery-space/datafile 2024-09-18T15:10:35.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:35.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj578 td/osd-recovery-space/datafile 2024-09-18T15:10:35.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:35.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj579 td/osd-recovery-space/datafile 2024-09-18T15:10:35.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:35.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj580 td/osd-recovery-space/datafile 2024-09-18T15:10:35.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:35.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj581 td/osd-recovery-space/datafile 2024-09-18T15:10:35.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:35.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj582 td/osd-recovery-space/datafile 2024-09-18T15:10:35.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:35.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj583 td/osd-recovery-space/datafile 2024-09-18T15:10:35.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:35.956 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj584 td/osd-recovery-space/datafile 2024-09-18T15:10:36.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:36.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj585 td/osd-recovery-space/datafile 2024-09-18T15:10:36.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:36.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj586 td/osd-recovery-space/datafile 2024-09-18T15:10:36.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:36.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj587 td/osd-recovery-space/datafile 2024-09-18T15:10:36.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:36.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj588 td/osd-recovery-space/datafile 2024-09-18T15:10:36.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:36.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj589 td/osd-recovery-space/datafile 2024-09-18T15:10:36.432 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:36.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj590 td/osd-recovery-space/datafile 2024-09-18T15:10:36.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:36.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj591 td/osd-recovery-space/datafile 2024-09-18T15:10:36.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:36.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj592 td/osd-recovery-space/datafile 2024-09-18T15:10:36.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:36.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj593 td/osd-recovery-space/datafile 2024-09-18T15:10:36.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:36.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj594 td/osd-recovery-space/datafile 2024-09-18T15:10:36.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:36.860 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj595 td/osd-recovery-space/datafile 2024-09-18T15:10:36.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:36.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj596 td/osd-recovery-space/datafile 2024-09-18T15:10:37.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:37.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj597 td/osd-recovery-space/datafile 2024-09-18T15:10:37.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:37.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj598 td/osd-recovery-space/datafile 2024-09-18T15:10:37.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:37.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj599 td/osd-recovery-space/datafile 2024-09-18T15:10:37.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-18T15:10:37.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj600 td/osd-recovery-space/datafile 2024-09-18T15:10:37.730 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:124: TEST_recovery_test_simple: expr 2 - 1 2024-09-18T15:10:37.730 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:124: TEST_recovery_test_simple: seq 0 1 2024-09-18T15:10:37.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:124: TEST_recovery_test_simple: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T15:10:37.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:126: TEST_recovery_test_simple: ceph tell osd.0 injectargs '--fake_statfs_for_testing 3686400' 2024-09-18T15:10:37.835 INFO:tasks.workunit.client.0.smithi047.stdout:{} 2024-09-18T15:10:37.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:124: TEST_recovery_test_simple: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-18T15:10:37.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:126: TEST_recovery_test_simple: ceph tell osd.1 injectargs '--fake_statfs_for_testing 3686400' 2024-09-18T15:10:37.947 INFO:tasks.workunit.client.0.smithi047.stdout:{} 2024-09-18T15:10:37.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:128: TEST_recovery_test_simple: sleep 5 2024-09-18T15:10:42.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:130: TEST_recovery_test_simple: ceph pg dump pgs 2024-09-18T15:10:43.319 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T15:10:43.319 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 600 0 0 0 0 3072000 0 0 600 0 600 active+clean 2024-09-18T15:09:44.779214+0000 20'600 20:618 [1] 1 [1] 1 0'0 2024-09-18T15:09:42.497930+0000 0'0 2024-09-18T15:09:42.497930+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:04:56.414316+0000 0 0 2024-09-18T15:10:43.320 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T15:10:43.320 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T15:10:43.320 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T15:10:43.330 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:132: TEST_recovery_test_simple: seq 1 1 2024-09-18T15:10:43.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:132: TEST_recovery_test_simple: for p in $(seq 1 $pools) 2024-09-18T15:10:43.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:134: TEST_recovery_test_simple: ceph osd pool set test1 size 2 2024-09-18T15:10:43.993 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 1 size to 2 2024-09-18T15:10:44.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:138: TEST_recovery_test_simple: wait_for_recovery_toofull 120 2024-09-18T15:10:44.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:77: wait_for_recovery_toofull: local timeout=120 2024-09-18T15:10:44.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:78: wait_for_recovery_toofull: wait_for_state recovery_toofull 120 2024-09-18T15:10:44.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:52: wait_for_state: local state=recovery_toofull 2024-09-18T15:10:44.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:53: wait_for_state: local cur_in_state 2024-09-18T15:10:44.006 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:54: wait_for_state: get_timeout_delays 120 5 2024-09-18T15:10:44.006 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:10:44.006 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:10:44.006 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:10:44.006 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:10:44.006 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:10:44.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:54: wait_for_state: delays=('5' '10' '15' '15' '15' '15' '15' '15' '15') 2024-09-18T15:10:44.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:54: wait_for_state: local -a delays 2024-09-18T15:10:44.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:55: wait_for_state: local -i loop=0 2024-09-18T15:10:44.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:57: wait_for_state: flush_pg_stats 2024-09-18T15:10:44.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:10:44.072 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:10:44.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:10:44.625 INFO:tasks.workunit.client.0.smithi047.stderr:1' 2024-09-18T15:10:44.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:10:44.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:10:44.626 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:10:44.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836508 2024-09-18T15:10:44.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836508 2024-09-18T15:10:44.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836508' 2024-09-18T15:10:44.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:10:44.737 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:10:44.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=38654705687 2024-09-18T15:10:44.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 38654705687 2024-09-18T15:10:44.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836508 1-38654705687' 2024-09-18T15:10:44.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:10:44.858 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836508 2024-09-18T15:10:44.858 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:10:44.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:10:44.859 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836508 2024-09-18T15:10:44.860 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:10:44.860 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836508 2024-09-18T15:10:44.861 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-09-18T15:10:44.861 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836508 2024-09-18T15:10:44.861 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:10:45.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836506 -lt 21474836508 2024-09-18T15:10:45.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:10:46.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:10:46.229 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:10:46.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836509 -lt 21474836508 2024-09-18T15:10:46.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:10:46.638 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-38654705687 2024-09-18T15:10:46.638 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:10:46.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:10:46.639 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-38654705687 2024-09-18T15:10:46.639 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:10:46.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=38654705687 2024-09-18T15:10:46.641 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 38654705687 2024-09-18T15:10:46.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 38654705687' 2024-09-18T15:10:46.641 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:10:47.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 38654705688 -lt 38654705687 2024-09-18T15:10:47.039 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:58: wait_for_state: get_num_pgs 2024-09-18T15:10:47.039 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:10:47.039 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:10:47.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:58: wait_for_state: test 1 == 0 2024-09-18T15:10:47.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:62: wait_for_state: true 2024-09-18T15:10:47.539 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:63: wait_for_state: get_num_in_state recovery_toofull 2024-09-18T15:10:47.539 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:43: get_num_in_state: local state=recovery_toofull 2024-09-18T15:10:47.539 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:44: get_num_in_state: local expression 2024-09-18T15:10:47.539 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:45: get_num_in_state: expression+='select(contains("recovery_toofull"))' 2024-09-18T15:10:47.539 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:46: get_num_in_state: ceph --format json pg dump pgs 2024-09-18T15:10:47.539 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:47: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("recovery_toofull"))] | length' 2024-09-18T15:10:47.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:63: wait_for_state: cur_in_state=1 2024-09-18T15:10:47.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:64: wait_for_state: test 1 -gt 0 2024-09-18T15:10:47.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:64: wait_for_state: break 2024-09-18T15:10:47.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:72: wait_for_state: return 0 2024-09-18T15:10:47.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:79: wait_for_recovery_toofull: '[' -ne 0 ']' 2024-09-18T15:10:47.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh: line 79: [: -ne: unary operator expected 2024-09-18T15:10:47.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:139: TEST_recovery_test_simple: '[' 0 -ne 0 ']' 2024-09-18T15:10:47.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:144: TEST_recovery_test_simple: ERRORS=0 2024-09-18T15:10:47.933 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:145: TEST_recovery_test_simple: ceph pg dump pgs 2024-09-18T15:10:47.933 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:145: TEST_recovery_test_simple: grep +recovery_toofull 2024-09-18T15:10:47.933 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:145: TEST_recovery_test_simple: wc -l 2024-09-18T15:10:48.279 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T15:10:48.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:145: TEST_recovery_test_simple: '[' 1 '!=' 1 ']' 2024-09-18T15:10:48.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:151: TEST_recovery_test_simple: ceph pg dump pgs 2024-09-18T15:10:48.705 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T15:10:48.706 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 600 0 600 0 0 3072000 0 0 600 0 600 active+recovery_toofull+undersized+degraded+remapped 2024-09-18T15:10:44.789006+0000 20'600 25:631 [1,0] 1 [1] 1 0'0 2024-09-18T15:09:42.497930+0000 0'0 2024-09-18T15:09:42.497930+0000 0 0 periodic scrub scheduled @ 2024-09-19T15:18:28.656794+0000 0 0 2024-09-18T15:10:48.706 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T15:10:48.706 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T15:10:48.706 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T15:10:48.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:152: TEST_recovery_test_simple: ceph status 2024-09-18T15:10:49.198 INFO:tasks.workunit.client.0.smithi047.stdout: cluster: 2024-09-18T15:10:49.198 INFO:tasks.workunit.client.0.smithi047.stdout: id: 961d5ebb-7a59-4c27-b062-d9452168f211 2024-09-18T15:10:49.198 INFO:tasks.workunit.client.0.smithi047.stdout: health: HEALTH_ERR 2024-09-18T15:10:49.198 INFO:tasks.workunit.client.0.smithi047.stdout: 2 full osd(s) 2024-09-18T15:10:49.198 INFO:tasks.workunit.client.0.smithi047.stdout: Degraded data redundancy: 600/1200 objects degraded (50.000%), 1 pg degraded 2024-09-18T15:10:49.198 INFO:tasks.workunit.client.0.smithi047.stdout: Full OSDs blocking recovery: 1 pg recovery_toofull 2024-09-18T15:10:49.198 INFO:tasks.workunit.client.0.smithi047.stdout: 1 pool(s) do not have an application enabled 2024-09-18T15:10:49.199 INFO:tasks.workunit.client.0.smithi047.stdout: 1 pool(s) full 2024-09-18T15:10:49.199 INFO:tasks.workunit.client.0.smithi047.stdout: OSD count 2 < osd_pool_default_size 3 2024-09-18T15:10:49.199 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T15:10:49.199 INFO:tasks.workunit.client.0.smithi047.stdout: services: 2024-09-18T15:10:49.199 INFO:tasks.workunit.client.0.smithi047.stdout: mon: 1 daemons, quorum a (age 2m) 2024-09-18T15:10:49.199 INFO:tasks.workunit.client.0.smithi047.stdout: mgr: x(active, since 2m) 2024-09-18T15:10:49.199 INFO:tasks.workunit.client.0.smithi047.stdout: osd: 2 osds: 2 up (since 69s), 2 in (since 102s); 1 remapped pgs 2024-09-18T15:10:49.199 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T15:10:49.199 INFO:tasks.workunit.client.0.smithi047.stdout: data: 2024-09-18T15:10:49.199 INFO:tasks.workunit.client.0.smithi047.stdout: pools: 1 pools, 1 pgs 2024-09-18T15:10:49.200 INFO:tasks.workunit.client.0.smithi047.stdout: objects: 600 objects, 2.9 MiB 2024-09-18T15:10:49.200 INFO:tasks.workunit.client.0.smithi047.stdout: usage: 5.9 MiB used, 1.2 MiB / 7.0 MiB avail 2024-09-18T15:10:49.200 INFO:tasks.workunit.client.0.smithi047.stdout: pgs: 600/1200 objects degraded (50.000%) 2024-09-18T15:10:49.200 INFO:tasks.workunit.client.0.smithi047.stdout: 1 active+recovery_toofull+undersized+degraded+remapped 2024-09-18T15:10:49.200 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T15:10:49.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:153: TEST_recovery_test_simple: ceph status --format=json-pretty 2024-09-18T15:10:49.746 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:155: TEST_recovery_test_simple: jq .health.checks.PG_RECOVERY_FULL.severity td/osd-recovery-space/stat.json 2024-09-18T15:10:49.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:155: TEST_recovery_test_simple: eval 'SEV="HEALTH_ERR"' 2024-09-18T15:10:49.749 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:155: TEST_recovery_test_simple: SEV=HEALTH_ERR 2024-09-18T15:10:49.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:156: TEST_recovery_test_simple: '[' HEALTH_ERR '!=' HEALTH_ERR ']' 2024-09-18T15:10:49.749 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:160: TEST_recovery_test_simple: jq .health.checks.PG_RECOVERY_FULL.summary.message td/osd-recovery-space/stat.json 2024-09-18T15:10:49.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:160: TEST_recovery_test_simple: eval 'MSG="Full' OSDs blocking recovery: 1 pg 'recovery_toofull"' 2024-09-18T15:10:49.752 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:160: TEST_recovery_test_simple: MSG='Full OSDs blocking recovery: 1 pg recovery_toofull' 2024-09-18T15:10:49.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:161: TEST_recovery_test_simple: '[' 'Full OSDs blocking recovery: 1 pg recovery_toofull' '!=' 'Full OSDs blocking recovery: 1 pg recovery_toofull' ']' 2024-09-18T15:10:49.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:165: TEST_recovery_test_simple: rm -f td/osd-recovery-space/stat.json 2024-09-18T15:10:49.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:167: TEST_recovery_test_simple: '[' 0 '!=' 0 ']' 2024-09-18T15:10:49.753 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:172: TEST_recovery_test_simple: seq 1 1 2024-09-18T15:10:49.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:172: TEST_recovery_test_simple: for i in $(seq 1 $pools) 2024-09-18T15:10:49.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:174: TEST_recovery_test_simple: delete_pool test1 2024-09-18T15:10:49.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test1 2024-09-18T15:10:49.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2024-09-18T15:10:50.287 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test1' does not exist 2024-09-18T15:10:50.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:176: TEST_recovery_test_simple: kill_daemons td/osd-recovery-space 2024-09-18T15:10:50.298 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:10:50.298 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:10:50.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:10:50.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:10:50.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:10:55.612 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:10:55.612 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:37: run: teardown td/osd-recovery-space 2024-09-18T15:10:55.612 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-space 2024-09-18T15:10:55.612 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:10:55.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-space KILL 2024-09-18T15:10:55.613 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:10:55.613 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:10:55.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:10:55.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:10:55.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:10:55.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:10:55.618 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:10:55.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:10:55.619 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:10:55.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:10:55.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:10:55.621 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:10:55.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:10:55.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:10:55.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:10:55.623 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:10:55.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:10:55.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:10:55.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-space 2024-09-18T15:10:55.654 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:10:55.654 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:10:55.655 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.271403 2024-09-18T15:10:55.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.271403 2024-09-18T15:10:55.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:10:55.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:10:55.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T15:10:55.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/osd-recovery-space 0 2024-09-18T15:10:55.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-space 2024-09-18T15:10:55.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T15:10:55.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-space KILL 2024-09-18T15:10:55.657 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:10:55.657 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:10:55.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:10:55.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:10:55.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:10:55.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:10:55.658 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:10:55.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:10:55.659 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:10:55.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:10:55.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:10:55.661 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:10:55.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:10:55.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:10:55.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:10:55.663 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:10:55.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:10:55.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T15:10:55.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-space 2024-09-18T15:10:55.666 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:10:55.666 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:10:55.666 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.271403 2024-09-18T15:10:55.666 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.271403 2024-09-18T15:10:55.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:10:55.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:10:55.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T15:10:55.669 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T15:10:55.669 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T15:10:55.697 INFO:tasks.workunit:Running workunit osd/osd-recovery-stats.sh... 2024-09-18T15:10:55.697 DEBUG:teuthology.orchestra.run.smithi047:workunit test osd/osd-recovery-stats.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh 2024-09-18T15:10:55.761 INFO:tasks.workunit.client.0.smithi047.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T15:10:55.767 INFO:tasks.workunit.client.0.smithi047.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T15:10:55.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T15:10:55.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T15:10:55.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T15:10:55.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T15:10:55.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T15:10:55.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T15:10:55.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T15:10:55.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T15:10:55.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/osd-recovery-stats 2024-09-18T15:10:55.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:21: run: local dir=td/osd-recovery-stats 2024-09-18T15:10:55.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:22: run: shift 2024-09-18T15:10:55.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:25: run: export CEPH_MON=127.0.0.1:7115 2024-09-18T15:10:55.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:25: run: CEPH_MON=127.0.0.1:7115 2024-09-18T15:10:55.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:26: run: export CEPH_ARGS 2024-09-18T15:10:55.768 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:27: run: uuidgen 2024-09-18T15:10:55.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:27: run: CEPH_ARGS+='--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none ' 2024-09-18T15:10:55.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2024-09-18T15:10:55.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:30: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:10:55.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:31: run: export margin=10 2024-09-18T15:10:55.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:31: run: margin=10 2024-09-18T15:10:55.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:32: run: export objects=200 2024-09-18T15:10:55.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:32: run: objects=200 2024-09-18T15:10:55.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:33: run: export poolname=test 2024-09-18T15:10:55.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:33: run: poolname=test 2024-09-18T15:10:55.770 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:35: run: set 2024-09-18T15:10:55.771 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:35: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-18T15:10:55.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:35: run: local 'funcs=TEST_recovery_erasure_out1 2024-09-18T15:10:55.772 INFO:tasks.workunit.client.0.smithi047.stderr:TEST_recovery_erasure_remapped 2024-09-18T15:10:55.772 INFO:tasks.workunit.client.0.smithi047.stderr:TEST_recovery_multi 2024-09-18T15:10:55.772 INFO:tasks.workunit.client.0.smithi047.stderr:TEST_recovery_replicated_out1 2024-09-18T15:10:55.772 INFO:tasks.workunit.client.0.smithi047.stderr:TEST_recovery_sizedown 2024-09-18T15:10:55.772 INFO:tasks.workunit.client.0.smithi047.stderr:TEST_recovery_sizeup 2024-09-18T15:10:55.773 INFO:tasks.workunit.client.0.smithi047.stderr:TEST_recovery_undersized' 2024-09-18T15:10:55.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-09-18T15:10:55.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-09-18T15:10:55.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-09-18T15:10:55.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-09-18T15:10:55.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-18T15:10:55.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:10:55.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-18T15:10:55.773 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:10:55.773 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:10:55.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:10:55.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:10:55.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:10:55.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:10:55.774 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:10:55.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:10:55.774 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:10:55.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:10:55.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:10:55.775 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:10:55.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:10:55.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:10:55.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:10:55.776 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:10:55.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:10:55.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:10:55.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-18T15:10:55.779 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:10:55.779 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:10:55.779 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:10:55.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.288083 2024-09-18T15:10:55.780 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:10:55.780 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:10:55.780 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-09-18T15:10:55.781 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T15:10:55.781 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:10:55.782 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:10:55.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.288083 2024-09-18T15:10:55.784 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T15:10:55.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T15:10:55.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T15:10:55.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T15:10:55.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-09-18T15:10:55.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_erasure_out1 td/osd-recovery-stats 2024-09-18T15:10:55.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:173: TEST_recovery_erasure_out1: local dir=td/osd-recovery-stats 2024-09-18T15:10:55.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:175: TEST_recovery_erasure_out1: do_recovery_out1 td/osd-recovery-stats erasure 2024-09-18T15:10:55.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:117: do_recovery_out1: local dir=td/osd-recovery-stats 2024-09-18T15:10:55.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:118: do_recovery_out1: shift 2024-09-18T15:10:55.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:119: do_recovery_out1: local type=erasure 2024-09-18T15:10:55.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:121: do_recovery_out1: run_mon td/osd-recovery-stats a 2024-09-18T15:10:55.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-09-18T15:10:55.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T15:10:55.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T15:10:55.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T15:10:55.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-09-18T15:10:55.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-09-18T15:10:56.164 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T15:10:56.164 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:10:56.164 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:10:56.164 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:10:56.164 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:10:56.165 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:10:56.165 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:10:56.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T15:10:56.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T15:10:56.417 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T15:10:56.417 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:10:56.417 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:10:56.417 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T15:10:56.417 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T15:10:56.417 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:10:56.417 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:10:56.417 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:10:56.418 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:10:56.418 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:10:56.418 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:10:56.418 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.288083/ceph-mon.a.asok 2024-09-18T15:10:56.419 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:10:56.419 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.288083/ceph-mon.a.asok config get fsid 2024-09-18T15:10:56.502 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T15:10:56.502 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:10:56.503 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:10:56.503 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T15:10:56.503 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T15:10:56.503 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:10:56.503 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:10:56.503 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:10:56.503 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:10:56.503 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:10:56.503 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:10:56.503 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.288083/ceph-mon.a.asok 2024-09-18T15:10:56.503 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:10:56.503 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.288083/ceph-mon.a.asok config get mon_host 2024-09-18T15:10:56.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:122: do_recovery_out1: run_mgr td/osd-recovery-stats x 2024-09-18T15:10:56.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-09-18T15:10:56.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T15:10:56.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T15:10:56.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T15:10:56.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-09-18T15:10:56.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T15:10:57.430 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T15:10:57.430 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:10:57.430 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:10:57.430 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:10:57.430 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:10:57.430 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:10:57.430 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:10:57.431 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:10:57.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:10:57.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:123: do_recovery_out1: run_osd td/osd-recovery-stats 0 2024-09-18T15:10:57.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:10:57.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:10:57.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T15:10:57.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:10:57.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-09-18T15:10:57.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:10:57.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:10:57.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:10:57.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:10:57.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-09-18T15:10:57.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-09-18T15:10:57.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:10:57.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:10:57.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:10:57.459 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:10:57.460 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:10:57.460 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:10:57.460 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:10:57.460 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:10:57.460 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:10:57.460 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:10:57.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:10:57.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:10:57.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:10:57.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:10:57.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:10:57.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:10:57.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:10:57.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:10:57.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:10:57.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:10:57.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:10:57.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:10:57.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-09-18T15:10:57.461 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:10:57.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ad6524c0-00a9-418f-8090-05fcc703550b 2024-09-18T15:10:57.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ad6524c0-00a9-418f-8090-05fcc703550b' 2024-09-18T15:10:57.462 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 ad6524c0-00a9-418f-8090-05fcc703550b 2024-09-18T15:10:57.462 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:10:57.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCB7epmhSkuHBAAaETEEu+bSibKWn/A9/sK1Q== 2024-09-18T15:10:57.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCB7epmhSkuHBAAaETEEu+bSibKWn/A9/sK1Q=="}' 2024-09-18T15:10:57.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ad6524c0-00a9-418f-8090-05fcc703550b -i td/osd-recovery-stats/0/new.json 2024-09-18T15:10:57.771 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:10:57.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-09-18T15:10:57.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCB7epmhSkuHBAAaETEEu+bSibKWn/A9/sK1Q== --osd-uuid ad6524c0-00a9-418f-8090-05fcc703550b 2024-09-18T15:10:57.807 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:10:57.805+0000 7f6ebc415440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:10:57.824 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:10:57.823+0000 7f6ebc415440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:10:57.825 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:10:57.824+0000 7f6ebc415440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:10:57.826 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:10:57.825+0000 7f6ebc415440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-18T15:10:57.826 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:10:57.825+0000 7f6ebc415440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-18T15:10:57.826 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:10:57.825+0000 7f6ebc415440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-18T15:10:57.856 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:10:57.855+0000 7f6ebc415440 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-09-18T15:11:03.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-09-18T15:11:03.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:11:03.239 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T15:11:03.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T15:11:03.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:11:03.776 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T15:11:03.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T15:11:03.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:11:03.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:11:03.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:11:03.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:11:03.798 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:11:03.797+0000 7fc5d8439440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:11:03.801 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:11:03.799+0000 7fc5d8439440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:11:03.802 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:11:03.800+0000 7fc5d8439440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:11:04.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T15:11:04.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:11:04.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:11:04.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:11:04.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:11:04.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:04.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:11:04.179 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:11:04.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:04.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:11:04.323 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:11:04.322+0000 7fc5d8439440 -1 Falling back to public interface 2024-09-18T15:11:04.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:05.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:05.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:05.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:11:05.581 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:11:05.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:05.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:11:05.989 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:06.794 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:11:06.792+0000 7fc5d8439440 -1 osd.0 0 log_to_monitors true 2024-09-18T15:11:06.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:06.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:06.991 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:11:06.991 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:11:06.991 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:06.991 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:11:07.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:08.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:08.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:08.496 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:11:08.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:11:08.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:08.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:11:08.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:09.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:09.893 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:11:09.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:09.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:11:09.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:09.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:11:10.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:11.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:11.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:11.298 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:11:11.299 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:11:11.299 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:11.299 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:11:11.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:12.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:12.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:12.766 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:11:12.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:11:12.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:12.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:11:13.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:14.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:14.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:14.185 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:11:14.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:11:14.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:14.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:11:14.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:15.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:15.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:15.582 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:11:15.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:11:15.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:15.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:11:15.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:16.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:16.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:16.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:11:16.996 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:11:16.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:16.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:11:17.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:18.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:18.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:18.463 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:11:18.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:11:18.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:18.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:11:19.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:20.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:20.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:20.290 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:11:20.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:11:20.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:20.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:11:20.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:21.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:21.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:21.886 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:11:21.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:11:21.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:21.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:11:22.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:23.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:23.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:23.519 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:11:23.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:11:23.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:23.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:11:24.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:25.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:25.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:25.524 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:11:25.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:11:25.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:25.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:11:26.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:27.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:27.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:27.057 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:11:27.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:11:27.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:27.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:11:27.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:28.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:28.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:28.437 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:11:28.437 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:11:28.437 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:28.437 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:11:29.031 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/289174,v1:127.0.0.1:6803/289174] [v2:127.0.0.1:6804/289174,v1:127.0.0.1:6805/289174] exists,up ad6524c0-00a9-418f-8090-05fcc703550b 2024-09-18T15:11:29.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:11:29.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:11:29.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:11:29.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:124: do_recovery_out1: run_osd td/osd-recovery-stats 1 2024-09-18T15:11:29.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:11:29.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:11:29.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T15:11:29.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:11:29.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-09-18T15:11:29.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:11:29.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:11:29.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:11:29.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:11:29.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-09-18T15:11:29.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-09-18T15:11:29.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:11:29.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:11:29.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:11:29.035 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:11:29.036 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:11:29.036 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:11:29.036 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:11:29.036 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:11:29.036 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:11:29.036 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:11:29.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:11:29.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:11:29.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:11:29.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:11:29.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:11:29.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:11:29.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:11:29.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:11:29.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:11:29.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:11:29.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:11:29.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:11:29.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-09-18T15:11:29.037 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:11:29.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f6b6eb9d-7f85-4942-ac79-68985b05ffb8 2024-09-18T15:11:29.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f6b6eb9d-7f85-4942-ac79-68985b05ffb8' 2024-09-18T15:11:29.037 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 f6b6eb9d-7f85-4942-ac79-68985b05ffb8 2024-09-18T15:11:29.038 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:11:29.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCh7epmidjZAhAA0t5PAYyrcp3w4QDXAgqKlg== 2024-09-18T15:11:29.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCh7epmidjZAhAA0t5PAYyrcp3w4QDXAgqKlg=="}' 2024-09-18T15:11:29.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f6b6eb9d-7f85-4942-ac79-68985b05ffb8 -i td/osd-recovery-stats/1/new.json 2024-09-18T15:11:29.566 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:11:29.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-09-18T15:11:29.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCh7epmidjZAhAA0t5PAYyrcp3w4QDXAgqKlg== --osd-uuid f6b6eb9d-7f85-4942-ac79-68985b05ffb8 2024-09-18T15:11:29.603 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:11:29.601+0000 7fd0fa520440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:11:29.604 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:11:29.603+0000 7fd0fa520440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:11:29.606 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:11:29.604+0000 7fd0fa520440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:11:29.606 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:11:29.605+0000 7fd0fa520440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-18T15:11:29.606 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:11:29.605+0000 7fd0fa520440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-18T15:11:29.606 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:11:29.605+0000 7fd0fa520440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-18T15:11:29.681 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:11:29.680+0000 7fd0fa520440 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-09-18T15:11:35.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-09-18T15:11:35.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:11:35.277 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T15:11:35.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T15:11:35.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:11:35.790 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T15:11:35.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T15:11:35.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:11:35.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:11:35.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:11:35.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:11:35.813 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:11:35.812+0000 7fdcbb84f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:11:35.815 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:11:35.814+0000 7fdcbb84f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:11:35.816 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:11:35.815+0000 7fdcbb84f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:11:36.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T15:11:36.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:11:36.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:11:36.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:11:36.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:11:36.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:36.184 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:11:36.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:11:36.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:36.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:11:36.343 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:11:36.342+0000 7fdcbb84f440 -1 Falling back to public interface 2024-09-18T15:11:36.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:37.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:37.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:37.586 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:11:37.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:11:37.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:37.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:11:37.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:38.808 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:11:38.807+0000 7fdcbb84f440 -1 osd.1 0 log_to_monitors true 2024-09-18T15:11:38.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:38.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:38.996 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:11:38.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:11:38.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:38.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:11:39.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:40.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:40.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:40.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:11:40.451 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:11:40.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:40.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:11:40.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:41.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:41.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:41.899 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:11:41.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:11:41.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:41.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:11:42.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:43.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:43.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:43.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:11:43.362 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:11:43.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:43.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:11:43.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:44.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:44.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:44.889 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:11:44.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:11:44.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:44.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:11:45.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:46.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:46.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:46.282 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:11:46.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:11:46.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:46.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:11:46.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:47.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:47.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:47.764 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:11:47.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:11:47.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:47.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:11:48.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:49.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:49.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:49.270 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:11:49.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:11:49.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:49.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:11:49.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:50.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:50.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:50.700 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:11:50.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:11:50.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:50.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:11:51.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:52.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:52.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:52.452 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:11:52.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:11:52.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:52.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:11:53.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:54.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:54.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:54.117 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:11:54.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:11:54.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:54.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:11:54.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:55.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:55.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:55.769 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:11:55.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:11:55.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:55.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:11:56.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:57.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:57.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:57.758 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:11:57.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:11:57.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:57.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:11:58.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:11:59.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:11:59.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:11:59.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:11:59.215 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:11:59.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:11:59.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:11:59.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:00.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:00.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:00.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:12:00.900 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:12:00.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:00.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:12:01.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:02.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:02.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:02.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:12:02.372 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T15:12:02.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:02.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:12:02.784 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/290903,v1:127.0.0.1:6811/290903] [v2:127.0.0.1:6812/290903,v1:127.0.0.1:6813/290903] exists,up f6b6eb9d-7f85-4942-ac79-68985b05ffb8 2024-09-18T15:12:02.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:12:02.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:12:02.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:12:02.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:125: do_recovery_out1: run_osd td/osd-recovery-stats 2 2024-09-18T15:12:02.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:12:02.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:12:02.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T15:12:02.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:12:02.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-09-18T15:12:02.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:12:02.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:12:02.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:12:02.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:12:02.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-09-18T15:12:02.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-09-18T15:12:02.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:12:02.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:12:02.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:12:02.793 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:12:02.793 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:12:02.793 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:12:02.793 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:12:02.793 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:12:02.793 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:12:02.793 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:12:02.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:12:02.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:12:02.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:12:02.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:12:02.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:12:02.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:12:02.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:12:02.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:12:02.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:12:02.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:12:02.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:12:02.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:12:02.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-09-18T15:12:02.795 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:12:02.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=de2f8034-3d25-49c4-ac07-5e3fc9092e7b 2024-09-18T15:12:02.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 de2f8034-3d25-49c4-ac07-5e3fc9092e7b' 2024-09-18T15:12:02.795 INFO:tasks.workunit.client.0.smithi047.stdout:add osd2 de2f8034-3d25-49c4-ac07-5e3fc9092e7b 2024-09-18T15:12:02.796 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:12:02.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDC7epmpW7FLxAANwlPZ7UWvVwN3TWAx0x7XA== 2024-09-18T15:12:02.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDC7epmpW7FLxAANwlPZ7UWvVwN3TWAx0x7XA=="}' 2024-09-18T15:12:02.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new de2f8034-3d25-49c4-ac07-5e3fc9092e7b -i td/osd-recovery-stats/2/new.json 2024-09-18T15:12:03.307 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:12:03.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-09-18T15:12:03.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDC7epmpW7FLxAANwlPZ7UWvVwN3TWAx0x7XA== --osd-uuid de2f8034-3d25-49c4-ac07-5e3fc9092e7b 2024-09-18T15:12:03.343 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:12:03.342+0000 7f20b6725440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:12:03.369 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:12:03.365+0000 7f20b6725440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:12:03.369 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:12:03.366+0000 7f20b6725440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:12:03.369 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:12:03.366+0000 7f20b6725440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-18T15:12:03.369 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:12:03.366+0000 7f20b6725440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-18T15:12:03.369 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:12:03.366+0000 7f20b6725440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-18T15:12:03.458 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:12:03.457+0000 7f20b6725440 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-09-18T15:12:09.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-09-18T15:12:09.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:12:09.064 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd2 key to auth repository 2024-09-18T15:12:09.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T15:12:09.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:12:09.571 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T15:12:09.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T15:12:09.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:12:09.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:12:09.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:12:09.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:12:09.594 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:12:09.593+0000 7f54ec524440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:12:09.596 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:12:09.595+0000 7f54ec524440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:12:09.597 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:12:09.596+0000 7f54ec524440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:12:09.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T15:12:09.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:12:09.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:12:09.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:12:09.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:12:09.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:09.940 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:12:09.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:12:09.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:09.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:12:10.111 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:12:10.110+0000 7f54ec524440 -1 Falling back to public interface 2024-09-18T15:12:10.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:11.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:11.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:11.309 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:12:11.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:12:11.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:11.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:12:11.780 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:12.585 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:12:12.583+0000 7f54ec524440 -1 osd.2 0 log_to_monitors true 2024-09-18T15:12:12.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:12.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:12.782 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:12:12.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:12:12.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:12.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:12:13.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:14.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:14.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:14.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:12:14.387 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:12:14.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:14.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:12:14.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:15.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:15.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:15.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:12:15.784 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:12:15.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:15.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:12:16.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:17.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:17.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:17.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:12:17.376 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:12:17.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:17.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:12:17.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:18.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:18.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:18.918 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:12:18.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:12:18.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:18.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:12:19.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:20.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:20.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:20.468 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:12:20.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:12:20.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:20.469 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:12:20.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:21.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:21.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:21.903 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:12:21.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:12:21.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:21.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:12:22.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:23.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:23.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:23.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:12:23.395 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:12:23.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:23.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:12:23.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:24.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:24.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:24.817 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:12:24.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:12:24.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:24.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:12:25.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:26.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:26.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:26.288 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:12:26.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:12:26.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:26.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:12:26.992 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:27.993 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:27.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:27.994 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:12:27.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:12:27.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:27.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:12:28.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:29.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:29.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:29.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:12:29.961 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:12:29.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:29.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:12:30.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:31.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:31.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:31.726 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:12:31.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:12:31.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:31.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:12:32.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:33.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:33.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:33.773 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:12:33.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:12:33.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:33.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:12:34.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:35.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:35.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:35.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:12:35.462 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:12:35.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:35.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:12:35.859 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/292657,v1:127.0.0.1:6819/292657] [v2:127.0.0.1:6820/292657,v1:127.0.0.1:6821/292657] exists,up de2f8034-3d25-49c4-ac07-5e3fc9092e7b 2024-09-18T15:12:35.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:12:35.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:12:35.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:12:35.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:126: do_recovery_out1: run_osd td/osd-recovery-stats 3 2024-09-18T15:12:35.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:12:35.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:12:35.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-18T15:12:35.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:12:35.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-09-18T15:12:35.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:12:35.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:12:35.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:12:35.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:12:35.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-09-18T15:12:35.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-09-18T15:12:35.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:12:35.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:12:35.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:12:35.864 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:12:35.864 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:12:35.864 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:12:35.864 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:12:35.864 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:12:35.865 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:12:35.865 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:12:35.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:12:35.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:12:35.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:12:35.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:12:35.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:12:35.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:12:35.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:12:35.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:12:35.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:12:35.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:12:35.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:12:35.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:12:35.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-09-18T15:12:35.866 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:12:35.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=384fc33d-2474-4ea7-8a00-91205df19f87 2024-09-18T15:12:35.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 384fc33d-2474-4ea7-8a00-91205df19f87' 2024-09-18T15:12:35.867 INFO:tasks.workunit.client.0.smithi047.stdout:add osd3 384fc33d-2474-4ea7-8a00-91205df19f87 2024-09-18T15:12:35.867 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:12:35.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDj7epmbnI/NBAAf8dP5UOLTpMnTwYlts1DVQ== 2024-09-18T15:12:35.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDj7epmbnI/NBAAf8dP5UOLTpMnTwYlts1DVQ=="}' 2024-09-18T15:12:35.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 384fc33d-2474-4ea7-8a00-91205df19f87 -i td/osd-recovery-stats/3/new.json 2024-09-18T15:12:36.407 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:12:36.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-09-18T15:12:36.419 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDj7epmbnI/NBAAf8dP5UOLTpMnTwYlts1DVQ== --osd-uuid 384fc33d-2474-4ea7-8a00-91205df19f87 2024-09-18T15:12:36.444 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:12:36.442+0000 7fd6d3d2a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:12:36.446 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:12:36.444+0000 7fd6d3d2a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:12:36.447 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:12:36.445+0000 7fd6d3d2a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:12:36.447 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:12:36.446+0000 7fd6d3d2a440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-18T15:12:36.447 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:12:36.446+0000 7fd6d3d2a440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-18T15:12:36.447 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:12:36.446+0000 7fd6d3d2a440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-18T15:12:36.497 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:12:36.496+0000 7fd6d3d2a440 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-09-18T15:12:42.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-09-18T15:12:42.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:12:42.498 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd3 key to auth repository 2024-09-18T15:12:42.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-18T15:12:42.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:12:43.010 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.3 2024-09-18T15:12:43.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-18T15:12:43.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:12:43.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:12:43.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:12:43.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:12:43.033 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:12:43.032+0000 7f7eb3c34440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:12:43.056 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:12:43.055+0000 7f7eb3c34440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:12:43.058 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:12:43.056+0000 7f7eb3c34440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:12:43.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-18T15:12:43.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:12:43.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T15:12:43.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:12:43.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:12:43.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:43.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:12:43.429 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:12:43.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:43.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:12:43.581 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:12:43.580+0000 7f7eb3c34440 -1 Falling back to public interface 2024-09-18T15:12:43.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:44.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:44.794 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:12:44.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:44.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:12:44.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:44.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:12:45.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:46.162 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:12:46.161+0000 7f7eb3c34440 -1 osd.3 0 log_to_monitors true 2024-09-18T15:12:46.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:46.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:46.170 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:12:46.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:12:46.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:46.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:12:46.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:47.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:47.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:47.698 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:12:47.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:12:47.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:47.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:12:48.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:49.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:49.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:49.170 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:12:49.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:12:49.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:49.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:12:49.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:50.549 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:50.549 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:50.550 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:12:50.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:12:50.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:50.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:12:51.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:52.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:52.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:52.010 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:12:52.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:12:52.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:52.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:12:52.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:53.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:53.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:53.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:12:53.457 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:12:53.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:53.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:12:53.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:54.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:54.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:54.915 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:12:54.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:12:54.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:54.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:12:55.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:56.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:56.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:56.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:12:56.296 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:12:56.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:56.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:12:56.699 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:57.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:57.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:57.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:12:57.701 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:12:57.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:57.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:12:58.265 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:12:59.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:12:59.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:12:59.266 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:12:59.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:12:59.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:12:59.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:12:59.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:00.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:00.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:00.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:13:00.800 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:13:00.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:00.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:13:01.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:02.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:02.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:02.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:13:02.554 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:13:02.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:02.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:13:03.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:04.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:04.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:04.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:13:04.192 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:13:04.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:04.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:13:04.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:05.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:05.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:05.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:13:05.928 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:13:05.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:05.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:13:07.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:08.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:08.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:08.076 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:13:08.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:13:08.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:08.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:13:08.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:09.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:09.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:09.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:13:09.590 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T15:13:09.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:09.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:13:09.963 INFO:tasks.workunit.client.0.smithi047.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/294386,v1:127.0.0.1:6827/294386] [v2:127.0.0.1:6828/294386,v1:127.0.0.1:6829/294386] exists,up 384fc33d-2474-4ea7-8a00-91205df19f87 2024-09-18T15:13:09.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:13:09.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:13:09.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:13:09.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:127: do_recovery_out1: run_osd td/osd-recovery-stats 4 2024-09-18T15:13:09.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:13:09.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:13:09.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-18T15:13:09.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:13:09.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2024-09-18T15:13:09.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:13:09.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:13:09.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:13:09.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:13:09.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2024-09-18T15:13:09.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2024-09-18T15:13:09.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:13:09.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:13:09.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:13:09.969 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:13:09.969 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:13:09.969 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:13:09.969 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:13:09.970 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:13:09.970 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:13:09.970 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:13:09.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:13:09.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:13:09.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:13:09.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:13:09.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:13:09.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:13:09.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:13:09.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:13:09.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:13:09.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:13:09.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:13:09.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:13:09.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2024-09-18T15:13:09.971 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:13:09.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b8633135-02d0-4326-879b-d94e05b65acb 2024-09-18T15:13:09.971 INFO:tasks.workunit.client.0.smithi047.stdout:add osd4 b8633135-02d0-4326-879b-d94e05b65acb 2024-09-18T15:13:09.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 b8633135-02d0-4326-879b-d94e05b65acb' 2024-09-18T15:13:09.972 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:13:09.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAF7upmMKt7OhAAZJ9ht62AyJqktAmFUjKAIg== 2024-09-18T15:13:09.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAF7upmMKt7OhAAZJ9ht62AyJqktAmFUjKAIg=="}' 2024-09-18T15:13:09.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b8633135-02d0-4326-879b-d94e05b65acb -i td/osd-recovery-stats/4/new.json 2024-09-18T15:13:10.458 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:13:10.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2024-09-18T15:13:10.469 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAF7upmMKt7OhAAZJ9ht62AyJqktAmFUjKAIg== --osd-uuid b8633135-02d0-4326-879b-d94e05b65acb 2024-09-18T15:13:10.494 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:13:10.493+0000 7ff456c45440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:13:10.496 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:13:10.495+0000 7ff456c45440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:13:10.497 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:13:10.496+0000 7ff456c45440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:13:10.498 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:13:10.496+0000 7ff456c45440 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-18T15:13:10.498 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:13:10.496+0000 7ff456c45440 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-18T15:13:10.498 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:13:10.496+0000 7ff456c45440 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-18T15:13:10.582 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:13:10.581+0000 7ff456c45440 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2024-09-18T15:13:16.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2024-09-18T15:13:16.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:13:16.468 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd4 key to auth repository 2024-09-18T15:13:16.469 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-18T15:13:16.469 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:13:16.991 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.4 2024-09-18T15:13:16.992 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-18T15:13:16.992 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:13:16.992 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:13:16.992 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:13:16.992 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:13:17.015 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:13:17.014+0000 7f1823926440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:13:17.057 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:13:17.055+0000 7f1823926440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:13:17.058 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:13:17.056+0000 7f1823926440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:13:17.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-18T15:13:17.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:13:17.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-18T15:13:17.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:13:17.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:13:17.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:17.361 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:13:17.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:13:17.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:17.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:13:17.586 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:13:17.585+0000 7f1823926440 -1 Falling back to public interface 2024-09-18T15:13:17.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:18.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:18.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:18.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:13:18.750 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:13:18.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:18.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:13:19.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:20.121 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:13:20.120+0000 7f1823926440 -1 osd.4 0 log_to_monitors true 2024-09-18T15:13:20.157 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:20.157 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:20.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:13:20.158 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:13:20.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:20.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:13:20.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:21.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:21.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:21.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:13:21.779 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:13:21.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:21.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:13:22.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:23.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:23.333 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:13:23.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:23.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:13:23.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:23.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:13:23.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:24.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:24.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:24.789 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:13:24.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:13:24.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:24.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:13:25.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:26.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:26.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:26.185 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:13:26.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:13:26.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:26.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:13:26.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:27.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:27.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:27.575 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:13:27.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:13:27.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:27.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:13:27.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:28.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:28.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:28.952 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:13:28.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:13:28.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:28.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:13:29.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:30.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:30.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:30.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:13:30.445 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:13:30.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:30.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:13:30.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:31.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:31.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:31.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:13:31.929 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:13:31.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:31.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:13:32.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:33.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:33.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:33.569 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:13:33.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:13:33.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:33.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:13:34.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:35.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:35.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:35.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:13:35.315 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:13:35.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:35.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:13:36.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:37.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:37.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:37.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:13:37.133 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:13:37.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:37.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:13:37.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:38.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:38.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:38.631 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:13:38.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:13:38.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:38.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:13:39.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:40.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:40.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:40.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:13:40.354 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:13:40.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:40.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:13:41.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:42.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:42.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:42.051 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:13:42.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:13:42.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:42.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:13:42.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:43.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:43.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:43.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:13:43.610 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T15:13:43.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:43.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:13:44.026 INFO:tasks.workunit.client.0.smithi047.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/296141,v1:127.0.0.1:6835/296141] [v2:127.0.0.1:6836/296141,v1:127.0.0.1:6837/296141] exists,up b8633135-02d0-4326-879b-d94e05b65acb 2024-09-18T15:13:44.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:13:44.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:13:44.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:13:44.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:128: do_recovery_out1: run_osd td/osd-recovery-stats 5 2024-09-18T15:13:44.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:13:44.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:13:44.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-18T15:13:44.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:13:44.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2024-09-18T15:13:44.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:13:44.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:13:44.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:13:44.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:13:44.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2024-09-18T15:13:44.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2024-09-18T15:13:44.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:13:44.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:13:44.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:13:44.031 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:13:44.031 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:13:44.031 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:13:44.032 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:13:44.032 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:13:44.032 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:13:44.032 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:13:44.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:13:44.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:13:44.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:13:44.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:13:44.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:13:44.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:13:44.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:13:44.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:13:44.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:13:44.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:13:44.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:13:44.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:13:44.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2024-09-18T15:13:44.033 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:13:44.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=eb7e2f1f-5ac1-4893-814c-83852b26f687 2024-09-18T15:13:44.033 INFO:tasks.workunit.client.0.smithi047.stdout:add osd5 eb7e2f1f-5ac1-4893-814c-83852b26f687 2024-09-18T15:13:44.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 eb7e2f1f-5ac1-4893-814c-83852b26f687' 2024-09-18T15:13:44.034 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:13:44.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAo7upmv3ekAhAAs2cQciQJZOj2HtBQ5BNw6w== 2024-09-18T15:13:44.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAo7upmv3ekAhAAs2cQciQJZOj2HtBQ5BNw6w=="}' 2024-09-18T15:13:44.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new eb7e2f1f-5ac1-4893-814c-83852b26f687 -i td/osd-recovery-stats/5/new.json 2024-09-18T15:13:44.536 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:13:44.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2024-09-18T15:13:44.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAo7upmv3ekAhAAs2cQciQJZOj2HtBQ5BNw6w== --osd-uuid eb7e2f1f-5ac1-4893-814c-83852b26f687 2024-09-18T15:13:44.573 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:13:44.572+0000 7fda1e357440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:13:44.577 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:13:44.576+0000 7fda1e357440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:13:44.578 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:13:44.577+0000 7fda1e357440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:13:44.579 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:13:44.577+0000 7fda1e357440 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-18T15:13:44.579 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:13:44.577+0000 7fda1e357440 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-18T15:13:44.579 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:13:44.577+0000 7fda1e357440 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-18T15:13:44.669 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:13:44.668+0000 7fda1e357440 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2024-09-18T15:13:50.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2024-09-18T15:13:50.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:13:50.803 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd5 key to auth repository 2024-09-18T15:13:50.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-18T15:13:50.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:13:51.315 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.5 2024-09-18T15:13:51.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-18T15:13:51.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:13:51.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:13:51.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:13:51.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:13:51.339 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:13:51.337+0000 7f59ec239440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:13:51.362 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:13:51.361+0000 7f59ec239440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:13:51.363 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:13:51.362+0000 7f59ec239440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:13:51.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-18T15:13:51.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:13:51.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-18T15:13:51.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:13:51.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:13:51.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:51.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:13:51.743 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:13:51.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:51.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:13:51.894 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:13:51.893+0000 7f59ec239440 -1 Falling back to public interface 2024-09-18T15:13:52.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:53.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:53.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:53.154 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:13:53.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:13:53.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:53.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:13:53.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:54.377 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:13:54.375+0000 7f59ec239440 -1 osd.5 0 log_to_monitors true 2024-09-18T15:13:54.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:54.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:54.572 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:13:54.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:13:54.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:54.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:13:55.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:56.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:56.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:56.047 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:13:56.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:13:56.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:56.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:13:56.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:57.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:57.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:57.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:13:57.474 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:13:57.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:57.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:13:57.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:13:58.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:13:58.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:13:58.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:13:58.960 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:13:58.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:13:58.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:13:59.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:14:00.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:14:00.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:14:00.425 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:14:00.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:14:00.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:14:00.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:14:00.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:14:01.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:14:01.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:14:01.802 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:14:01.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:14:01.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:14:01.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:14:02.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:14:03.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:14:03.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:14:03.323 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:14:03.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:14:03.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:14:03.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:14:03.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:14:04.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:14:04.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:14:04.821 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:14:04.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:14:04.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:14:04.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:14:05.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:14:06.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:14:06.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:14:06.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:14:06.294 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:14:06.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:14:06.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:14:06.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:14:07.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:14:07.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:14:07.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:14:07.767 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:14:07.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:14:07.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:14:08.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:14:09.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:14:09.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:14:09.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:14:09.297 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:14:09.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:14:09.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:14:09.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:14:10.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:14:10.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:14:10.770 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:14:10.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:14:10.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:14:10.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:14:11.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:14:12.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:14:12.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:14:12.686 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:14:12.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:14:12.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:14:12.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:14:13.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:14:14.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:14:14.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:14:14.447 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:14:14.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:14:14.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:14:14.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:14:15.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:14:16.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:14:16.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:14:16.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:14:16.134 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:14:16.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:14:16.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:14:16.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:14:17.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:14:17.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:14:17.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:14:17.870 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T15:14:17.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:14:17.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:14:18.268 INFO:tasks.workunit.client.0.smithi047.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/297895,v1:127.0.0.1:6843/297895] [v2:127.0.0.1:6844/297895,v1:127.0.0.1:6845/297895] exists,up eb7e2f1f-5ac1-4893-814c-83852b26f687 2024-09-18T15:14:18.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:14:18.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:14:18.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:14:18.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:130: do_recovery_out1: '[' erasure = erasure ']' 2024-09-18T15:14:18.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:132: do_recovery_out1: ceph osd erasure-code-profile set myprofile plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-18T15:14:18.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:133: do_recovery_out1: create_pool test 1 1 erasure myprofile 2024-09-18T15:14:18.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 erasure myprofile 2024-09-18T15:14:19.666 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test' already exists 2024-09-18T15:14:19.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:14:20.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:138: do_recovery_out1: wait_for_clean 2024-09-18T15:14:20.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:14:20.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:14:20.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:14:20.679 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:14:20.680 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:14:20.680 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:14:20.680 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:14:20.680 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:14:20.680 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:14:20.780 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:14:20.780 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:14:20.780 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:14:20.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:14:20.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:14:20.781 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:14:21.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:14:21.161 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:14:21.161 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T15:14:21.161 INFO:tasks.workunit.client.0.smithi047.stderr:3 2024-09-18T15:14:21.161 INFO:tasks.workunit.client.0.smithi047.stderr:4 2024-09-18T15:14:21.161 INFO:tasks.workunit.client.0.smithi047.stderr:5' 2024-09-18T15:14:21.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:14:21.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:14:21.162 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:14:21.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836518 2024-09-18T15:14:21.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836518 2024-09-18T15:14:21.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836518' 2024-09-18T15:14:21.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:14:21.270 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:14:21.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672993 2024-09-18T15:14:21.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672993 2024-09-18T15:14:21.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836518 1-42949672993' 2024-09-18T15:14:21.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:14:21.378 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:14:21.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509470 2024-09-18T15:14:21.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509470 2024-09-18T15:14:21.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836518 1-42949672993 2-64424509470' 2024-09-18T15:14:21.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:14:21.487 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:14:21.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345939 2024-09-18T15:14:21.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345939 2024-09-18T15:14:21.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836518 1-42949672993 2-64424509470 3-85899345939' 2024-09-18T15:14:21.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:14:21.635 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:14:21.741 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182415 2024-09-18T15:14:21.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182415 2024-09-18T15:14:21.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836518 1-42949672993 2-64424509470 3-85899345939 4-107374182415' 2024-09-18T15:14:21.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:14:21.742 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:14:21.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018891 2024-09-18T15:14:21.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018891 2024-09-18T15:14:21.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836518 1-42949672993 2-64424509470 3-85899345939 4-107374182415 5-128849018891' 2024-09-18T15:14:21.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:14:21.851 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836518 2024-09-18T15:14:21.851 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:14:21.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:14:21.852 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836518 2024-09-18T15:14:21.852 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:14:21.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836518 2024-09-18T15:14:21.853 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836518 2024-09-18T15:14:21.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2024-09-18T15:14:21.854 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:14:22.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836516 -lt 21474836518 2024-09-18T15:14:22.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:14:23.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:14:23.220 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:14:23.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836518 -lt 21474836518 2024-09-18T15:14:23.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:14:23.633 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672993 2024-09-18T15:14:23.633 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:14:23.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:14:23.634 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672993 2024-09-18T15:14:23.634 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:14:23.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672993 2024-09-18T15:14:23.635 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672993 2024-09-18T15:14:23.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672993' 2024-09-18T15:14:23.636 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:14:24.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672994 -lt 42949672993 2024-09-18T15:14:24.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:14:24.042 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509470 2024-09-18T15:14:24.042 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:14:24.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:14:24.044 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509470 2024-09-18T15:14:24.044 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:14:24.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509470 2024-09-18T15:14:24.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509470' 2024-09-18T15:14:24.045 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509470 2024-09-18T15:14:24.045 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:14:24.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509470 -lt 64424509470 2024-09-18T15:14:24.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:14:24.412 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345939 2024-09-18T15:14:24.412 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:14:24.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:14:24.413 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345939 2024-09-18T15:14:24.413 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:14:24.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345939 2024-09-18T15:14:24.414 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 85899345939 2024-09-18T15:14:24.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345939' 2024-09-18T15:14:24.415 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:14:24.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345939 -lt 85899345939 2024-09-18T15:14:24.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:14:24.819 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182415 2024-09-18T15:14:24.819 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:14:24.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:14:24.821 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182415 2024-09-18T15:14:24.821 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:14:24.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182415 2024-09-18T15:14:24.822 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.4 seq 107374182415 2024-09-18T15:14:24.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182415' 2024-09-18T15:14:24.823 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:14:25.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182415 -lt 107374182415 2024-09-18T15:14:25.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:14:25.185 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018891 2024-09-18T15:14:25.185 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:14:25.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:14:25.187 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018891 2024-09-18T15:14:25.187 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:14:25.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018891 2024-09-18T15:14:25.188 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.5 seq 128849018891 2024-09-18T15:14:25.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018891' 2024-09-18T15:14:25.188 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:14:25.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018892 -lt 128849018891 2024-09-18T15:14:25.581 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:14:25.581 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:14:25.581 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:14:26.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:14:26.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:14:26.083 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:14:26.084 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:14:26.084 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:14:26.084 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:14:26.084 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:14:26.084 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:14:26.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:14:26.446 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:14:26.446 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:14:26.446 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:14:26.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:14:26.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:14:26.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:14:26.941 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: seq 1 200 2024-09-18T15:14:26.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:26.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj1 /dev/null 2024-09-18T15:14:27.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:27.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj2 /dev/null 2024-09-18T15:14:27.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:27.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj3 /dev/null 2024-09-18T15:14:27.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:27.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj4 /dev/null 2024-09-18T15:14:27.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:27.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj5 /dev/null 2024-09-18T15:14:27.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:27.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj6 /dev/null 2024-09-18T15:14:28.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:28.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj7 /dev/null 2024-09-18T15:14:28.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:28.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj8 /dev/null 2024-09-18T15:14:28.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:28.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj9 /dev/null 2024-09-18T15:14:28.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:28.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj10 /dev/null 2024-09-18T15:14:28.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:28.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj11 /dev/null 2024-09-18T15:14:28.956 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:28.956 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj12 /dev/null 2024-09-18T15:14:29.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:29.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj13 /dev/null 2024-09-18T15:14:29.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:29.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj14 /dev/null 2024-09-18T15:14:29.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:29.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj15 /dev/null 2024-09-18T15:14:29.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:29.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj16 /dev/null 2024-09-18T15:14:29.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:29.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj17 /dev/null 2024-09-18T15:14:30.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:30.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj18 /dev/null 2024-09-18T15:14:30.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:30.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj19 /dev/null 2024-09-18T15:14:30.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:30.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj20 /dev/null 2024-09-18T15:14:30.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:30.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj21 /dev/null 2024-09-18T15:14:30.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:30.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj22 /dev/null 2024-09-18T15:14:31.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:31.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj23 /dev/null 2024-09-18T15:14:31.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:31.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj24 /dev/null 2024-09-18T15:14:31.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:31.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj25 /dev/null 2024-09-18T15:14:31.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:31.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj26 /dev/null 2024-09-18T15:14:31.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:31.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj27 /dev/null 2024-09-18T15:14:31.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:31.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj28 /dev/null 2024-09-18T15:14:32.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:32.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj29 /dev/null 2024-09-18T15:14:32.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:32.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj30 /dev/null 2024-09-18T15:14:32.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:32.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj31 /dev/null 2024-09-18T15:14:32.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:32.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj32 /dev/null 2024-09-18T15:14:32.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:32.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj33 /dev/null 2024-09-18T15:14:32.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:32.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj34 /dev/null 2024-09-18T15:14:33.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:33.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj35 /dev/null 2024-09-18T15:14:33.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:33.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj36 /dev/null 2024-09-18T15:14:33.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:33.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj37 /dev/null 2024-09-18T15:14:33.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:33.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj38 /dev/null 2024-09-18T15:14:33.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:33.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj39 /dev/null 2024-09-18T15:14:34.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:34.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj40 /dev/null 2024-09-18T15:14:34.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:34.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj41 /dev/null 2024-09-18T15:14:34.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:34.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj42 /dev/null 2024-09-18T15:14:34.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:34.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj43 /dev/null 2024-09-18T15:14:34.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:34.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj44 /dev/null 2024-09-18T15:14:34.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:34.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj45 /dev/null 2024-09-18T15:14:35.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:35.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj46 /dev/null 2024-09-18T15:14:35.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:35.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj47 /dev/null 2024-09-18T15:14:35.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:35.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj48 /dev/null 2024-09-18T15:14:35.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:35.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj49 /dev/null 2024-09-18T15:14:36.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:36.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj50 /dev/null 2024-09-18T15:14:36.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:36.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj51 /dev/null 2024-09-18T15:14:36.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:36.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj52 /dev/null 2024-09-18T15:14:36.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:36.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj53 /dev/null 2024-09-18T15:14:36.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:36.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj54 /dev/null 2024-09-18T15:14:36.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:36.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj55 /dev/null 2024-09-18T15:14:37.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:37.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj56 /dev/null 2024-09-18T15:14:37.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:37.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj57 /dev/null 2024-09-18T15:14:37.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:37.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj58 /dev/null 2024-09-18T15:14:37.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:37.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj59 /dev/null 2024-09-18T15:14:37.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:37.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj60 /dev/null 2024-09-18T15:14:38.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:38.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj61 /dev/null 2024-09-18T15:14:38.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:38.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj62 /dev/null 2024-09-18T15:14:38.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:38.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj63 /dev/null 2024-09-18T15:14:38.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:38.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj64 /dev/null 2024-09-18T15:14:38.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:38.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj65 /dev/null 2024-09-18T15:14:39.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:39.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj66 /dev/null 2024-09-18T15:14:39.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:39.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj67 /dev/null 2024-09-18T15:14:39.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:39.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj68 /dev/null 2024-09-18T15:14:39.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:39.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj69 /dev/null 2024-09-18T15:14:39.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:39.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj70 /dev/null 2024-09-18T15:14:39.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:39.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj71 /dev/null 2024-09-18T15:14:40.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:40.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj72 /dev/null 2024-09-18T15:14:40.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:40.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj73 /dev/null 2024-09-18T15:14:40.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:40.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj74 /dev/null 2024-09-18T15:14:40.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:40.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj75 /dev/null 2024-09-18T15:14:40.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:40.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj76 /dev/null 2024-09-18T15:14:41.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:41.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj77 /dev/null 2024-09-18T15:14:41.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:41.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj78 /dev/null 2024-09-18T15:14:41.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:41.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj79 /dev/null 2024-09-18T15:14:41.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:41.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj80 /dev/null 2024-09-18T15:14:41.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:41.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj81 /dev/null 2024-09-18T15:14:41.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:41.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj82 /dev/null 2024-09-18T15:14:42.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:42.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj83 /dev/null 2024-09-18T15:14:42.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:42.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj84 /dev/null 2024-09-18T15:14:42.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:42.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj85 /dev/null 2024-09-18T15:14:42.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:42.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj86 /dev/null 2024-09-18T15:14:42.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:42.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj87 /dev/null 2024-09-18T15:14:42.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:42.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj88 /dev/null 2024-09-18T15:14:43.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:43.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj89 /dev/null 2024-09-18T15:14:43.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:43.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj90 /dev/null 2024-09-18T15:14:43.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:43.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj91 /dev/null 2024-09-18T15:14:43.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:43.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj92 /dev/null 2024-09-18T15:14:43.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:43.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj93 /dev/null 2024-09-18T15:14:44.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:44.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj94 /dev/null 2024-09-18T15:14:44.247 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:44.248 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj95 /dev/null 2024-09-18T15:14:44.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:44.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj96 /dev/null 2024-09-18T15:14:44.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:44.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj97 /dev/null 2024-09-18T15:14:44.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:44.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj98 /dev/null 2024-09-18T15:14:44.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:44.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj99 /dev/null 2024-09-18T15:14:45.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:45.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj100 /dev/null 2024-09-18T15:14:45.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:45.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj101 /dev/null 2024-09-18T15:14:45.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:45.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj102 /dev/null 2024-09-18T15:14:45.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:45.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj103 /dev/null 2024-09-18T15:14:45.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:45.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj104 /dev/null 2024-09-18T15:14:46.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:46.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj105 /dev/null 2024-09-18T15:14:46.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:46.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj106 /dev/null 2024-09-18T15:14:46.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:46.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj107 /dev/null 2024-09-18T15:14:46.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:46.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj108 /dev/null 2024-09-18T15:14:46.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:46.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj109 /dev/null 2024-09-18T15:14:46.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:46.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj110 /dev/null 2024-09-18T15:14:47.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:47.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj111 /dev/null 2024-09-18T15:14:47.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:47.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj112 /dev/null 2024-09-18T15:14:47.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:47.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj113 /dev/null 2024-09-18T15:14:47.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:47.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj114 /dev/null 2024-09-18T15:14:48.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:48.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj115 /dev/null 2024-09-18T15:14:48.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:48.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj116 /dev/null 2024-09-18T15:14:48.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:48.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj117 /dev/null 2024-09-18T15:14:48.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:48.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj118 /dev/null 2024-09-18T15:14:48.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:48.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj119 /dev/null 2024-09-18T15:14:48.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:48.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj120 /dev/null 2024-09-18T15:14:49.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:49.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj121 /dev/null 2024-09-18T15:14:49.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:49.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj122 /dev/null 2024-09-18T15:14:49.491 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:49.491 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj123 /dev/null 2024-09-18T15:14:49.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:49.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj124 /dev/null 2024-09-18T15:14:49.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:49.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj125 /dev/null 2024-09-18T15:14:50.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:50.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj126 /dev/null 2024-09-18T15:14:50.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:50.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj127 /dev/null 2024-09-18T15:14:50.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:50.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj128 /dev/null 2024-09-18T15:14:50.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:50.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj129 /dev/null 2024-09-18T15:14:50.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:50.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj130 /dev/null 2024-09-18T15:14:51.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:51.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj131 /dev/null 2024-09-18T15:14:51.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:51.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj132 /dev/null 2024-09-18T15:14:51.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:51.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj133 /dev/null 2024-09-18T15:14:51.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:51.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj134 /dev/null 2024-09-18T15:14:51.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:51.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj135 /dev/null 2024-09-18T15:14:51.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:51.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj136 /dev/null 2024-09-18T15:14:52.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:52.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj137 /dev/null 2024-09-18T15:14:52.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:52.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj138 /dev/null 2024-09-18T15:14:52.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:52.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj139 /dev/null 2024-09-18T15:14:52.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:52.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj140 /dev/null 2024-09-18T15:14:52.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:52.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj141 /dev/null 2024-09-18T15:14:52.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:52.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj142 /dev/null 2024-09-18T15:14:53.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:53.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj143 /dev/null 2024-09-18T15:14:53.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:53.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj144 /dev/null 2024-09-18T15:14:53.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:53.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj145 /dev/null 2024-09-18T15:14:53.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:53.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj146 /dev/null 2024-09-18T15:14:53.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:53.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj147 /dev/null 2024-09-18T15:14:54.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:54.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj148 /dev/null 2024-09-18T15:14:54.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:54.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj149 /dev/null 2024-09-18T15:14:54.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:54.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj150 /dev/null 2024-09-18T15:14:54.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:54.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj151 /dev/null 2024-09-18T15:14:54.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:54.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj152 /dev/null 2024-09-18T15:14:54.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:54.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj153 /dev/null 2024-09-18T15:14:55.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:55.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj154 /dev/null 2024-09-18T15:14:55.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:55.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj155 /dev/null 2024-09-18T15:14:55.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:55.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj156 /dev/null 2024-09-18T15:14:55.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:55.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj157 /dev/null 2024-09-18T15:14:55.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:55.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj158 /dev/null 2024-09-18T15:14:55.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:55.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj159 /dev/null 2024-09-18T15:14:56.155 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:56.155 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj160 /dev/null 2024-09-18T15:14:56.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:56.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj161 /dev/null 2024-09-18T15:14:56.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:56.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj162 /dev/null 2024-09-18T15:14:56.642 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:56.642 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj163 /dev/null 2024-09-18T15:14:56.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:56.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj164 /dev/null 2024-09-18T15:14:56.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:56.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj165 /dev/null 2024-09-18T15:14:57.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:57.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj166 /dev/null 2024-09-18T15:14:57.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:57.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj167 /dev/null 2024-09-18T15:14:57.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:57.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj168 /dev/null 2024-09-18T15:14:57.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:57.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj169 /dev/null 2024-09-18T15:14:57.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:57.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj170 /dev/null 2024-09-18T15:14:58.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:58.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj171 /dev/null 2024-09-18T15:14:58.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:58.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj172 /dev/null 2024-09-18T15:14:58.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:58.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj173 /dev/null 2024-09-18T15:14:58.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:58.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj174 /dev/null 2024-09-18T15:14:58.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:58.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj175 /dev/null 2024-09-18T15:14:58.949 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:58.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj176 /dev/null 2024-09-18T15:14:59.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:59.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj177 /dev/null 2024-09-18T15:14:59.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:59.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj178 /dev/null 2024-09-18T15:14:59.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:59.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj179 /dev/null 2024-09-18T15:14:59.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:59.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj180 /dev/null 2024-09-18T15:14:59.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:14:59.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj181 /dev/null 2024-09-18T15:15:00.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:15:00.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj182 /dev/null 2024-09-18T15:15:00.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:15:00.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj183 /dev/null 2024-09-18T15:15:00.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:15:00.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj184 /dev/null 2024-09-18T15:15:00.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:15:00.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj185 /dev/null 2024-09-18T15:15:00.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:15:00.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj186 /dev/null 2024-09-18T15:15:00.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:15:00.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj187 /dev/null 2024-09-18T15:15:01.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:15:01.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj188 /dev/null 2024-09-18T15:15:01.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:15:01.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj189 /dev/null 2024-09-18T15:15:01.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:15:01.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj190 /dev/null 2024-09-18T15:15:01.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:15:01.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj191 /dev/null 2024-09-18T15:15:01.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:15:01.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj192 /dev/null 2024-09-18T15:15:02.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:15:02.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj193 /dev/null 2024-09-18T15:15:02.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:15:02.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj194 /dev/null 2024-09-18T15:15:02.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:15:02.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj195 /dev/null 2024-09-18T15:15:02.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:15:02.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj196 /dev/null 2024-09-18T15:15:02.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:15:02.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj197 /dev/null 2024-09-18T15:15:02.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:15:02.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj198 /dev/null 2024-09-18T15:15:03.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:15:03.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj199 /dev/null 2024-09-18T15:15:03.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:15:03.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj200 /dev/null 2024-09-18T15:15:03.507 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: get_primary test obj1 2024-09-18T15:15:03.507 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-18T15:15:03.507 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T15:15:03.507 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-18T15:15:03.507 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:15:03.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: local primary=1 2024-09-18T15:15:03.927 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: get_pg test obj1 2024-09-18T15:15:03.927 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-18T15:15:03.927 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-09-18T15:15:03.927 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-09-18T15:15:03.928 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:15:04.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: local PG=1.0 2024-09-18T15:15:04.295 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: get_not_primary test obj1 2024-09-18T15:15:04.296 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-09-18T15:15:04.296 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-09-18T15:15:04.296 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-09-18T15:15:04.296 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-18T15:15:04.296 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T15:15:04.296 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-18T15:15:04.296 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:15:04.669 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-18T15:15:04.669 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-09-18T15:15:04.669 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-18T15:15:05.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: local otherosd=0 2024-09-18T15:15:05.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:150: do_recovery_out1: ceph osd set norecover 2024-09-18T15:15:05.864 INFO:tasks.workunit.client.0.smithi047.stderr:norecover is set 2024-09-18T15:15:05.876 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: cat td/osd-recovery-stats/osd.0.pid 2024-09-18T15:15:05.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: kill 289174 2024-09-18T15:15:05.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:152: do_recovery_out1: ceph osd down osd.0 2024-09-18T15:15:06.369 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already down. 2024-09-18T15:15:06.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:153: do_recovery_out1: ceph osd out osd.0 2024-09-18T15:15:07.391 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already out. 2024-09-18T15:15:07.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:154: do_recovery_out1: ceph osd unset norecover 2024-09-18T15:15:08.806 INFO:tasks.workunit.client.0.smithi047.stderr:norecover is unset 2024-09-18T15:15:08.821 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: get_primary test obj1 2024-09-18T15:15:08.821 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-18T15:15:08.821 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T15:15:08.821 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-18T15:15:08.821 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:15:09.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-18T15:15:09.340 INFO:tasks.workunit.client.0.smithi047.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-18T15:15:09.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:156: do_recovery_out1: sleep 2 2024-09-18T15:15:11.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:158: do_recovery_out1: wait_for_clean 2024-09-18T15:15:11.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:15:11.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:15:11.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:15:11.350 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:15:11.351 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:15:11.351 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:15:11.351 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:15:11.351 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:15:11.351 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:15:11.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:15:11.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:15:11.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:15:11.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:15:11.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:15:11.452 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:15:11.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:15:11.878 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:15:11.878 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T15:15:11.878 INFO:tasks.workunit.client.0.smithi047.stderr:3 2024-09-18T15:15:11.878 INFO:tasks.workunit.client.0.smithi047.stderr:4 2024-09-18T15:15:11.878 INFO:tasks.workunit.client.0.smithi047.stderr:5' 2024-09-18T15:15:11.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:15:11.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:15:11.879 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:15:11.968 INFO:tasks.workunit.client.0.smithi047.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-18T15:15:11.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq= 2024-09-18T15:15:11.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z '' 2024-09-18T15:15:11.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: flush_pg_stats: continue 2024-09-18T15:15:11.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:15:11.970 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:15:12.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673005 2024-09-18T15:15:12.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673005 2024-09-18T15:15:12.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 1-42949673005' 2024-09-18T15:15:12.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:15:12.162 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:15:12.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509482 2024-09-18T15:15:12.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509482 2024-09-18T15:15:12.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 1-42949673005 2-64424509482' 2024-09-18T15:15:12.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:15:12.271 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:15:12.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345951 2024-09-18T15:15:12.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345951 2024-09-18T15:15:12.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 1-42949673005 2-64424509482 3-85899345951' 2024-09-18T15:15:12.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:15:12.381 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:15:12.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182427 2024-09-18T15:15:12.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182427 2024-09-18T15:15:12.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 1-42949673005 2-64424509482 3-85899345951 4-107374182427' 2024-09-18T15:15:12.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:15:12.490 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:15:12.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018903 2024-09-18T15:15:12.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018903 2024-09-18T15:15:12.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 1-42949673005 2-64424509482 3-85899345951 4-107374182427 5-128849018903' 2024-09-18T15:15:12.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:15:12.601 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673005 2024-09-18T15:15:12.601 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:15:12.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:15:12.602 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673005 2024-09-18T15:15:12.602 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:15:12.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673005 2024-09-18T15:15:12.603 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949673005 2024-09-18T15:15:12.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673005' 2024-09-18T15:15:12.603 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:15:12.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673003 -lt 42949673005 2024-09-18T15:15:12.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:15:13.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:15:13.972 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:15:14.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673006 -lt 42949673005 2024-09-18T15:15:14.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:15:14.354 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509482 2024-09-18T15:15:14.354 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:15:14.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:15:14.355 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509482 2024-09-18T15:15:14.355 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:15:14.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509482 2024-09-18T15:15:14.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509482' 2024-09-18T15:15:14.357 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509482 2024-09-18T15:15:14.357 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:15:14.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509482 -lt 64424509482 2024-09-18T15:15:14.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:15:14.834 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345951 2024-09-18T15:15:14.834 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:15:14.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:15:14.835 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345951 2024-09-18T15:15:14.835 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:15:14.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345951 2024-09-18T15:15:14.837 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 85899345951 2024-09-18T15:15:14.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345951' 2024-09-18T15:15:14.837 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:15:15.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345951 -lt 85899345951 2024-09-18T15:15:15.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:15:15.205 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182427 2024-09-18T15:15:15.205 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:15:15.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:15:15.207 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182427 2024-09-18T15:15:15.207 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:15:15.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182427 2024-09-18T15:15:15.208 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.4 seq 107374182427 2024-09-18T15:15:15.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182427' 2024-09-18T15:15:15.208 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:15:15.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182428 -lt 107374182427 2024-09-18T15:15:15.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:15:15.709 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018903 2024-09-18T15:15:15.709 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:15:15.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:15:15.711 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018903 2024-09-18T15:15:15.711 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:15:15.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018903 2024-09-18T15:15:15.712 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.5 seq 128849018903 2024-09-18T15:15:15.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018903' 2024-09-18T15:15:15.712 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:15:16.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018904 -lt 128849018903 2024-09-18T15:15:16.162 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:15:16.162 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:15:16.162 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:15:16.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:15:16.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:15:16.680 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:15:16.680 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:15:16.680 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:15:16.680 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:15:16.680 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:15:16.681 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:15:17.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:15:17.041 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:15:17.041 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:15:17.042 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:15:17.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:15:17.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-18T15:15:17.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T15:15:17.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-18T15:15:17.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:15:17.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:15:17.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:15:17.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:15:17.639 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:15:17.639 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:15:17.639 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:15:17.639 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:15:17.639 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:15:17.639 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:15:18.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:15:18.054 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:15:18.054 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:15:18.054 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:15:18.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:15:18.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:15:18.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:15:18.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:15:18.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:15:18.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:15:18.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:15:18.590 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:15:18.590 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:15:19.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-18T15:15:19.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-18T15:15:19.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:15:19.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:15:19.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:15:19.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:15:19.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:15:19.314 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:15:19.314 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:15:19.314 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:15:19.314 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:15:19.314 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:15:19.315 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:15:19.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:15:19.681 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:15:19.681 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:15:19.681 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:15:20.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:15:20.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:15:20.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:15:20.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:15:20.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:15:20.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:15:20.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:15:20.279 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:15:20.279 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:15:20.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14 2024-09-18T15:15:20.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14 '!=' null 2024-09-18T15:15:20.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:15:20.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:15:20.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:15:20.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:15:20.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:15:20.923 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:15:20.923 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:15:20.923 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:15:20.923 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:15:20.923 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:15:20.923 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:15:21.406 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:15:21.406 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:15:21.407 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:15:21.407 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:15:21.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:15:21.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:15:21.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:15:21.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:15:21.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:15:21.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:15:21.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:15:21.902 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:15:21.902 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:15:22.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11 2024-09-18T15:15:22.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11 '!=' null 2024-09-18T15:15:22.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:15:22.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:15:22.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:15:22.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:15:22.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:15:22.540 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:15:22.540 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:15:22.541 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:15:22.541 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:15:22.541 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:15:22.541 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:15:22.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:15:22.899 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:15:22.899 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:15:22.899 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:15:23.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:15:23.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:15:23.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:15:23.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:15:23.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:15:23.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:15:23.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:15:23.399 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:15:23.399 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:15:23.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18 2024-09-18T15:15:23.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18 '!=' null 2024-09-18T15:15:23.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:15:23.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:15:23.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:15:23.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:15:23.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:15:23.997 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:15:23.997 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:15:23.997 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:15:23.997 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:15:23.997 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:15:23.997 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:15:24.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:15:24.352 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:15:24.352 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:15:24.352 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:15:24.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:15:24.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:15:24.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:15:24.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:160: do_recovery_out1: check td/osd-recovery-stats 1.0 1 erasure 200 0 0 0 2024-09-18T15:15:24.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-09-18T15:15:24.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-09-18T15:15:24.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=1 2024-09-18T15:15:24.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=erasure 2024-09-18T15:15:24.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=200 2024-09-18T15:15:24.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-09-18T15:15:24.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=0 2024-09-18T15:15:24.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-09-18T15:15:24.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2024-09-18T15:15:24.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2024-09-18T15:15:24.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.1.log 2024-09-18T15:15:24.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-09-18T15:15:24.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' erasure = erasure ']' 2024-09-18T15:15:24.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:80: check: addp=p 2024-09-18T15:15:24.854 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-09-18T15:15:24.854 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.1.log 2024-09-18T15:15:24.854 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-09-18T15:15:24.854 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-09-18T15:15:24.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[1,3,2]/[1,NONE,2]' 2024-09-18T15:15:24.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-09-18T15:15:24.871 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-18T15:15:24.872 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-09-18T15:15:24.872 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [1,3,2]/[1,NONE,2]p' 2024-09-18T15:15:24.872 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-09-18T15:15:24.872 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-09-18T15:15:24.872 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-18T15:15:24.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=200 2024-09-18T15:15:24.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 200 200 2024-09-18T15:15:24.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-09-18T15:15:24.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-18T15:15:24.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-09-18T15:15:24.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-18T15:15:24.879 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-18T15:15:24.880 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-09-18T15:15:24.880 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-09-18T15:15:24.880 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-18T15:15:24.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-09-18T15:15:24.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-09-18T15:15:24.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-18T15:15:24.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-18T15:15:24.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-18T15:15:24.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-18T15:15:24.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-09-18T15:15:24.892 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-18T15:15:24.892 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-09-18T15:15:24.892 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [1,3,2]/[1,NONE,2]p' 2024-09-18T15:15:24.893 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-09-18T15:15:24.893 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-09-18T15:15:24.893 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-18T15:15:24.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=0 2024-09-18T15:15:24.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 0 0 2024-09-18T15:15:24.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=0 2024-09-18T15:15:24.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-18T15:15:24.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=0 2024-09-18T15:15:24.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-18T15:15:24.900 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-18T15:15:24.900 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-09-18T15:15:24.900 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-09-18T15:15:24.901 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-18T15:15:24.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-09-18T15:15:24.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-09-18T15:15:24.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-18T15:15:24.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-18T15:15:24.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-18T15:15:24.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-18T15:15:24.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2024-09-18T15:15:24.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:162: do_recovery_out1: delete_pool test 2024-09-18T15:15:24.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-18T15:15:24.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-18T15:15:25.451 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test' does not exist 2024-09-18T15:15:25.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:163: do_recovery_out1: kill_daemons td/osd-recovery-stats 2024-09-18T15:15:25.463 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:15:25.463 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:15:25.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:15:25.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:15:25.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:15:30.780 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:15:30.780 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-09-18T15:15:30.780 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-18T15:15:30.780 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:15:30.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-18T15:15:30.781 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:15:30.781 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:15:30.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:15:30.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:15:30.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:15:30.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:15:30.789 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:15:30.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:15:30.791 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:15:30.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:15:30.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:15:30.792 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:15:30.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:15:30.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:15:30.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:15:30.794 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:15:30.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:15:30.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:15:30.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-18T15:15:30.836 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:15:30.837 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:15:30.837 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:15:30.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.288083 2024-09-18T15:15:30.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:15:30.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:15:30.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-09-18T15:15:30.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-09-18T15:15:30.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-09-18T15:15:30.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-09-18T15:15:30.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-18T15:15:30.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:15:30.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-18T15:15:30.839 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:15:30.839 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:15:30.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:15:30.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:15:30.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:15:30.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:15:30.841 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:15:30.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:15:30.842 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:15:30.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:15:30.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:15:30.843 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:15:30.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:15:30.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:15:30.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:15:30.845 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:15:30.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:15:30.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:15:30.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-18T15:15:30.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:15:30.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:15:30.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:15:30.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.288083 2024-09-18T15:15:30.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:15:30.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:15:30.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-09-18T15:15:30.851 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T15:15:30.851 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:15:30.851 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:15:30.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.288083 2024-09-18T15:15:30.853 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T15:15:30.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T15:15:30.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T15:15:30.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-09-18T15:15:30.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_erasure_remapped td/osd-recovery-stats 2024-09-18T15:15:30.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:384: TEST_recovery_erasure_remapped: local dir=td/osd-recovery-stats 2024-09-18T15:15:30.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:386: TEST_recovery_erasure_remapped: run_mon td/osd-recovery-stats a 2024-09-18T15:15:30.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-09-18T15:15:30.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T15:15:30.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T15:15:30.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T15:15:30.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-09-18T15:15:30.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-09-18T15:15:31.224 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T15:15:31.224 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:15:31.224 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:15:31.224 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:15:31.224 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:15:31.224 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:15:31.224 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:15:31.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T15:15:31.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T15:15:31.525 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T15:15:31.526 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:15:31.526 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:15:31.526 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T15:15:31.526 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T15:15:31.526 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:15:31.527 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:15:31.527 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:15:31.527 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:15:31.527 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:15:31.527 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:15:31.528 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.288083/ceph-mon.a.asok 2024-09-18T15:15:31.528 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:15:31.528 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.288083/ceph-mon.a.asok config get fsid 2024-09-18T15:15:31.616 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T15:15:31.616 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:15:31.616 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:15:31.616 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T15:15:31.616 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T15:15:31.616 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:15:31.616 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:15:31.616 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:15:31.616 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:15:31.616 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:15:31.616 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:15:31.617 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.288083/ceph-mon.a.asok 2024-09-18T15:15:31.617 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:15:31.617 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.288083/ceph-mon.a.asok config get mon_host 2024-09-18T15:15:31.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:387: TEST_recovery_erasure_remapped: run_mgr td/osd-recovery-stats x 2024-09-18T15:15:31.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-09-18T15:15:31.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T15:15:31.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T15:15:31.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T15:15:31.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-09-18T15:15:31.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T15:15:32.538 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T15:15:32.538 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:15:32.538 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:15:32.538 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:15:32.538 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:15:32.538 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:15:32.538 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:15:32.539 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:15:32.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:15:32.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:388: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 0 2024-09-18T15:15:32.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:15:32.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:15:32.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T15:15:32.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:15:32.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-09-18T15:15:32.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:15:32.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:15:32.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:15:32.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:15:32.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-09-18T15:15:32.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-09-18T15:15:32.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:15:32.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:15:32.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:15:32.566 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:15:32.566 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:15:32.566 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:15:32.567 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:15:32.567 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:15:32.567 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:15:32.567 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:15:32.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:15:32.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:15:32.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:15:32.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:15:32.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:15:32.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:15:32.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:15:32.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:15:32.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:15:32.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:15:32.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:15:32.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:15:32.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-09-18T15:15:32.568 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:15:32.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2dff520d-eba2-440c-97c5-79ad13f74b79 2024-09-18T15:15:32.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2dff520d-eba2-440c-97c5-79ad13f74b79' 2024-09-18T15:15:32.569 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 2dff520d-eba2-440c-97c5-79ad13f74b79 2024-09-18T15:15:32.569 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:15:32.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCU7upmHAWTIhAAdf9HX1zjdAmc8pv2EUfPig== 2024-09-18T15:15:32.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCU7upmHAWTIhAAdf9HX1zjdAmc8pv2EUfPig=="}' 2024-09-18T15:15:32.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2dff520d-eba2-440c-97c5-79ad13f74b79 -i td/osd-recovery-stats/0/new.json 2024-09-18T15:15:32.897 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:15:32.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-09-18T15:15:32.907 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCU7upmHAWTIhAAdf9HX1zjdAmc8pv2EUfPig== --osd-uuid 2dff520d-eba2-440c-97c5-79ad13f74b79 2024-09-18T15:15:32.932 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:15:32.931+0000 7f63c163b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:15:32.950 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:15:32.949+0000 7f63c163b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:15:32.951 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:15:32.950+0000 7f63c163b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:15:32.952 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:15:32.950+0000 7f63c163b440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-18T15:15:32.952 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:15:32.950+0000 7f63c163b440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-18T15:15:32.952 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:15:32.950+0000 7f63c163b440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-18T15:15:32.982 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:15:32.981+0000 7f63c163b440 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-09-18T15:15:38.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-09-18T15:15:38.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:15:38.429 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T15:15:38.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T15:15:38.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:15:38.944 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T15:15:38.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T15:15:38.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:15:38.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:15:38.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:15:38.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:15:38.967 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:15:38.966+0000 7fa840614440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:15:38.989 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:15:38.987+0000 7fa840614440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:15:38.990 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:15:38.989+0000 7fa840614440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:15:39.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T15:15:39.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:15:39.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:15:39.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:15:39.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:15:39.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:15:39.386 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:15:39.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:15:39.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:15:39.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:15:39.524 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:15:39.523+0000 7fa840614440 -1 Falling back to public interface 2024-09-18T15:15:39.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:15:40.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:15:40.796 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:15:40.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:15:40.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:15:40.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:15:40.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:15:41.212 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:15:42.002 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:15:42.000+0000 7fa840614440 -1 osd.0 0 log_to_monitors true 2024-09-18T15:15:42.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:15:42.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:15:42.214 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:15:42.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:15:42.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:15:42.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:15:42.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:15:43.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:15:43.711 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:15:43.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:15:43.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:15:43.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:15:43.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:15:44.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:15:45.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:15:45.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:15:45.091 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:15:45.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:15:45.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:15:45.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:15:45.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:15:46.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:15:46.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:15:46.489 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:15:46.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:15:46.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:15:46.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:15:46.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:15:47.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:15:47.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:15:47.894 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:15:47.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:15:47.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:15:47.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:15:48.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:15:49.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:15:49.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:15:49.308 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:15:49.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:15:49.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:15:49.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:15:49.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:15:50.714 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:15:50.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:15:50.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:15:50.715 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:15:50.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:15:50.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:15:51.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:15:52.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:15:52.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:15:52.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:15:52.110 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:15:52.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:15:52.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:15:52.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:15:53.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:15:53.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:15:53.490 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:15:53.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:15:53.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:15:53.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:15:54.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:15:55.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:15:55.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:15:55.330 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:15:55.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:15:55.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:15:55.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:15:55.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:15:56.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:15:56.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:15:56.798 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:15:56.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:15:56.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:15:56.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:15:57.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:15:58.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:15:58.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:15:58.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:15:58.354 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:15:58.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:15:58.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:15:59.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:00.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:00.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:00.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:16:00.473 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:16:00.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:00.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:16:01.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:02.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:02.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:02.168 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:16:02.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:16:02.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:02.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:16:02.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:03.946 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:03.946 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:03.946 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:16:03.947 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:16:03.947 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:03.947 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:16:04.395 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/305633,v1:127.0.0.1:6803/305633] [v2:127.0.0.1:6804/305633,v1:127.0.0.1:6805/305633] exists,up 2dff520d-eba2-440c-97c5-79ad13f74b79 2024-09-18T15:16:04.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:16:04.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:16:04.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:16:04.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:389: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 1 2024-09-18T15:16:04.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:16:04.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:16:04.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T15:16:04.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:16:04.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-09-18T15:16:04.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:16:04.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:16:04.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:16:04.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:16:04.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-09-18T15:16:04.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-09-18T15:16:04.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:16:04.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:16:04.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:16:04.400 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:16:04.400 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:16:04.400 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:16:04.400 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:16:04.400 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:16:04.401 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:16:04.401 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:16:04.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:16:04.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:16:04.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:16:04.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:16:04.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:16:04.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:16:04.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:16:04.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:16:04.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:16:04.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:16:04.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:16:04.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:16:04.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-09-18T15:16:04.402 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:16:04.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d43d62d8-47c3-4e06-a763-d754f61cca9c 2024-09-18T15:16:04.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d43d62d8-47c3-4e06-a763-d754f61cca9c' 2024-09-18T15:16:04.402 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 d43d62d8-47c3-4e06-a763-d754f61cca9c 2024-09-18T15:16:04.402 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:16:04.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC07upm2a+UGBAAYfvk63Oe6hm7h7crflC+wg== 2024-09-18T15:16:04.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC07upm2a+UGBAAYfvk63Oe6hm7h7crflC+wg=="}' 2024-09-18T15:16:04.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d43d62d8-47c3-4e06-a763-d754f61cca9c -i td/osd-recovery-stats/1/new.json 2024-09-18T15:16:04.993 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:16:05.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-09-18T15:16:05.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC07upm2a+UGBAAYfvk63Oe6hm7h7crflC+wg== --osd-uuid d43d62d8-47c3-4e06-a763-d754f61cca9c 2024-09-18T15:16:05.029 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:16:05.028+0000 7ff5b6a0f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:05.031 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:16:05.030+0000 7ff5b6a0f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:05.032 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:16:05.031+0000 7ff5b6a0f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:05.033 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:16:05.031+0000 7ff5b6a0f440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-18T15:16:05.033 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:16:05.031+0000 7ff5b6a0f440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-18T15:16:05.033 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:16:05.031+0000 7ff5b6a0f440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-18T15:16:05.109 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:16:05.107+0000 7ff5b6a0f440 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-09-18T15:16:10.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-09-18T15:16:10.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:16:10.709 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T15:16:10.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T15:16:10.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:16:11.217 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T15:16:11.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T15:16:11.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:16:11.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:16:11.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:16:11.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:16:11.240 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:16:11.239+0000 7f9cead2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:11.242 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:16:11.241+0000 7f9cead2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:11.243 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:16:11.242+0000 7f9cead2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:11.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T15:16:11.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:16:11.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:16:11.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:16:11.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:16:11.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:11.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:16:11.586 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:16:11.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:11.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:16:11.757 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:16:11.756+0000 7f9cead2d440 -1 Falling back to public interface 2024-09-18T15:16:11.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:12.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:12.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:12.997 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:16:12.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:16:12.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:12.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:16:13.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:14.295 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:16:14.293+0000 7f9cead2d440 -1 osd.1 0 log_to_monitors true 2024-09-18T15:16:14.406 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:14.406 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:14.406 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:16:14.406 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:16:14.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:14.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:16:14.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:15.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:15.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:15.871 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:16:15.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:16:15.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:15.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:16:16.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:17.343 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:17.343 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:17.343 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:16:17.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:16:17.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:17.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:16:17.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:18.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:18.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:18.719 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:16:18.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:16:18.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:18.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:16:19.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:20.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:20.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:20.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:16:20.242 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:16:20.242 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:20.242 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:16:20.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:21.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:21.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:21.688 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:16:21.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:16:21.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:21.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:16:22.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:23.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:23.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:23.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:16:23.153 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:16:23.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:23.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:16:23.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:24.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:24.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:24.575 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:16:24.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:16:24.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:24.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:16:24.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:25.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:25.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:25.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:16:25.995 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:16:25.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:25.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:16:26.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:27.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:27.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:27.829 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:16:27.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:16:27.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:27.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:16:28.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:29.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:29.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:29.540 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:16:29.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:16:29.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:29.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:16:30.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:31.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:31.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:31.092 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:16:31.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:16:31.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:31.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:16:32.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:33.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:33.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:33.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:16:33.184 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:16:33.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:33.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:16:33.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:34.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:34.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:34.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:16:34.900 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:16:34.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:34.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:16:35.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:36.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:36.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:36.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:16:36.536 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:16:36.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:36.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:16:36.954 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/307362,v1:127.0.0.1:6811/307362] [v2:127.0.0.1:6812/307362,v1:127.0.0.1:6813/307362] exists,up d43d62d8-47c3-4e06-a763-d754f61cca9c 2024-09-18T15:16:36.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:16:36.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:16:36.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:16:36.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:390: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 2 2024-09-18T15:16:36.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:16:36.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:16:36.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T15:16:36.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:16:36.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-09-18T15:16:36.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:16:36.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:16:36.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:16:36.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:16:36.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-09-18T15:16:36.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-09-18T15:16:36.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:16:36.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:16:36.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:16:36.959 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:16:36.959 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:16:36.959 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:16:36.959 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:16:36.959 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:16:36.960 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:16:36.960 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:16:36.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:16:36.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:16:36.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:16:36.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:16:36.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:16:36.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:16:36.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:16:36.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:16:36.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:16:36.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:16:36.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:16:36.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:16:36.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-09-18T15:16:36.961 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:16:36.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=66bcacb3-cb00-43fa-9c21-5b95dad40269 2024-09-18T15:16:36.961 INFO:tasks.workunit.client.0.smithi047.stdout:add osd2 66bcacb3-cb00-43fa-9c21-5b95dad40269 2024-09-18T15:16:36.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 66bcacb3-cb00-43fa-9c21-5b95dad40269' 2024-09-18T15:16:36.962 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:16:36.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDU7upm8pLvORAA5vSWtyfytopit9Bt7CjA/A== 2024-09-18T15:16:36.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDU7upm8pLvORAA5vSWtyfytopit9Bt7CjA/A=="}' 2024-09-18T15:16:36.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 66bcacb3-cb00-43fa-9c21-5b95dad40269 -i td/osd-recovery-stats/2/new.json 2024-09-18T15:16:37.638 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:16:37.648 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-09-18T15:16:37.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDU7upm8pLvORAA5vSWtyfytopit9Bt7CjA/A== --osd-uuid 66bcacb3-cb00-43fa-9c21-5b95dad40269 2024-09-18T15:16:37.674 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:16:37.673+0000 7f1df4e22440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:37.676 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:16:37.675+0000 7f1df4e22440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:37.677 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:16:37.676+0000 7f1df4e22440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:37.678 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:16:37.676+0000 7f1df4e22440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-18T15:16:37.678 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:16:37.676+0000 7f1df4e22440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-18T15:16:37.678 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:16:37.676+0000 7f1df4e22440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-18T15:16:37.720 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:16:37.719+0000 7f1df4e22440 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-09-18T15:16:43.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-09-18T15:16:43.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:16:43.933 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd2 key to auth repository 2024-09-18T15:16:43.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T15:16:43.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:16:44.444 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T15:16:44.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T15:16:44.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:16:44.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:16:44.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:16:44.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:16:44.467 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:16:44.466+0000 7f667e723440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:44.493 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:16:44.491+0000 7f667e723440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:44.494 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:16:44.492+0000 7f667e723440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:16:44.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T15:16:44.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:16:44.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:16:44.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:16:44.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:16:44.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:44.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:16:44.867 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:16:44.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:44.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:16:45.090 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:16:45.025+0000 7f667e723440 -1 Falling back to public interface 2024-09-18T15:16:45.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:46.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:46.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:46.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:16:46.284 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:16:46.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:46.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:16:46.660 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:47.497 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:16:47.495+0000 7f667e723440 -1 osd.2 0 log_to_monitors true 2024-09-18T15:16:47.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:47.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:47.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:16:47.661 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:16:47.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:47.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:16:48.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:49.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:49.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:49.167 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:16:49.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:16:49.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:49.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:16:49.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:50.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:50.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:50.731 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:16:50.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:16:50.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:50.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:16:51.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:52.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:52.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:52.296 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:16:52.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:16:52.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:52.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:16:52.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:53.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:53.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:53.885 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:16:53.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:16:53.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:53.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:16:54.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:55.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:55.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:55.303 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:16:55.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:16:55.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:55.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:16:55.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:56.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:56.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:56.720 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:16:56.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:16:56.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:56.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:16:57.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:58.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:58.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:58.270 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:16:58.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:16:58.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:58.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:16:58.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:16:59.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:16:59.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:16:59.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:16:59.657 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:16:59.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:16:59.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:17:00.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:17:01.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:17:01.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:01.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:17:01.229 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:17:01.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:01.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:17:01.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:17:02.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:17:02.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:02.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:17:02.964 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:17:02.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:02.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:17:03.861 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:17:04.862 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:17:04.862 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:04.862 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:17:04.863 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:17:04.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:04.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:17:05.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:17:06.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:17:06.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:06.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:17:06.615 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:17:06.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:06.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:17:07.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:17:08.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:17:08.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:08.370 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:17:08.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:17:08.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:08.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:17:08.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:17:09.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:17:09.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:09.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:17:09.790 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:17:09.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:09.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:17:10.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:17:11.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:17:11.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:11.298 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T15:17:11.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:17:11.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:11.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:17:11.676 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/309091,v1:127.0.0.1:6819/309091] [v2:127.0.0.1:6820/309091,v1:127.0.0.1:6821/309091] exists,up 66bcacb3-cb00-43fa-9c21-5b95dad40269 2024-09-18T15:17:11.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:17:11.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:17:11.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:17:11.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:391: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 3 2024-09-18T15:17:11.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:17:11.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:17:11.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-18T15:17:11.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:17:11.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-09-18T15:17:11.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:17:11.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:17:11.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:17:11.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:17:11.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-09-18T15:17:11.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-09-18T15:17:11.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:17:11.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:17:11.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:17:11.681 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:17:11.681 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:17:11.681 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:17:11.681 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:17:11.681 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:17:11.681 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:17:11.682 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:17:11.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:17:11.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:17:11.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:17:11.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:17:11.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:17:11.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:17:11.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:17:11.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:17:11.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:17:11.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:17:11.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:17:11.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:17:11.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-09-18T15:17:11.683 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:17:11.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9d206a72-b970-405d-86d6-8f5273257a6f 2024-09-18T15:17:11.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 9d206a72-b970-405d-86d6-8f5273257a6f' 2024-09-18T15:17:11.683 INFO:tasks.workunit.client.0.smithi047.stdout:add osd3 9d206a72-b970-405d-86d6-8f5273257a6f 2024-09-18T15:17:11.683 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:17:11.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD37upmVZxQKRAAQzvrKeZ4N5OErVUkVGdrwA== 2024-09-18T15:17:11.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD37upmVZxQKRAAQzvrKeZ4N5OErVUkVGdrwA=="}' 2024-09-18T15:17:11.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9d206a72-b970-405d-86d6-8f5273257a6f -i td/osd-recovery-stats/3/new.json 2024-09-18T15:17:12.176 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:17:12.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-09-18T15:17:12.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD37upmVZxQKRAAQzvrKeZ4N5OErVUkVGdrwA== --osd-uuid 9d206a72-b970-405d-86d6-8f5273257a6f 2024-09-18T15:17:12.211 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:17:12.210+0000 7fb33abfe440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:12.213 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:17:12.212+0000 7fb33abfe440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:12.214 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:17:12.213+0000 7fb33abfe440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:12.214 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:17:12.213+0000 7fb33abfe440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-18T15:17:12.214 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:17:12.213+0000 7fb33abfe440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-18T15:17:12.215 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:17:12.213+0000 7fb33abfe440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-18T15:17:12.291 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:17:12.290+0000 7fb33abfe440 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-09-18T15:17:18.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-09-18T15:17:18.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:17:18.143 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd3 key to auth repository 2024-09-18T15:17:18.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-18T15:17:18.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:17:18.670 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.3 2024-09-18T15:17:18.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-18T15:17:18.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:17:18.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:17:18.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:17:18.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:17:18.693 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:17:18.691+0000 7faef0d22440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:18.715 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:17:18.714+0000 7faef0d22440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:18.716 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:17:18.715+0000 7faef0d22440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:17:19.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-18T15:17:19.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:17:19.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T15:17:19.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:17:19.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:17:19.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:19.113 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:17:19.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:17:19.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:19.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:17:19.242 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:17:19.241+0000 7faef0d22440 -1 Falling back to public interface 2024-09-18T15:17:19.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:17:20.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:17:20.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:20.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:17:20.540 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:17:20.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:20.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:17:20.956 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:17:21.712 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:17:21.711+0000 7faef0d22440 -1 osd.3 0 log_to_monitors true 2024-09-18T15:17:21.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:17:21.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:21.961 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:17:21.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:17:21.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:21.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:17:22.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:17:23.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:17:23.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:23.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:17:23.447 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:17:23.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:23.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:17:23.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:17:24.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:17:24.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:24.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:17:24.823 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:17:24.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:24.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:17:25.343 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:17:26.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:17:26.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:26.345 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:17:26.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:17:26.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:26.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:17:26.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:17:27.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:17:27.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:27.964 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:17:27.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:17:27.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:27.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:17:28.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:17:29.343 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:17:29.343 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:29.343 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:17:29.343 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:17:29.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:29.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:17:29.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:17:30.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:17:30.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:30.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:17:30.821 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:17:30.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:30.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:17:31.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:17:32.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:17:32.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:32.223 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:17:32.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:17:32.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:32.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:17:32.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:17:33.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:17:33.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:33.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:17:33.696 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:17:33.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:33.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:17:34.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:17:35.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:17:35.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:35.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:17:35.134 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:17:35.135 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:35.135 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:17:36.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:17:37.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:17:37.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:37.115 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:17:37.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:17:37.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:37.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:17:38.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:17:39.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:17:39.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:39.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:17:39.041 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:17:39.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:39.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:17:40.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:17:41.155 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:17:41.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:41.156 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:17:41.157 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:17:41.157 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:41.157 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:17:41.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:17:42.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:17:42.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:42.859 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:17:42.860 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:17:42.860 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:42.861 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:17:43.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:17:44.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:17:44.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:17:44.537 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:17:44.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:17:44.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:17:44.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:17:44.981 INFO:tasks.workunit.client.0.smithi047.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/310846,v1:127.0.0.1:6827/310846] [v2:127.0.0.1:6828/310846,v1:127.0.0.1:6829/310846] exists,up 9d206a72-b970-405d-86d6-8f5273257a6f 2024-09-18T15:17:44.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:17:44.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:17:44.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:17:44.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:393: TEST_recovery_erasure_remapped: ceph osd erasure-code-profile set myprofile plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-18T15:17:45.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:394: TEST_recovery_erasure_remapped: create_pool test 1 1 erasure myprofile 2024-09-18T15:17:45.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 erasure myprofile 2024-09-18T15:17:46.311 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test' already exists 2024-09-18T15:17:46.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:17:47.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:395: TEST_recovery_erasure_remapped: ceph osd pool set test min_size 2 2024-09-18T15:17:48.382 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 1 min_size to 2 2024-09-18T15:17:48.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:397: TEST_recovery_erasure_remapped: wait_for_clean 2024-09-18T15:17:48.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:17:48.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:17:48.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:17:48.393 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:17:48.394 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:17:48.394 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:17:48.394 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:17:48.394 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:17:48.394 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:17:48.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:17:48.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:17:48.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:17:48.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:17:48.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:17:48.484 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:17:48.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:17:48.904 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:17:48.904 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T15:17:48.904 INFO:tasks.workunit.client.0.smithi047.stderr:3' 2024-09-18T15:17:48.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:17:48.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:17:48.904 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:17:49.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836505 2024-09-18T15:17:49.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836505 2024-09-18T15:17:49.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505' 2024-09-18T15:17:49.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:17:49.012 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:17:49.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672977 2024-09-18T15:17:49.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672977 2024-09-18T15:17:49.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505 1-42949672977' 2024-09-18T15:17:49.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:17:49.120 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:17:49.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509451 2024-09-18T15:17:49.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509451 2024-09-18T15:17:49.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505 1-42949672977 2-64424509451' 2024-09-18T15:17:49.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:17:49.229 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:17:49.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345925 2024-09-18T15:17:49.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345925 2024-09-18T15:17:49.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505 1-42949672977 2-64424509451 3-85899345925' 2024-09-18T15:17:49.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:17:49.337 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836505 2024-09-18T15:17:49.337 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:17:49.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:17:49.338 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836505 2024-09-18T15:17:49.338 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:17:49.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836505 2024-09-18T15:17:49.339 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836505 2024-09-18T15:17:49.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-09-18T15:17:49.340 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:17:49.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836503 -lt 21474836505 2024-09-18T15:17:49.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:17:50.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:17:50.749 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:17:51.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836506 -lt 21474836505 2024-09-18T15:17:51.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:17:51.128 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672977 2024-09-18T15:17:51.128 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:17:51.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:17:51.130 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672977 2024-09-18T15:17:51.130 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:17:51.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672977 2024-09-18T15:17:51.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-09-18T15:17:51.131 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672977 2024-09-18T15:17:51.131 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:17:51.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672977 -lt 42949672977 2024-09-18T15:17:51.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:17:51.504 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509451 2024-09-18T15:17:51.504 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:17:51.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:17:51.506 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509451 2024-09-18T15:17:51.506 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:17:51.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509451 2024-09-18T15:17:51.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-09-18T15:17:51.507 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509451 2024-09-18T15:17:51.507 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:17:51.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509451 -lt 64424509451 2024-09-18T15:17:51.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:17:51.873 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345925 2024-09-18T15:17:51.873 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:17:51.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:17:51.874 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345925 2024-09-18T15:17:51.874 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:17:51.875 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345925 2024-09-18T15:17:51.875 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345925' 2024-09-18T15:17:51.876 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 85899345925 2024-09-18T15:17:51.876 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:17:52.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345925 -lt 85899345925 2024-09-18T15:17:52.246 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:17:52.246 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:17:52.247 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:17:52.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:17:52.746 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:17:52.746 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:17:52.746 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:17:52.746 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:17:52.746 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:17:52.746 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:17:52.746 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:17:53.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:17:53.100 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:17:53.100 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:17:53.100 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:17:53.599 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:17:53.599 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:17:53.599 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:17:53.599 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: seq 1 200 2024-09-18T15:17:53.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:17:53.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj1 /dev/null 2024-09-18T15:17:53.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:17:53.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj2 /dev/null 2024-09-18T15:17:53.985 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:17:53.985 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj3 /dev/null 2024-09-18T15:17:54.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:17:54.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj4 /dev/null 2024-09-18T15:17:54.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:17:54.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj5 /dev/null 2024-09-18T15:17:54.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:17:54.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj6 /dev/null 2024-09-18T15:17:54.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:17:54.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj7 /dev/null 2024-09-18T15:17:54.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:17:54.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj8 /dev/null 2024-09-18T15:17:55.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:17:55.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj9 /dev/null 2024-09-18T15:17:55.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:17:55.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj10 /dev/null 2024-09-18T15:17:55.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:17:55.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj11 /dev/null 2024-09-18T15:17:55.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:17:55.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj12 /dev/null 2024-09-18T15:17:55.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:17:55.907 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj13 /dev/null 2024-09-18T15:17:56.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:17:56.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj14 /dev/null 2024-09-18T15:17:56.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:17:56.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj15 /dev/null 2024-09-18T15:17:56.469 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:17:56.469 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj16 /dev/null 2024-09-18T15:17:56.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:17:56.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj17 /dev/null 2024-09-18T15:17:56.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:17:56.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj18 /dev/null 2024-09-18T15:17:57.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:17:57.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj19 /dev/null 2024-09-18T15:17:57.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:17:57.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj20 /dev/null 2024-09-18T15:17:57.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:17:57.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj21 /dev/null 2024-09-18T15:17:57.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:17:57.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj22 /dev/null 2024-09-18T15:17:57.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:17:57.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj23 /dev/null 2024-09-18T15:17:58.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:17:58.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj24 /dev/null 2024-09-18T15:17:58.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:17:58.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj25 /dev/null 2024-09-18T15:17:58.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:17:58.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj26 /dev/null 2024-09-18T15:17:58.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:17:58.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj27 /dev/null 2024-09-18T15:17:58.860 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:17:58.860 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj28 /dev/null 2024-09-18T15:17:59.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:17:59.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj29 /dev/null 2024-09-18T15:17:59.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:17:59.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj30 /dev/null 2024-09-18T15:17:59.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:17:59.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj31 /dev/null 2024-09-18T15:17:59.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:17:59.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj32 /dev/null 2024-09-18T15:17:59.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:17:59.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj33 /dev/null 2024-09-18T15:17:59.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:17:59.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj34 /dev/null 2024-09-18T15:18:00.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:00.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj35 /dev/null 2024-09-18T15:18:00.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:00.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj36 /dev/null 2024-09-18T15:18:00.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:00.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj37 /dev/null 2024-09-18T15:18:00.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:00.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj38 /dev/null 2024-09-18T15:18:00.907 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:00.907 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj39 /dev/null 2024-09-18T15:18:01.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:01.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj40 /dev/null 2024-09-18T15:18:01.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:01.285 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj41 /dev/null 2024-09-18T15:18:01.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:01.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj42 /dev/null 2024-09-18T15:18:01.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:01.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj43 /dev/null 2024-09-18T15:18:01.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:01.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj44 /dev/null 2024-09-18T15:18:02.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:02.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj45 /dev/null 2024-09-18T15:18:02.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:02.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj46 /dev/null 2024-09-18T15:18:02.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:02.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj47 /dev/null 2024-09-18T15:18:02.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:02.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj48 /dev/null 2024-09-18T15:18:02.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:02.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj49 /dev/null 2024-09-18T15:18:03.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:03.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj50 /dev/null 2024-09-18T15:18:03.197 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:03.197 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj51 /dev/null 2024-09-18T15:18:03.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:03.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj52 /dev/null 2024-09-18T15:18:03.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:03.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj53 /dev/null 2024-09-18T15:18:03.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:03.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj54 /dev/null 2024-09-18T15:18:03.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:03.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj55 /dev/null 2024-09-18T15:18:04.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:04.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj56 /dev/null 2024-09-18T15:18:04.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:04.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj57 /dev/null 2024-09-18T15:18:04.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:04.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj58 /dev/null 2024-09-18T15:18:04.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:04.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj59 /dev/null 2024-09-18T15:18:04.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:04.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj60 /dev/null 2024-09-18T15:18:05.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:05.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj61 /dev/null 2024-09-18T15:18:05.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:05.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj62 /dev/null 2024-09-18T15:18:05.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:05.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj63 /dev/null 2024-09-18T15:18:05.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:05.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj64 /dev/null 2024-09-18T15:18:05.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:05.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj65 /dev/null 2024-09-18T15:18:06.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:06.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj66 /dev/null 2024-09-18T15:18:06.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:06.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj67 /dev/null 2024-09-18T15:18:06.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:06.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj68 /dev/null 2024-09-18T15:18:06.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:06.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj69 /dev/null 2024-09-18T15:18:06.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:06.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj70 /dev/null 2024-09-18T15:18:06.956 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:06.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj71 /dev/null 2024-09-18T15:18:07.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:07.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj72 /dev/null 2024-09-18T15:18:07.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:07.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj73 /dev/null 2024-09-18T15:18:07.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:07.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj74 /dev/null 2024-09-18T15:18:07.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:07.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj75 /dev/null 2024-09-18T15:18:07.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:07.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj76 /dev/null 2024-09-18T15:18:08.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:08.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj77 /dev/null 2024-09-18T15:18:08.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:08.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj78 /dev/null 2024-09-18T15:18:08.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:08.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj79 /dev/null 2024-09-18T15:18:08.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:08.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj80 /dev/null 2024-09-18T15:18:08.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:08.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj81 /dev/null 2024-09-18T15:18:09.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:09.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj82 /dev/null 2024-09-18T15:18:09.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:09.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj83 /dev/null 2024-09-18T15:18:09.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:09.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj84 /dev/null 2024-09-18T15:18:09.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:09.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj85 /dev/null 2024-09-18T15:18:09.992 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:09.992 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj86 /dev/null 2024-09-18T15:18:10.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:10.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj87 /dev/null 2024-09-18T15:18:10.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:10.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj88 /dev/null 2024-09-18T15:18:10.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:10.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj89 /dev/null 2024-09-18T15:18:10.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:10.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj90 /dev/null 2024-09-18T15:18:11.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:11.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj91 /dev/null 2024-09-18T15:18:11.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:11.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj92 /dev/null 2024-09-18T15:18:11.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:11.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj93 /dev/null 2024-09-18T15:18:11.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:11.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj94 /dev/null 2024-09-18T15:18:11.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:11.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj95 /dev/null 2024-09-18T15:18:12.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:12.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj96 /dev/null 2024-09-18T15:18:12.264 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:12.264 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj97 /dev/null 2024-09-18T15:18:12.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:12.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj98 /dev/null 2024-09-18T15:18:12.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:12.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj99 /dev/null 2024-09-18T15:18:12.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:12.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj100 /dev/null 2024-09-18T15:18:13.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:13.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj101 /dev/null 2024-09-18T15:18:13.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:13.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj102 /dev/null 2024-09-18T15:18:13.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:13.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj103 /dev/null 2024-09-18T15:18:13.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:13.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj104 /dev/null 2024-09-18T15:18:13.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:13.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj105 /dev/null 2024-09-18T15:18:13.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:13.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj106 /dev/null 2024-09-18T15:18:14.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:14.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj107 /dev/null 2024-09-18T15:18:14.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:14.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj108 /dev/null 2024-09-18T15:18:14.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:14.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj109 /dev/null 2024-09-18T15:18:14.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:14.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj110 /dev/null 2024-09-18T15:18:14.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:14.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj111 /dev/null 2024-09-18T15:18:15.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:15.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj112 /dev/null 2024-09-18T15:18:15.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:15.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj113 /dev/null 2024-09-18T15:18:15.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:15.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj114 /dev/null 2024-09-18T15:18:15.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:15.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj115 /dev/null 2024-09-18T15:18:15.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:15.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj116 /dev/null 2024-09-18T15:18:16.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:16.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj117 /dev/null 2024-09-18T15:18:16.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:16.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj118 /dev/null 2024-09-18T15:18:16.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:16.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj119 /dev/null 2024-09-18T15:18:16.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:16.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj120 /dev/null 2024-09-18T15:18:16.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:16.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj121 /dev/null 2024-09-18T15:18:17.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:17.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj122 /dev/null 2024-09-18T15:18:17.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:17.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj123 /dev/null 2024-09-18T15:18:17.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:17.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj124 /dev/null 2024-09-18T15:18:17.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:17.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj125 /dev/null 2024-09-18T15:18:17.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:17.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj126 /dev/null 2024-09-18T15:18:17.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:17.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj127 /dev/null 2024-09-18T15:18:18.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:18.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj128 /dev/null 2024-09-18T15:18:18.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:18.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj129 /dev/null 2024-09-18T15:18:18.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:18.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj130 /dev/null 2024-09-18T15:18:18.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:18.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj131 /dev/null 2024-09-18T15:18:19.052 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:19.052 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj132 /dev/null 2024-09-18T15:18:19.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:19.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj133 /dev/null 2024-09-18T15:18:19.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:19.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj134 /dev/null 2024-09-18T15:18:19.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:19.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj135 /dev/null 2024-09-18T15:18:19.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:19.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj136 /dev/null 2024-09-18T15:18:19.991 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:19.991 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj137 /dev/null 2024-09-18T15:18:20.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:20.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj138 /dev/null 2024-09-18T15:18:20.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:20.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj139 /dev/null 2024-09-18T15:18:20.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:20.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj140 /dev/null 2024-09-18T15:18:20.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:20.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj141 /dev/null 2024-09-18T15:18:20.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:20.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj142 /dev/null 2024-09-18T15:18:21.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:21.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj143 /dev/null 2024-09-18T15:18:21.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:21.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj144 /dev/null 2024-09-18T15:18:21.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:21.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj145 /dev/null 2024-09-18T15:18:22.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:22.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj146 /dev/null 2024-09-18T15:18:22.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:22.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj147 /dev/null 2024-09-18T15:18:22.406 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:22.406 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj148 /dev/null 2024-09-18T15:18:22.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:22.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj149 /dev/null 2024-09-18T15:18:22.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:22.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj150 /dev/null 2024-09-18T15:18:22.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:22.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj151 /dev/null 2024-09-18T15:18:23.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:23.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj152 /dev/null 2024-09-18T15:18:23.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:23.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj153 /dev/null 2024-09-18T15:18:23.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:23.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj154 /dev/null 2024-09-18T15:18:23.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:23.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj155 /dev/null 2024-09-18T15:18:23.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:23.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj156 /dev/null 2024-09-18T15:18:24.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:24.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj157 /dev/null 2024-09-18T15:18:24.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:24.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj158 /dev/null 2024-09-18T15:18:24.470 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:24.470 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj159 /dev/null 2024-09-18T15:18:24.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:24.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj160 /dev/null 2024-09-18T15:18:24.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:24.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj161 /dev/null 2024-09-18T15:18:25.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:25.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj162 /dev/null 2024-09-18T15:18:25.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:25.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj163 /dev/null 2024-09-18T15:18:25.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:25.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj164 /dev/null 2024-09-18T15:18:25.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:25.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj165 /dev/null 2024-09-18T15:18:25.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:25.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj166 /dev/null 2024-09-18T15:18:26.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:26.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj167 /dev/null 2024-09-18T15:18:26.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:26.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj168 /dev/null 2024-09-18T15:18:26.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:26.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj169 /dev/null 2024-09-18T15:18:26.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:26.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj170 /dev/null 2024-09-18T15:18:26.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:26.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj171 /dev/null 2024-09-18T15:18:27.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:27.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj172 /dev/null 2024-09-18T15:18:27.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:27.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj173 /dev/null 2024-09-18T15:18:27.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:27.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj174 /dev/null 2024-09-18T15:18:27.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:27.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj175 /dev/null 2024-09-18T15:18:27.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:27.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj176 /dev/null 2024-09-18T15:18:27.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:27.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj177 /dev/null 2024-09-18T15:18:28.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:28.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj178 /dev/null 2024-09-18T15:18:28.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:28.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj179 /dev/null 2024-09-18T15:18:28.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:28.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj180 /dev/null 2024-09-18T15:18:28.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:28.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj181 /dev/null 2024-09-18T15:18:28.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:28.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj182 /dev/null 2024-09-18T15:18:29.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:29.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj183 /dev/null 2024-09-18T15:18:29.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:29.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj184 /dev/null 2024-09-18T15:18:29.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:29.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj185 /dev/null 2024-09-18T15:18:29.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:29.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj186 /dev/null 2024-09-18T15:18:29.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:29.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj187 /dev/null 2024-09-18T15:18:30.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:30.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj188 /dev/null 2024-09-18T15:18:30.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:30.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj189 /dev/null 2024-09-18T15:18:30.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:30.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj190 /dev/null 2024-09-18T15:18:30.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:30.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj191 /dev/null 2024-09-18T15:18:30.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:30.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj192 /dev/null 2024-09-18T15:18:31.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:31.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj193 /dev/null 2024-09-18T15:18:31.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:31.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj194 /dev/null 2024-09-18T15:18:31.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:31.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj195 /dev/null 2024-09-18T15:18:31.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:31.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj196 /dev/null 2024-09-18T15:18:31.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:31.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj197 /dev/null 2024-09-18T15:18:32.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:32.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj198 /dev/null 2024-09-18T15:18:32.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:32.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj199 /dev/null 2024-09-18T15:18:32.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-18T15:18:32.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj200 /dev/null 2024-09-18T15:18:32.568 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:404: TEST_recovery_erasure_remapped: get_primary test obj1 2024-09-18T15:18:32.568 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-18T15:18:32.568 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T15:18:32.568 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-18T15:18:32.568 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:18:32.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:404: TEST_recovery_erasure_remapped: local primary=1 2024-09-18T15:18:32.974 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:405: TEST_recovery_erasure_remapped: get_pg test obj1 2024-09-18T15:18:32.974 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-18T15:18:32.974 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-09-18T15:18:32.975 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-09-18T15:18:32.975 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:18:33.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:405: TEST_recovery_erasure_remapped: local PG=1.0 2024-09-18T15:18:33.381 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:406: TEST_recovery_erasure_remapped: get_not_primary test obj1 2024-09-18T15:18:33.381 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-09-18T15:18:33.382 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-09-18T15:18:33.382 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-09-18T15:18:33.382 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-18T15:18:33.382 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T15:18:33.382 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-18T15:18:33.382 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:18:33.751 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-18T15:18:33.751 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-09-18T15:18:33.751 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-18T15:18:34.119 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:406: TEST_recovery_erasure_remapped: local otherosd=0 2024-09-18T15:18:34.119 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:408: TEST_recovery_erasure_remapped: ceph osd set norecover 2024-09-18T15:18:34.781 INFO:tasks.workunit.client.0.smithi047.stderr:norecover is set 2024-09-18T15:18:34.792 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:409: TEST_recovery_erasure_remapped: cat td/osd-recovery-stats/osd.0.pid 2024-09-18T15:18:34.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:409: TEST_recovery_erasure_remapped: kill 305633 2024-09-18T15:18:34.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:410: TEST_recovery_erasure_remapped: ceph osd down osd.0 2024-09-18T15:18:35.251 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already down. 2024-09-18T15:18:35.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:411: TEST_recovery_erasure_remapped: ceph osd out osd.0 2024-09-18T15:18:35.832 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already out. 2024-09-18T15:18:35.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:414: TEST_recovery_erasure_remapped: for i in 0 1 2 3 2024-09-18T15:18:35.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:416: TEST_recovery_erasure_remapped: '[' 0 = 1 ']' 2024-09-18T15:18:35.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:420: TEST_recovery_erasure_remapped: '[' 0 = 0 ']' 2024-09-18T15:18:35.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:422: TEST_recovery_erasure_remapped: continue 2024-09-18T15:18:35.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:414: TEST_recovery_erasure_remapped: for i in 0 1 2 3 2024-09-18T15:18:35.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:416: TEST_recovery_erasure_remapped: '[' 1 = 1 ']' 2024-09-18T15:18:35.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:418: TEST_recovery_erasure_remapped: continue 2024-09-18T15:18:35.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:414: TEST_recovery_erasure_remapped: for i in 0 1 2 3 2024-09-18T15:18:35.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:416: TEST_recovery_erasure_remapped: '[' 2 = 1 ']' 2024-09-18T15:18:35.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:420: TEST_recovery_erasure_remapped: '[' 2 = 0 ']' 2024-09-18T15:18:35.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:424: TEST_recovery_erasure_remapped: ceph osd out osd.2 2024-09-18T15:18:36.455 INFO:tasks.workunit.client.0.smithi047.stderr:osd.2 is already out. 2024-09-18T15:18:36.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:425: TEST_recovery_erasure_remapped: break 2024-09-18T15:18:36.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:427: TEST_recovery_erasure_remapped: ceph osd unset norecover 2024-09-18T15:18:37.386 INFO:tasks.workunit.client.0.smithi047.stderr:norecover is unset 2024-09-18T15:18:37.398 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:428: TEST_recovery_erasure_remapped: get_primary test obj1 2024-09-18T15:18:37.398 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-18T15:18:37.398 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T15:18:37.398 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-18T15:18:37.398 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:18:37.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:428: TEST_recovery_erasure_remapped: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-18T15:18:38.082 INFO:tasks.workunit.client.0.smithi047.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-18T15:18:38.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:429: TEST_recovery_erasure_remapped: sleep 2 2024-09-18T15:18:40.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:431: TEST_recovery_erasure_remapped: wait_for_clean 2024-09-18T15:18:40.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:18:40.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:18:40.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:18:40.092 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:18:40.093 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:18:40.093 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:18:40.093 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:18:40.093 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:18:40.093 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:18:40.194 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:18:40.194 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:18:40.194 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:18:40.194 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:18:40.194 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:18:40.194 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:18:40.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:18:40.565 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:18:40.565 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T15:18:40.565 INFO:tasks.workunit.client.0.smithi047.stderr:3' 2024-09-18T15:18:40.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:18:40.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:40.565 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:18:40.642 INFO:tasks.workunit.client.0.smithi047.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-18T15:18:40.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq= 2024-09-18T15:18:40.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z '' 2024-09-18T15:18:40.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: flush_pg_stats: continue 2024-09-18T15:18:40.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:40.644 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:18:40.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672989 2024-09-18T15:18:40.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672989 2024-09-18T15:18:40.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 1-42949672989' 2024-09-18T15:18:40.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:40.756 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:18:40.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509463 2024-09-18T15:18:40.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509463 2024-09-18T15:18:40.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 1-42949672989 2-64424509463' 2024-09-18T15:18:40.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:18:40.934 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:18:41.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345937 2024-09-18T15:18:41.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345937 2024-09-18T15:18:41.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 1-42949672989 2-64424509463 3-85899345937' 2024-09-18T15:18:41.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:41.043 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672989 2024-09-18T15:18:41.043 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:41.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:18:41.044 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672989 2024-09-18T15:18:41.044 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:41.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672989 2024-09-18T15:18:41.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2024-09-18T15:18:41.045 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672989 2024-09-18T15:18:41.045 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:18:41.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672987 -lt 42949672989 2024-09-18T15:18:41.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:18:42.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:18:42.465 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:18:42.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672990 -lt 42949672989 2024-09-18T15:18:42.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:42.872 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509463 2024-09-18T15:18:42.872 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:42.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:18:42.874 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509463 2024-09-18T15:18:42.874 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:42.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509463 2024-09-18T15:18:42.875 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509463 2024-09-18T15:18:42.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-09-18T15:18:42.876 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:18:43.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509464 -lt 64424509463 2024-09-18T15:18:43.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:18:43.244 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345937 2024-09-18T15:18:43.244 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:18:43.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:18:43.245 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345937 2024-09-18T15:18:43.246 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:18:43.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345937 2024-09-18T15:18:43.247 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 85899345937 2024-09-18T15:18:43.247 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345937' 2024-09-18T15:18:43.247 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:18:43.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345937 -lt 85899345937 2024-09-18T15:18:43.648 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:18:43.648 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:18:43.648 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:18:44.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:18:44.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:18:44.222 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:18:44.223 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:18:44.223 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:18:44.223 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:18:44.223 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:18:44.223 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:18:44.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:18:44.581 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:18:44.581 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:18:44.581 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:18:45.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:18:45.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-18T15:18:45.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T15:18:45.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-18T15:18:45.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:18:45.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:18:45.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:18:45.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:18:45.288 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:18:45.288 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:18:45.288 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:18:45.289 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:18:45.289 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:18:45.289 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:18:45.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:18:45.651 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:18:45.652 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:18:45.652 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:18:46.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:18:46.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:18:46.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:18:46.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:18:46.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:18:46.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:18:46.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:18:46.202 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:18:46.202 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:18:46.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=7 2024-09-18T15:18:46.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 7 '!=' null 2024-09-18T15:18:46.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:18:46.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:18:46.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:18:46.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:18:46.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:18:46.874 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:18:46.874 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:18:46.874 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:18:46.874 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:18:46.874 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:18:46.875 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:18:47.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:18:47.262 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:18:47.262 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:18:47.262 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:18:47.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:18:47.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:18:47.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:18:47.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:18:47.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:18:47.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:18:47.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:18:47.778 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:18:47.778 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:18:48.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=7 2024-09-18T15:18:48.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 7 '!=' null 2024-09-18T15:18:48.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:18:48.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:18:48.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:18:48.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:18:48.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:18:48.408 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:18:48.408 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:18:48.408 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:18:48.408 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:18:48.408 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:18:48.408 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:18:48.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:18:48.774 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:18:48.774 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:18:48.774 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:18:49.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:18:49.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:18:49.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:18:49.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:18:49.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:18:49.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:18:49.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:18:49.288 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:18:49.288 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:18:49.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14 2024-09-18T15:18:49.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14 '!=' null 2024-09-18T15:18:49.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:18:49.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:18:49.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:18:49.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:18:49.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:18:49.908 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:18:49.908 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:18:49.909 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:18:49.909 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:18:49.909 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:18:49.909 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:18:50.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:18:50.292 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:18:50.292 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:18:50.293 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:18:50.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:18:50.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:18:50.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:18:50.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:18:50.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:18:50.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:18:50.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:18:50.906 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:18:50.907 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:18:51.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15 2024-09-18T15:18:51.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15 '!=' null 2024-09-18T15:18:51.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:18:51.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:18:51.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:18:51.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:18:51.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:18:51.505 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:18:51.505 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:18:51.505 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:18:51.505 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:18:51.505 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:18:51.505 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:18:51.861 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:18:51.861 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:18:51.861 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:18:51.861 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:18:52.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:18:52.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:18:52.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:18:52.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:18:52.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:18:52.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:18:52.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:18:52.361 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:18:52.361 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:18:52.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16 2024-09-18T15:18:52.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16 '!=' null 2024-09-18T15:18:52.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:18:52.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:18:52.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:18:53.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:18:53.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:18:53.077 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:18:53.077 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:18:53.077 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:18:53.077 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:18:53.077 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:18:53.077 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:18:53.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:18:53.473 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:18:53.473 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:18:53.473 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:18:53.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:18:53.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:18:53.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:18:53.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:433: TEST_recovery_erasure_remapped: local log=td/osd-recovery-stats/osd.1.log 2024-09-18T15:18:53.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:434: TEST_recovery_erasure_remapped: check td/osd-recovery-stats 1.0 1 erasure 200 0 200 200 2024-09-18T15:18:53.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-09-18T15:18:53.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-09-18T15:18:53.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=1 2024-09-18T15:18:53.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=erasure 2024-09-18T15:18:53.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=200 2024-09-18T15:18:53.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-09-18T15:18:53.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=200 2024-09-18T15:18:53.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=200 2024-09-18T15:18:53.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2024-09-18T15:18:53.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2024-09-18T15:18:53.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.1.log 2024-09-18T15:18:53.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-09-18T15:18:53.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' erasure = erasure ']' 2024-09-18T15:18:53.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:80: check: addp=p 2024-09-18T15:18:53.976 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-09-18T15:18:53.976 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.1.log 2024-09-18T15:18:53.976 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-09-18T15:18:53.976 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-09-18T15:18:53.991 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[1,3,NONE]/[1,NONE,2]' 2024-09-18T15:18:53.992 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-09-18T15:18:53.992 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-18T15:18:53.992 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-09-18T15:18:53.993 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [1,3,NONE]/[1,NONE,2]p' 2024-09-18T15:18:53.993 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-09-18T15:18:53.993 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-09-18T15:18:53.993 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-18T15:18:53.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=200 2024-09-18T15:18:53.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 200 200 2024-09-18T15:18:53.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-09-18T15:18:54.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-18T15:18:54.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-09-18T15:18:54.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-18T15:18:54.000 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-18T15:18:54.000 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-09-18T15:18:54.000 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-09-18T15:18:54.000 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-18T15:18:54.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-09-18T15:18:54.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-09-18T15:18:54.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-18T15:18:54.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-18T15:18:54.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-18T15:18:54.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-18T15:18:54.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-09-18T15:18:54.012 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-18T15:18:54.012 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-09-18T15:18:54.012 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [1,3,NONE]/[1,NONE,2]p' 2024-09-18T15:18:54.012 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-09-18T15:18:54.012 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-09-18T15:18:54.012 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-18T15:18:54.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=200 2024-09-18T15:18:54.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 200 200 2024-09-18T15:18:54.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-09-18T15:18:54.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-18T15:18:54.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-09-18T15:18:54.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-18T15:18:54.019 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-18T15:18:54.019 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-09-18T15:18:54.019 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-09-18T15:18:54.020 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-18T15:18:54.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=200 2024-09-18T15:18:54.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 200 200 2024-09-18T15:18:54.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=200 2024-09-18T15:18:54.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-18T15:18:54.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=200 2024-09-18T15:18:54.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-18T15:18:54.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2024-09-18T15:18:54.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:436: TEST_recovery_erasure_remapped: delete_pool test 2024-09-18T15:18:54.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-18T15:18:54.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-18T15:18:54.519 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test' does not exist 2024-09-18T15:18:54.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:437: TEST_recovery_erasure_remapped: kill_daemons td/osd-recovery-stats 2024-09-18T15:18:54.530 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:18:54.530 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:18:54.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:18:54.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:18:54.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:18:59.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:18:59.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-09-18T15:18:59.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-18T15:18:59.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:18:59.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-18T15:18:59.846 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:18:59.846 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:18:59.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:18:59.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:18:59.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:18:59.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:18:59.853 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:18:59.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:18:59.854 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:18:59.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:18:59.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:18:59.856 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:18:59.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:18:59.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:18:59.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:18:59.857 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:18:59.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:18:59.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:18:59.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-18T15:18:59.891 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:18:59.891 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:18:59.891 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:18:59.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.288083 2024-09-18T15:18:59.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:18:59.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:18:59.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-09-18T15:18:59.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-09-18T15:18:59.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-09-18T15:18:59.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-09-18T15:18:59.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-18T15:18:59.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:18:59.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-18T15:18:59.894 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:18:59.894 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:18:59.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:18:59.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:18:59.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:18:59.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:18:59.895 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:18:59.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:18:59.896 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:18:59.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:18:59.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:18:59.898 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:18:59.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:18:59.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:18:59.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:18:59.899 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:18:59.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:18:59.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:18:59.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-18T15:18:59.903 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:18:59.903 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:18:59.903 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:18:59.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.288083 2024-09-18T15:18:59.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:18:59.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:18:59.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-09-18T15:18:59.905 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T15:18:59.905 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:18:59.906 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:18:59.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.288083 2024-09-18T15:18:59.908 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T15:18:59.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T15:18:59.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T15:18:59.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-09-18T15:18:59.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_multi td/osd-recovery-stats 2024-09-18T15:18:59.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:441: TEST_recovery_multi: local dir=td/osd-recovery-stats 2024-09-18T15:18:59.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:443: TEST_recovery_multi: local osds=6 2024-09-18T15:18:59.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:444: TEST_recovery_multi: run_mon td/osd-recovery-stats a 2024-09-18T15:18:59.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-09-18T15:18:59.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T15:18:59.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T15:18:59.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T15:18:59.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-09-18T15:18:59.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-09-18T15:19:00.246 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T15:19:00.246 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:19:00.246 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:19:00.246 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:19:00.246 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:19:00.246 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:19:00.246 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:19:00.247 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T15:19:00.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T15:19:00.481 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T15:19:00.481 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:19:00.481 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:19:00.482 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T15:19:00.482 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T15:19:00.483 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:19:00.483 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:19:00.483 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:19:00.483 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:19:00.483 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:19:00.483 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:19:00.483 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.288083/ceph-mon.a.asok 2024-09-18T15:19:00.484 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:19:00.484 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.288083/ceph-mon.a.asok config get fsid 2024-09-18T15:19:00.570 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T15:19:00.570 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:19:00.570 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:19:00.570 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T15:19:00.570 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T15:19:00.570 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:19:00.570 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:19:00.570 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:19:00.570 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:19:00.570 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:19:00.571 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:19:00.571 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.288083/ceph-mon.a.asok 2024-09-18T15:19:00.571 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:19:00.571 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.288083/ceph-mon.a.asok config get mon_host 2024-09-18T15:19:00.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:445: TEST_recovery_multi: run_mgr td/osd-recovery-stats x 2024-09-18T15:19:00.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-09-18T15:19:00.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T15:19:00.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T15:19:00.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T15:19:00.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-09-18T15:19:00.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T15:19:01.502 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T15:19:01.502 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:19:01.502 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:19:01.502 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:19:01.502 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:19:01.502 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:19:01.502 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:19:01.502 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:19:01.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:19:01.527 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: expr 6 - 1 2024-09-18T15:19:01.528 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: seq 0 5 2024-09-18T15:19:01.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-09-18T15:19:01.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 0 2024-09-18T15:19:01.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:19:01.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:19:01.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T15:19:01.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:19:01.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-09-18T15:19:01.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:19:01.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:19:01.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:19:01.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:19:01.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-09-18T15:19:01.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-09-18T15:19:01.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:19:01.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:19:01.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:19:01.532 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:19:01.532 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:19:01.532 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:19:01.532 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:19:01.533 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:19:01.533 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:19:01.533 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:19:01.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:19:01.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:19:01.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:19:01.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:19:01.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:19:01.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:19:01.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:19:01.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:19:01.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:19:01.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:19:01.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:19:01.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:19:01.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-09-18T15:19:01.534 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:19:01.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c33e7a09-4276-4cea-9133-77e0b0e8f9aa 2024-09-18T15:19:01.535 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 c33e7a09-4276-4cea-9133-77e0b0e8f9aa 2024-09-18T15:19:01.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c33e7a09-4276-4cea-9133-77e0b0e8f9aa' 2024-09-18T15:19:01.535 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:19:01.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBl7+pmoTSNIBAAaYpiee7eQOeAOx5Pezdn2Q== 2024-09-18T15:19:01.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBl7+pmoTSNIBAAaYpiee7eQOeAOx5Pezdn2Q=="}' 2024-09-18T15:19:01.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c33e7a09-4276-4cea-9133-77e0b0e8f9aa -i td/osd-recovery-stats/0/new.json 2024-09-18T15:19:01.860 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:19:01.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-09-18T15:19:01.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBl7+pmoTSNIBAAaYpiee7eQOeAOx5Pezdn2Q== --osd-uuid c33e7a09-4276-4cea-9133-77e0b0e8f9aa 2024-09-18T15:19:01.896 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:19:01.894+0000 7f5f9990f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:01.905 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:19:01.904+0000 7f5f9990f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:01.906 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:19:01.905+0000 7f5f9990f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:01.907 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:19:01.905+0000 7f5f9990f440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-18T15:19:01.907 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:19:01.905+0000 7f5f9990f440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-18T15:19:01.907 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:19:01.905+0000 7f5f9990f440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-18T15:19:01.945 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:19:01.944+0000 7f5f9990f440 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-09-18T15:19:07.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-09-18T15:19:07.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:19:07.304 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T15:19:07.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T15:19:07.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:19:07.818 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T15:19:07.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T15:19:07.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:19:07.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:19:07.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:19:07.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:19:07.841 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:19:07.839+0000 7fa8ce71f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:07.851 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:19:07.850+0000 7fa8ce71f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:07.852 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:19:07.851+0000 7fa8ce71f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:08.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T15:19:08.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:19:08.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:19:08.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:19:08.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:19:08.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:08.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:19:08.241 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:19:08.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:08.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:19:08.386 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:19:08.385+0000 7fa8ce71f440 -1 Falling back to public interface 2024-09-18T15:19:08.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:09.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:09.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:09.607 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:19:09.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:19:09.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:09.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:19:09.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:10.874 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:19:10.872+0000 7fa8ce71f440 -1 osd.0 0 log_to_monitors true 2024-09-18T15:19:10.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:10.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:10.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:19:10.983 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:19:10.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:10.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:19:11.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:12.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:12.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:12.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:19:12.558 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:19:12.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:12.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:19:12.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:13.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:13.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:13.936 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:19:13.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:19:13.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:13.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:19:14.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:15.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:15.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:15.352 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:19:15.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:19:15.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:15.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:19:15.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:16.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:16.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:16.866 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:19:16.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:19:16.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:16.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:19:17.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:18.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:18.297 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:19:18.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:18.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:19:18.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:18.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:19:18.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:19.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:19.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:19.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:19:19.744 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:19:19.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:19.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:19:20.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:21.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:21.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:21.119 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:19:21.119 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:19:21.119 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:21.119 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:19:21.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:22.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:22.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:22.606 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:19:22.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:19:22.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:22.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:19:23.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:24.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:24.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:24.293 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:19:24.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:19:24.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:24.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:19:24.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:25.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:25.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:25.890 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:19:25.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:19:25.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:25.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:19:26.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:27.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:27.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:27.489 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:19:27.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:19:27.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:27.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:19:28.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:29.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:29.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:29.193 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:19:29.194 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:19:29.194 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:29.194 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:19:29.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:30.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:30.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:30.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:19:30.631 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:19:30.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:30.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:19:31.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:32.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:32.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:32.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:19:32.305 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:19:32.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:32.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:19:32.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:33.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:33.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:33.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:19:33.814 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T15:19:33.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:33.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:19:34.201 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/318444,v1:127.0.0.1:6803/318444] [v2:127.0.0.1:6804/318444,v1:127.0.0.1:6805/318444] exists,up c33e7a09-4276-4cea-9133-77e0b0e8f9aa 2024-09-18T15:19:34.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:19:34.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:19:34.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:19:34.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-09-18T15:19:34.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 1 2024-09-18T15:19:34.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:19:34.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:19:34.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T15:19:34.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:19:34.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-09-18T15:19:34.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:19:34.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:19:34.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:19:34.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:19:34.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-09-18T15:19:34.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-09-18T15:19:34.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:19:34.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:19:34.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:19:34.207 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:19:34.207 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:19:34.207 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:19:34.207 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:19:34.207 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:19:34.207 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:19:34.207 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:19:34.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:19:34.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:19:34.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:19:34.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:19:34.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:19:34.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:19:34.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:19:34.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:19:34.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:19:34.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:19:34.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:19:34.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:19:34.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-09-18T15:19:34.208 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:19:34.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=352d8ac3-c62f-4781-a7bb-0f34888ce050 2024-09-18T15:19:34.209 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 352d8ac3-c62f-4781-a7bb-0f34888ce050 2024-09-18T15:19:34.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 352d8ac3-c62f-4781-a7bb-0f34888ce050' 2024-09-18T15:19:34.212 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:19:34.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCG7+pmyqcHDRAAse9kJfYqMfTFbpHBTZflXg== 2024-09-18T15:19:34.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCG7+pmyqcHDRAAse9kJfYqMfTFbpHBTZflXg=="}' 2024-09-18T15:19:34.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 352d8ac3-c62f-4781-a7bb-0f34888ce050 -i td/osd-recovery-stats/1/new.json 2024-09-18T15:19:34.694 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:19:34.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-09-18T15:19:34.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCG7+pmyqcHDRAAse9kJfYqMfTFbpHBTZflXg== --osd-uuid 352d8ac3-c62f-4781-a7bb-0f34888ce050 2024-09-18T15:19:34.730 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:19:34.729+0000 7f89b672b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:34.732 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:19:34.731+0000 7f89b672b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:34.733 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:19:34.732+0000 7f89b672b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:34.733 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:19:34.732+0000 7f89b672b440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-18T15:19:34.734 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:19:34.732+0000 7f89b672b440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-18T15:19:34.734 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:19:34.732+0000 7f89b672b440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-18T15:19:34.809 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:19:34.808+0000 7f89b672b440 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-09-18T15:19:40.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-09-18T15:19:40.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:19:40.758 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T15:19:40.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T15:19:40.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:19:41.262 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T15:19:41.264 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T15:19:41.264 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:19:41.264 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:19:41.264 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:19:41.264 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:19:41.285 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:19:41.284+0000 7f9c8e234440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:41.287 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:19:41.286+0000 7f9c8e234440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:41.288 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:19:41.287+0000 7f9c8e234440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:19:41.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T15:19:41.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:19:41.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:19:41.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:19:41.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:19:41.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:41.674 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:19:41.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:19:41.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:41.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:19:41.817 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:19:41.816+0000 7f9c8e234440 -1 Falling back to public interface 2024-09-18T15:19:42.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:43.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:43.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:43.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:19:43.036 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:19:43.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:43.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:19:43.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:44.355 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:19:44.354+0000 7f9c8e234440 -1 osd.1 0 log_to_monitors true 2024-09-18T15:19:44.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:44.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:44.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:19:44.454 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:19:44.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:44.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:19:45.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:46.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:46.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:46.044 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:19:46.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:19:46.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:46.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:19:46.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:47.483 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:47.483 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:47.483 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:19:47.483 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:19:47.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:47.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:19:47.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:48.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:48.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:48.924 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:19:48.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:19:48.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:48.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:19:49.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:50.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:50.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:50.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:19:50.433 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:19:50.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:50.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:19:50.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:51.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:51.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:51.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:19:51.857 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:19:51.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:51.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:19:52.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:53.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:53.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:53.379 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:19:53.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:19:53.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:53.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:19:53.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:54.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:54.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:54.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:19:54.821 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:19:54.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:54.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:19:55.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:56.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:56.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:56.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:19:56.292 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:19:56.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:56.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:19:56.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:57.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:57.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:57.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:19:57.696 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:19:57.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:57.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:19:58.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:19:59.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:19:59.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:19:59.373 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:19:59.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:19:59.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:19:59.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:19:59.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:00.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:00.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:00.872 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:20:00.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:20:00.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:00.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:20:01.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:02.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:02.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:02.638 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:20:02.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:20:02.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:02.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:20:03.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:04.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:04.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:04.397 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:20:04.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:20:04.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:04.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:20:05.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:06.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:06.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:06.423 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:20:06.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:20:06.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:06.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:20:06.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:07.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:07.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:07.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:20:07.976 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T15:20:07.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:07.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:20:08.441 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/320198,v1:127.0.0.1:6811/320198] [v2:127.0.0.1:6812/320198,v1:127.0.0.1:6813/320198] exists,up 352d8ac3-c62f-4781-a7bb-0f34888ce050 2024-09-18T15:20:08.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:20:08.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:20:08.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:20:08.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-09-18T15:20:08.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 2 2024-09-18T15:20:08.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:20:08.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:20:08.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T15:20:08.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:20:08.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-09-18T15:20:08.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:20:08.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:20:08.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:20:08.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:20:08.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-09-18T15:20:08.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-09-18T15:20:08.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:20:08.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:20:08.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:20:08.446 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:20:08.446 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:20:08.446 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:20:08.446 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:20:08.446 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:20:08.446 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:20:08.446 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:20:08.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:20:08.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:20:08.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:20:08.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:20:08.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:20:08.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:20:08.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:20:08.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:20:08.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:20:08.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:20:08.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:20:08.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:20:08.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-09-18T15:20:08.447 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:20:08.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f1478bca-0c74-4c65-9e83-c36352a9a513 2024-09-18T15:20:08.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f1478bca-0c74-4c65-9e83-c36352a9a513' 2024-09-18T15:20:08.447 INFO:tasks.workunit.client.0.smithi047.stdout:add osd2 f1478bca-0c74-4c65-9e83-c36352a9a513 2024-09-18T15:20:08.448 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:20:08.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCo7+pmsipLGxAARZ2rQqgTL13Es0sIDbXE+A== 2024-09-18T15:20:08.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCo7+pmsipLGxAARZ2rQqgTL13Es0sIDbXE+A=="}' 2024-09-18T15:20:08.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f1478bca-0c74-4c65-9e83-c36352a9a513 -i td/osd-recovery-stats/2/new.json 2024-09-18T15:20:08.950 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:20:08.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-09-18T15:20:08.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCo7+pmsipLGxAARZ2rQqgTL13Es0sIDbXE+A== --osd-uuid f1478bca-0c74-4c65-9e83-c36352a9a513 2024-09-18T15:20:08.987 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:20:08.985+0000 7faafa63b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:20:08.989 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:20:08.987+0000 7faafa63b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:20:08.990 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:20:08.988+0000 7faafa63b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:20:08.990 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:20:08.989+0000 7faafa63b440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-18T15:20:08.990 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:20:08.989+0000 7faafa63b440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-18T15:20:08.990 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:20:08.989+0000 7faafa63b440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-18T15:20:09.074 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:20:09.073+0000 7faafa63b440 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-09-18T15:20:14.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-09-18T15:20:14.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:20:14.922 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd2 key to auth repository 2024-09-18T15:20:14.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T15:20:14.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:20:15.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T15:20:15.448 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T15:20:15.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:20:15.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:20:15.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:20:15.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:20:15.471 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:20:15.469+0000 7f4777232440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:20:15.488 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:20:15.487+0000 7f4777232440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:20:15.489 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:20:15.488+0000 7f4777232440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:20:15.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T15:20:15.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:20:15.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:20:15.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:20:15.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:20:15.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:15.869 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:20:15.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:20:15.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:15.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:20:16.025 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:20:16.023+0000 7f4777232440 -1 Falling back to public interface 2024-09-18T15:20:16.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:17.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:17.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:17.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:20:17.279 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:20:17.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:17.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:20:17.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:18.527 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:20:18.525+0000 7f4777232440 -1 osd.2 0 log_to_monitors true 2024-09-18T15:20:18.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:18.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:18.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:20:18.698 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:20:18.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:18.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:20:19.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:20.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:20.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:20.205 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:20:20.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:20:20.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:20.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:20:20.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:21.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:21.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:21.635 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:20:21.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:20:21.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:21.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:20:22.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:23.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:23.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:23.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:20:23.108 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:20:23.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:23.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:20:23.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:24.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:24.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:24.597 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:20:24.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:20:24.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:24.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:20:25.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:26.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:26.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:26.005 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:20:26.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:20:26.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:26.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:20:26.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:27.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:27.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:27.525 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:20:27.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:20:27.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:27.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:20:27.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:28.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:28.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:28.965 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:20:28.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:20:28.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:28.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:20:29.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:30.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:30.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:30.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:20:30.405 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:20:30.406 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:30.406 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:20:30.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:31.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:31.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:31.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:20:31.785 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:20:31.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:31.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:20:32.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:33.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:33.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:33.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:20:33.605 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:20:33.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:33.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:20:34.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:35.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:35.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:35.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:20:35.727 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:20:35.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:35.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:20:36.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:37.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:37.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:37.531 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:20:37.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:20:37.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:37.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:20:38.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:39.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:39.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:39.463 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:20:39.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:20:39.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:39.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:20:40.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:41.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:41.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:41.454 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:20:41.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:20:41.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:41.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:20:41.947 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:42.948 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:42.949 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:42.949 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:20:42.949 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T15:20:42.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:42.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:20:43.391 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/321955,v1:127.0.0.1:6819/321955] [v2:127.0.0.1:6820/321955,v1:127.0.0.1:6821/321955] exists,up f1478bca-0c74-4c65-9e83-c36352a9a513 2024-09-18T15:20:43.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:20:43.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:20:43.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:20:43.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-09-18T15:20:43.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 3 2024-09-18T15:20:43.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:20:43.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:20:43.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-18T15:20:43.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:20:43.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-09-18T15:20:43.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:20:43.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:20:43.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:20:43.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:20:43.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-09-18T15:20:43.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-09-18T15:20:43.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:20:43.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:20:43.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:20:43.397 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:20:43.397 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:20:43.397 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:20:43.397 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:20:43.397 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:20:43.397 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:20:43.397 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:20:43.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:20:43.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:20:43.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:20:43.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:20:43.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:20:43.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:20:43.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:20:43.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:20:43.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:20:43.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:20:43.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:20:43.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:20:43.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-09-18T15:20:43.398 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:20:43.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3d704e38-d885-479c-b019-bef2441fe2f1 2024-09-18T15:20:43.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 3d704e38-d885-479c-b019-bef2441fe2f1' 2024-09-18T15:20:43.399 INFO:tasks.workunit.client.0.smithi047.stdout:add osd3 3d704e38-d885-479c-b019-bef2441fe2f1 2024-09-18T15:20:43.399 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:20:43.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDL7+pmIM9cGBAAjignX1QtcTzZQergn+3oxw== 2024-09-18T15:20:43.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDL7+pmIM9cGBAAjignX1QtcTzZQergn+3oxw=="}' 2024-09-18T15:20:43.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3d704e38-d885-479c-b019-bef2441fe2f1 -i td/osd-recovery-stats/3/new.json 2024-09-18T15:20:43.934 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:20:43.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-09-18T15:20:43.946 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDL7+pmIM9cGBAAjignX1QtcTzZQergn+3oxw== --osd-uuid 3d704e38-d885-479c-b019-bef2441fe2f1 2024-09-18T15:20:43.972 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:20:43.971+0000 7f0e26b1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:20:43.974 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:20:43.972+0000 7f0e26b1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:20:43.975 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:20:43.973+0000 7f0e26b1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:20:43.975 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:20:43.974+0000 7f0e26b1d440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-18T15:20:43.975 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:20:43.974+0000 7f0e26b1d440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-18T15:20:43.975 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:20:43.974+0000 7f0e26b1d440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-18T15:20:44.066 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:20:44.065+0000 7f0e26b1d440 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-09-18T15:20:50.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-09-18T15:20:50.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:20:50.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-18T15:20:50.222 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd3 key to auth repository 2024-09-18T15:20:50.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:20:50.735 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.3 2024-09-18T15:20:50.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-18T15:20:50.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:20:50.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:20:50.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:20:50.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:20:50.759 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:20:50.757+0000 7fc23301e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:20:50.761 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:20:50.759+0000 7fc23301e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:20:50.762 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:20:50.761+0000 7fc23301e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:20:51.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-18T15:20:51.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:20:51.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T15:20:51.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:20:51.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:20:51.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:51.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:20:51.107 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:20:51.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:51.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:20:51.280 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:20:51.279+0000 7fc23301e440 -1 Falling back to public interface 2024-09-18T15:20:51.477 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:52.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:52.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:52.479 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:20:52.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:20:52.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:52.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:20:52.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:53.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:53.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:53.857 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:20:53.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:20:53.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:53.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:20:53.919 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:20:53.918+0000 7fc23301e440 -1 osd.3 0 log_to_monitors true 2024-09-18T15:20:54.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:55.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:55.236 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:20:55.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:55.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:20:55.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:55.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:20:55.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:56.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:56.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:56.704 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:20:56.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:20:56.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:56.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:20:57.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:58.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:58.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:58.107 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:20:58.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:20:58.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:58.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:20:58.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:20:59.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:20:59.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:20:59.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:20:59.497 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:20:59.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:20:59.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:20:59.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:00.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:00.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:00.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:21:00.921 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:21:00.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:00.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:21:01.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:02.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:02.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:02.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:21:02.335 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:21:02.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:02.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:21:02.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:03.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:03.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:03.770 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:21:03.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:21:03.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:03.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:21:04.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:05.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:05.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:05.181 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:21:05.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:21:05.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:05.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:21:05.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:06.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:06.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:06.697 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:21:06.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:21:06.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:06.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:21:07.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:08.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:08.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:08.399 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:21:08.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:21:08.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:08.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:21:09.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:10.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:10.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:10.211 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:21:10.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:21:10.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:10.212 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:21:10.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:11.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:11.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:11.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:21:11.964 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:21:11.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:11.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:21:12.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:13.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:13.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:13.595 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:21:13.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:21:13.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:13.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:21:14.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:15.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:15.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:15.177 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:21:15.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:21:15.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:15.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:21:15.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:16.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:16.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:16.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:21:16.569 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T15:21:16.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:16.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:21:17.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:18.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:18.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:18.210 INFO:tasks.workunit.client.0.smithi047.stdout:18 2024-09-18T15:21:18.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:21:18.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:18.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:21:18.610 INFO:tasks.workunit.client.0.smithi047.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/323709,v1:127.0.0.1:6827/323709] [v2:127.0.0.1:6828/323709,v1:127.0.0.1:6829/323709] exists,up 3d704e38-d885-479c-b019-bef2441fe2f1 2024-09-18T15:21:18.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:21:18.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:21:18.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:21:18.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-09-18T15:21:18.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 4 2024-09-18T15:21:18.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:21:18.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:21:18.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-18T15:21:18.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:21:18.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2024-09-18T15:21:18.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:21:18.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:21:18.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:21:18.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:21:18.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2024-09-18T15:21:18.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2024-09-18T15:21:18.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:21:18.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:21:18.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:21:18.615 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:21:18.616 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:21:18.616 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:21:18.616 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:21:18.616 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:21:18.616 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:21:18.616 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:21:18.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:21:18.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:21:18.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:21:18.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:21:18.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:21:18.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:21:18.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:21:18.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:21:18.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:21:18.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:21:18.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:21:18.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:21:18.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2024-09-18T15:21:18.617 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:21:18.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6d3aad50-55ab-4643-b51a-60c559d3a6b2 2024-09-18T15:21:18.617 INFO:tasks.workunit.client.0.smithi047.stdout:add osd4 6d3aad50-55ab-4643-b51a-60c559d3a6b2 2024-09-18T15:21:18.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 6d3aad50-55ab-4643-b51a-60c559d3a6b2' 2024-09-18T15:21:18.618 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:21:18.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDu7+pmEEBmJRAAgvgkdwR+oyNB3gXdoxmH8g== 2024-09-18T15:21:18.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDu7+pmEEBmJRAAgvgkdwR+oyNB3gXdoxmH8g=="}' 2024-09-18T15:21:18.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6d3aad50-55ab-4643-b51a-60c559d3a6b2 -i td/osd-recovery-stats/4/new.json 2024-09-18T15:21:19.118 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:21:19.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2024-09-18T15:21:19.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDu7+pmEEBmJRAAgvgkdwR+oyNB3gXdoxmH8g== --osd-uuid 6d3aad50-55ab-4643-b51a-60c559d3a6b2 2024-09-18T15:21:19.154 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:21:19.153+0000 7f6026f21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:21:19.173 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:21:19.172+0000 7f6026f21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:21:19.174 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:21:19.173+0000 7f6026f21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:21:19.175 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:21:19.173+0000 7f6026f21440 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-18T15:21:19.175 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:21:19.173+0000 7f6026f21440 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-18T15:21:19.175 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:21:19.173+0000 7f6026f21440 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-18T15:21:19.213 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:21:19.212+0000 7f6026f21440 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2024-09-18T15:21:25.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2024-09-18T15:21:25.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:21:25.146 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd4 key to auth repository 2024-09-18T15:21:25.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-18T15:21:25.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:21:25.654 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.4 2024-09-18T15:21:25.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-18T15:21:25.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:21:25.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:21:25.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:21:25.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:21:25.677 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:21:25.676+0000 7f6c5511d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:21:25.679 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:21:25.678+0000 7f6c5511d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:21:25.681 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:21:25.679+0000 7f6c5511d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:21:26.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-18T15:21:26.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:21:26.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-18T15:21:26.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:21:26.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:21:26.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:26.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:21:26.061 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:21:26.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:26.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:21:26.204 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:21:26.203+0000 7f6c5511d440 -1 Falling back to public interface 2024-09-18T15:21:26.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:27.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:27.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:27.425 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:21:27.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:21:27.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:27.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:21:27.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:28.685 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:21:28.684+0000 7f6c5511d440 -1 osd.4 0 log_to_monitors true 2024-09-18T15:21:28.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:28.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:28.814 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:21:28.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:21:28.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:28.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:21:29.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:30.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:30.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:30.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:21:30.220 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:21:30.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:30.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:21:30.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:31.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:31.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:31.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:21:31.659 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:21:31.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:31.660 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:21:32.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:33.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:33.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:33.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:21:33.117 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:21:33.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:33.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:21:33.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:34.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:34.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:34.643 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:21:34.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:21:34.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:34.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:21:35.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:36.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:36.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:36.046 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:21:36.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:21:36.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:36.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:21:36.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:37.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:37.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:37.530 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:21:37.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:21:37.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:37.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:21:38.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:39.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:39.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:39.044 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:21:39.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:21:39.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:39.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:21:39.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:40.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:40.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:40.558 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:21:40.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:21:40.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:40.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:21:41.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:42.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:42.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:42.122 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:21:42.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:21:42.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:42.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:21:42.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:43.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:43.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:43.789 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:21:43.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:21:43.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:43.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:21:45.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:46.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:46.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:46.165 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:21:46.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:21:46.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:46.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:21:46.907 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:47.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:47.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:47.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:21:47.909 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:21:47.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:47.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:21:48.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:49.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:49.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:49.645 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:21:49.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:21:49.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:49.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:21:50.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:51.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:51.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:51.426 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:21:51.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:21:51.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:51.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:21:51.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:21:52.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:21:52.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:21:52.934 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T15:21:52.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:21:52.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:21:52.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:21:53.379 INFO:tasks.workunit.client.0.smithi047.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/325488,v1:127.0.0.1:6835/325488] [v2:127.0.0.1:6836/325488,v1:127.0.0.1:6837/325488] exists,up 6d3aad50-55ab-4643-b51a-60c559d3a6b2 2024-09-18T15:21:53.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:21:53.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:21:53.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:21:53.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-09-18T15:21:53.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 5 2024-09-18T15:21:53.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:21:53.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:21:53.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-18T15:21:53.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:21:53.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2024-09-18T15:21:53.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:21:53.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:21:53.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:21:53.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:21:53.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2024-09-18T15:21:53.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2024-09-18T15:21:53.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:21:53.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:21:53.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:21:53.385 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:21:53.385 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:21:53.385 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:21:53.385 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:21:53.385 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:21:53.385 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:21:53.385 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:21:53.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:21:53.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:21:53.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:21:53.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:21:53.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:21:53.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:21:53.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:21:53.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:21:53.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:21:53.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:21:53.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:21:53.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:21:53.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2024-09-18T15:21:53.386 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:21:53.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8b078dcf-5209-400e-9f28-22e240d9595c 2024-09-18T15:21:53.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 8b078dcf-5209-400e-9f28-22e240d9595c' 2024-09-18T15:21:53.387 INFO:tasks.workunit.client.0.smithi047.stdout:add osd5 8b078dcf-5209-400e-9f28-22e240d9595c 2024-09-18T15:21:53.387 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:21:53.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAR8OpmXLCiFxAAk8kr/845uakq6jD5oHKtew== 2024-09-18T15:21:53.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAR8OpmXLCiFxAAk8kr/845uakq6jD5oHKtew=="}' 2024-09-18T15:21:53.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8b078dcf-5209-400e-9f28-22e240d9595c -i td/osd-recovery-stats/5/new.json 2024-09-18T15:21:53.965 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:21:53.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2024-09-18T15:21:53.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAR8OpmXLCiFxAAk8kr/845uakq6jD5oHKtew== --osd-uuid 8b078dcf-5209-400e-9f28-22e240d9595c 2024-09-18T15:21:54.001 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:21:54.000+0000 7f9eb051f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:21:54.003 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:21:54.002+0000 7f9eb051f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:21:54.004 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:21:54.003+0000 7f9eb051f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:21:54.005 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:21:54.003+0000 7f9eb051f440 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-18T15:21:54.005 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:21:54.003+0000 7f9eb051f440 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-18T15:21:54.005 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:21:54.003+0000 7f9eb051f440 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-18T15:21:54.080 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:21:54.078+0000 7f9eb051f440 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2024-09-18T15:22:00.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2024-09-18T15:22:00.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:22:00.206 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd5 key to auth repository 2024-09-18T15:22:00.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-18T15:22:00.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:22:00.719 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.5 2024-09-18T15:22:00.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-18T15:22:00.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:22:00.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:22:00.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:22:00.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:22:00.742 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:22:00.741+0000 7fae37c0c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:22:00.764 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:22:00.762+0000 7fae37c0c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:22:00.765 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:22:00.763+0000 7fae37c0c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:22:01.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-18T15:22:01.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:22:01.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-18T15:22:01.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:22:01.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:22:01.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:01.169 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:22:01.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:22:01.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:01.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:22:01.300 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:22:01.299+0000 7fae37c0c440 -1 Falling back to public interface 2024-09-18T15:22:01.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:02.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:02.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:02.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:22:02.563 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:22:02.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:02.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:22:02.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:03.938 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:03.938 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:03.939 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:22:03.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:22:03.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:03.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:22:03.953 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:22:03.952+0000 7fae37c0c440 -1 osd.5 0 log_to_monitors true 2024-09-18T15:22:04.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:05.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:05.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:05.396 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:22:05.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:22:05.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:05.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:22:05.875 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:06.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:06.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:06.877 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:22:06.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:22:06.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:06.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:22:07.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:08.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:08.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:08.254 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:22:08.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:22:08.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:08.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:22:08.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:09.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:09.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:09.694 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:22:09.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:22:09.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:09.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:22:10.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:11.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:11.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:11.206 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:22:11.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:22:11.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:11.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:22:11.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:12.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:12.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:12.705 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:22:12.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:22:12.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:12.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:22:13.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:14.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:14.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:14.110 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:22:14.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:22:14.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:14.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:22:14.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:15.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:15.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:15.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:22:15.533 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:22:15.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:15.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:22:16.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:17.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:17.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:17.181 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:22:17.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:22:17.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:17.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:22:17.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:18.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:18.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:18.687 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:22:18.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:22:18.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:18.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:22:19.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:20.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:20.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:20.388 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:22:20.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:22:20.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:20.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:22:21.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:22.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:22.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:22.107 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:22:22.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:22:22.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:22.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:22:22.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:23.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:23.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:23.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:22:23.937 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:22:23.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:23.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:22:25.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:26.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:26.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:26.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:22:26.109 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:22:26.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:26.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:22:26.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:27.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:27.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:27.764 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T15:22:27.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:22:27.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:27.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:22:28.142 INFO:tasks.workunit.client.0.smithi047.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/327242,v1:127.0.0.1:6843/327242] [v2:127.0.0.1:6844/327242,v1:127.0.0.1:6845/327242] exists,up 8b078dcf-5209-400e-9f28-22e240d9595c 2024-09-18T15:22:28.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:22:28.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:22:28.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:22:28.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:451: TEST_recovery_multi: create_pool test 1 1 2024-09-18T15:22:28.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-18T15:22:28.711 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test' already exists 2024-09-18T15:22:28.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:22:29.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:452: TEST_recovery_multi: ceph osd pool set test size 3 2024-09-18T15:22:30.626 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 1 size to 3 2024-09-18T15:22:30.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:453: TEST_recovery_multi: ceph osd pool set test min_size 1 2024-09-18T15:22:31.674 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 1 min_size to 1 2024-09-18T15:22:31.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:455: TEST_recovery_multi: wait_for_clean 2024-09-18T15:22:31.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:22:31.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:22:31.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:22:31.687 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:22:31.688 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:22:31.688 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:22:31.688 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:22:31.688 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:22:31.688 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:22:31.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:22:31.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:22:31.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:22:31.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:22:31.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:22:31.782 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:22:32.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:22:32.193 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:22:32.193 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T15:22:32.193 INFO:tasks.workunit.client.0.smithi047.stderr:3 2024-09-18T15:22:32.193 INFO:tasks.workunit.client.0.smithi047.stderr:4 2024-09-18T15:22:32.193 INFO:tasks.workunit.client.0.smithi047.stderr:5' 2024-09-18T15:22:32.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:22:32.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:22:32.193 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:22:32.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836521 2024-09-18T15:22:32.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836521 2024-09-18T15:22:32.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521' 2024-09-18T15:22:32.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:22:32.367 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:22:32.477 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672993 2024-09-18T15:22:32.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672993 2024-09-18T15:22:32.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521 1-42949672993' 2024-09-18T15:22:32.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:22:32.478 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:22:32.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509471 2024-09-18T15:22:32.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509471 2024-09-18T15:22:32.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521 1-42949672993 2-64424509471' 2024-09-18T15:22:32.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:22:32.605 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:22:32.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345943 2024-09-18T15:22:32.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345943 2024-09-18T15:22:32.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521 1-42949672993 2-64424509471 3-85899345943' 2024-09-18T15:22:32.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:22:32.713 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:22:32.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182412 2024-09-18T15:22:32.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182412 2024-09-18T15:22:32.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521 1-42949672993 2-64424509471 3-85899345943 4-107374182412' 2024-09-18T15:22:32.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:22:32.821 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:22:32.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018888 2024-09-18T15:22:32.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018888 2024-09-18T15:22:32.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521 1-42949672993 2-64424509471 3-85899345943 4-107374182412 5-128849018888' 2024-09-18T15:22:32.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:22:32.929 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836521 2024-09-18T15:22:32.929 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:22:32.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:22:32.930 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836521 2024-09-18T15:22:32.930 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:22:32.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836521 2024-09-18T15:22:32.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836521' 2024-09-18T15:22:32.932 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836521 2024-09-18T15:22:32.932 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:22:33.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836519 -lt 21474836521 2024-09-18T15:22:33.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:22:34.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:22:34.302 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:22:34.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836521 -lt 21474836521 2024-09-18T15:22:34.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:22:34.719 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672993 2024-09-18T15:22:34.719 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:22:34.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:22:34.720 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672993 2024-09-18T15:22:34.721 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:22:34.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672993 2024-09-18T15:22:34.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672993' 2024-09-18T15:22:34.722 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672993 2024-09-18T15:22:34.722 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:22:35.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672994 -lt 42949672993 2024-09-18T15:22:35.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:22:35.130 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509471 2024-09-18T15:22:35.130 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:22:35.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:22:35.131 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509471 2024-09-18T15:22:35.131 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:22:35.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509471 2024-09-18T15:22:35.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509471' 2024-09-18T15:22:35.132 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509471 2024-09-18T15:22:35.133 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:22:35.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509472 -lt 64424509471 2024-09-18T15:22:35.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:22:35.520 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345943 2024-09-18T15:22:35.520 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:22:35.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:22:35.521 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345943 2024-09-18T15:22:35.521 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:22:35.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345943 2024-09-18T15:22:35.523 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 85899345943 2024-09-18T15:22:35.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345943' 2024-09-18T15:22:35.523 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:22:35.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345943 -lt 85899345943 2024-09-18T15:22:35.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:22:35.890 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182412 2024-09-18T15:22:35.890 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:22:35.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:22:35.891 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182412 2024-09-18T15:22:35.891 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:22:35.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182412 2024-09-18T15:22:35.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182412' 2024-09-18T15:22:35.892 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.4 seq 107374182412 2024-09-18T15:22:35.892 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:22:36.264 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182412 -lt 107374182412 2024-09-18T15:22:36.264 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:22:36.264 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018888 2024-09-18T15:22:36.264 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:22:36.265 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:22:36.266 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018888 2024-09-18T15:22:36.266 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:22:36.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018888 2024-09-18T15:22:36.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018888' 2024-09-18T15:22:36.267 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.5 seq 128849018888 2024-09-18T15:22:36.267 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:22:36.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018888 -lt 128849018888 2024-09-18T15:22:36.675 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:22:36.676 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:22:36.676 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:22:37.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:22:37.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:22:37.173 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:22:37.173 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:22:37.173 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:22:37.173 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:22:37.173 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:22:37.173 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:22:37.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:22:37.565 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:22:37.565 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:22:37.565 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:22:38.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:22:38.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:22:38.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:22:38.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:457: TEST_recovery_multi: rados -p test put obj1 /dev/null 2024-09-18T15:22:38.245 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:459: TEST_recovery_multi: get_primary test obj1 2024-09-18T15:22:38.245 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-18T15:22:38.245 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T15:22:38.245 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-18T15:22:38.245 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:22:38.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:459: TEST_recovery_multi: local primary=1 2024-09-18T15:22:38.611 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:460: TEST_recovery_multi: get_not_primary test obj1 2024-09-18T15:22:38.612 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-09-18T15:22:38.612 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-09-18T15:22:38.612 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-09-18T15:22:38.612 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-18T15:22:38.612 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T15:22:38.612 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-18T15:22:38.613 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:22:38.983 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-18T15:22:38.983 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-09-18T15:22:38.983 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-18T15:22:39.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:460: TEST_recovery_multi: local otherosd=0 2024-09-18T15:22:39.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:462: TEST_recovery_multi: ceph osd set noout 2024-09-18T15:22:39.988 INFO:tasks.workunit.client.0.smithi047.stderr:noout is set 2024-09-18T15:22:39.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:463: TEST_recovery_multi: ceph osd set norecover 2024-09-18T15:22:40.889 INFO:tasks.workunit.client.0.smithi047.stderr:norecover is set 2024-09-18T15:22:40.901 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:464: TEST_recovery_multi: cat td/osd-recovery-stats/osd.0.pid 2024-09-18T15:22:40.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:464: TEST_recovery_multi: kill 318444 2024-09-18T15:22:40.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:465: TEST_recovery_multi: ceph osd down osd.0 2024-09-18T15:22:41.312 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already down. 2024-09-18T15:22:41.324 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:467: TEST_recovery_multi: expr 200 / 2 2024-09-18T15:22:41.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:467: TEST_recovery_multi: local half=100 2024-09-18T15:22:41.325 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: seq 2 100 2024-09-18T15:22:41.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:41.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj2 /dev/null 2024-09-18T15:22:42.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:42.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj3 /dev/null 2024-09-18T15:22:42.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:42.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj4 /dev/null 2024-09-18T15:22:42.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:42.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj5 /dev/null 2024-09-18T15:22:42.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:42.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj6 /dev/null 2024-09-18T15:22:42.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:42.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj7 /dev/null 2024-09-18T15:22:42.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:42.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj8 /dev/null 2024-09-18T15:22:42.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:42.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj9 /dev/null 2024-09-18T15:22:42.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:42.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj10 /dev/null 2024-09-18T15:22:42.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:42.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj11 /dev/null 2024-09-18T15:22:43.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:43.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj12 /dev/null 2024-09-18T15:22:43.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:43.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj13 /dev/null 2024-09-18T15:22:43.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:43.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj14 /dev/null 2024-09-18T15:22:43.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:43.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj15 /dev/null 2024-09-18T15:22:43.470 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:43.470 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj16 /dev/null 2024-09-18T15:22:43.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:43.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj17 /dev/null 2024-09-18T15:22:43.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:43.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj18 /dev/null 2024-09-18T15:22:43.907 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:43.907 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj19 /dev/null 2024-09-18T15:22:44.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:44.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj20 /dev/null 2024-09-18T15:22:44.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:44.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj21 /dev/null 2024-09-18T15:22:44.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:44.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj22 /dev/null 2024-09-18T15:22:44.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:44.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj23 /dev/null 2024-09-18T15:22:44.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:44.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj24 /dev/null 2024-09-18T15:22:44.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:44.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj25 /dev/null 2024-09-18T15:22:44.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:44.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj26 /dev/null 2024-09-18T15:22:44.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:44.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj27 /dev/null 2024-09-18T15:22:44.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:44.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj28 /dev/null 2024-09-18T15:22:44.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:44.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj29 /dev/null 2024-09-18T15:22:45.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:45.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj30 /dev/null 2024-09-18T15:22:45.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:45.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj31 /dev/null 2024-09-18T15:22:45.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:45.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj32 /dev/null 2024-09-18T15:22:45.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:45.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj33 /dev/null 2024-09-18T15:22:45.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:45.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj34 /dev/null 2024-09-18T15:22:45.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:45.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj35 /dev/null 2024-09-18T15:22:45.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:45.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj36 /dev/null 2024-09-18T15:22:45.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:45.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj37 /dev/null 2024-09-18T15:22:45.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:45.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj38 /dev/null 2024-09-18T15:22:46.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:46.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj39 /dev/null 2024-09-18T15:22:46.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:46.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj40 /dev/null 2024-09-18T15:22:46.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:46.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj41 /dev/null 2024-09-18T15:22:46.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:46.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj42 /dev/null 2024-09-18T15:22:46.477 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:46.477 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj43 /dev/null 2024-09-18T15:22:46.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:46.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj44 /dev/null 2024-09-18T15:22:46.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:46.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj45 /dev/null 2024-09-18T15:22:46.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:46.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj46 /dev/null 2024-09-18T15:22:46.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:46.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj47 /dev/null 2024-09-18T15:22:46.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:46.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj48 /dev/null 2024-09-18T15:22:47.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:47.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj49 /dev/null 2024-09-18T15:22:47.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:47.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj50 /dev/null 2024-09-18T15:22:47.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:47.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj51 /dev/null 2024-09-18T15:22:47.432 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:47.432 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj52 /dev/null 2024-09-18T15:22:47.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:47.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj53 /dev/null 2024-09-18T15:22:47.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:47.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj54 /dev/null 2024-09-18T15:22:47.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:47.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj55 /dev/null 2024-09-18T15:22:47.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:47.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj56 /dev/null 2024-09-18T15:22:47.993 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:47.993 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj57 /dev/null 2024-09-18T15:22:48.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:48.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj58 /dev/null 2024-09-18T15:22:48.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:48.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj59 /dev/null 2024-09-18T15:22:48.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:48.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj60 /dev/null 2024-09-18T15:22:48.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:48.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj61 /dev/null 2024-09-18T15:22:48.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:48.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj62 /dev/null 2024-09-18T15:22:48.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:48.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj63 /dev/null 2024-09-18T15:22:48.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:48.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj64 /dev/null 2024-09-18T15:22:48.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:48.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj65 /dev/null 2024-09-18T15:22:48.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:48.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj66 /dev/null 2024-09-18T15:22:49.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:49.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj67 /dev/null 2024-09-18T15:22:49.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:49.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj68 /dev/null 2024-09-18T15:22:49.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:49.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj69 /dev/null 2024-09-18T15:22:49.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:49.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj70 /dev/null 2024-09-18T15:22:49.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:49.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj71 /dev/null 2024-09-18T15:22:49.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:49.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj72 /dev/null 2024-09-18T15:22:49.676 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:49.676 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj73 /dev/null 2024-09-18T15:22:49.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:49.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj74 /dev/null 2024-09-18T15:22:49.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:49.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj75 /dev/null 2024-09-18T15:22:50.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:50.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj76 /dev/null 2024-09-18T15:22:50.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:50.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj77 /dev/null 2024-09-18T15:22:50.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:50.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj78 /dev/null 2024-09-18T15:22:50.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:50.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj79 /dev/null 2024-09-18T15:22:50.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:50.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj80 /dev/null 2024-09-18T15:22:50.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:50.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj81 /dev/null 2024-09-18T15:22:50.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:50.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj82 /dev/null 2024-09-18T15:22:50.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:50.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj83 /dev/null 2024-09-18T15:22:50.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:50.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj84 /dev/null 2024-09-18T15:22:50.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:50.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj85 /dev/null 2024-09-18T15:22:51.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:51.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj86 /dev/null 2024-09-18T15:22:51.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:51.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj87 /dev/null 2024-09-18T15:22:51.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:51.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj88 /dev/null 2024-09-18T15:22:51.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:51.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj89 /dev/null 2024-09-18T15:22:51.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:51.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj90 /dev/null 2024-09-18T15:22:51.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:51.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj91 /dev/null 2024-09-18T15:22:51.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:51.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj92 /dev/null 2024-09-18T15:22:51.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:51.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj93 /dev/null 2024-09-18T15:22:51.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:51.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj94 /dev/null 2024-09-18T15:22:52.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:52.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj95 /dev/null 2024-09-18T15:22:52.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:52.155 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj96 /dev/null 2024-09-18T15:22:52.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:52.264 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj97 /dev/null 2024-09-18T15:22:52.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:52.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj98 /dev/null 2024-09-18T15:22:52.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:52.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj99 /dev/null 2024-09-18T15:22:52.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-18T15:22:52.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj100 /dev/null 2024-09-18T15:22:52.674 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:473: TEST_recovery_multi: cat td/osd-recovery-stats/osd.1.pid 2024-09-18T15:22:52.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:473: TEST_recovery_multi: kill 320198 2024-09-18T15:22:52.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:474: TEST_recovery_multi: ceph osd down osd.1 2024-09-18T15:22:53.243 INFO:tasks.workunit.client.0.smithi047.stderr:osd.1 is already down. 2024-09-18T15:22:53.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:475: TEST_recovery_multi: activate_osd td/osd-recovery-stats 0 2024-09-18T15:22:53.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-recovery-stats 2024-09-18T15:22:53.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:22:53.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:22:53.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:22:53.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-recovery-stats/0 2024-09-18T15:22:53.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:22:53.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:22:53.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:22:53.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:22:53.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-09-18T15:22:53.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-09-18T15:22:53.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:22:53.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:22:53.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:22:53.258 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:22:53.258 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:22:53.258 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:22:53.258 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:22:53.259 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:22:53.259 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:22:53.259 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:22:53.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:22:53.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:22:53.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:22:53.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:22:53.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:22:53.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:22:53.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:22:53.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:22:53.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:22:53.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:22:53.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-recovery-stats/0 2024-09-18T15:22:53.260 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T15:22:53.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:22:53.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:22:53.261 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-recovery-stats/0/whoami 2024-09-18T15:22:53.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:22:53.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:22:53.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:22:53.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:22:53.280 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:22:53.278+0000 7fa4a4c18440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:22:53.302 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:22:53.300+0000 7fa4a4c18440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:22:53.303 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:22:53.301+0000 7fa4a4c18440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:22:53.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:22:53.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:22:53.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:22:53.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:22:53.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:22:53.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:53.651 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:22:53.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:22:53.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:53.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:22:53.816 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:22:53.815+0000 7fa4a4c18440 -1 Falling back to public interface 2024-09-18T15:22:54.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:55.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:55.052 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:55.052 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:22:55.052 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:22:55.052 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:55.052 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:22:55.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:56.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:56.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:56.487 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:22:56.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:22:56.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:56.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:22:56.738 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:22:56.737+0000 7fa4a4c18440 -1 osd.0 39 log_to_monitors true 2024-09-18T15:22:56.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:22:57.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:22:57.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:22:57.943 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:22:57.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:22:57.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:22:57.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:22:58.366 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 47 up_thru 47 down_at 42 last_clean_interval [5,39) [v2:127.0.0.1:6802/330901,v1:127.0.0.1:6803/330901] [v2:127.0.0.1:6804/330901,v1:127.0.0.1:6805/330901] exists,up c33e7a09-4276-4cea-9133-77e0b0e8f9aa 2024-09-18T15:22:58.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:22:58.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:22:58.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:22:58.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:476: TEST_recovery_multi: sleep 3 2024-09-18T15:23:01.368 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: expr 100 + 1 2024-09-18T15:23:01.369 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: seq 101 200 2024-09-18T15:23:01.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:01.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj101 /dev/null 2024-09-18T15:23:01.483 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:01.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj102 /dev/null 2024-09-18T15:23:01.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:01.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj103 /dev/null 2024-09-18T15:23:01.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:01.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj104 /dev/null 2024-09-18T15:23:01.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:01.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj105 /dev/null 2024-09-18T15:23:01.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:01.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj106 /dev/null 2024-09-18T15:23:02.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:02.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj107 /dev/null 2024-09-18T15:23:02.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:02.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj108 /dev/null 2024-09-18T15:23:02.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:02.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj109 /dev/null 2024-09-18T15:23:02.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:02.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj110 /dev/null 2024-09-18T15:23:02.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:02.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj111 /dev/null 2024-09-18T15:23:02.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:02.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj112 /dev/null 2024-09-18T15:23:02.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:02.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj113 /dev/null 2024-09-18T15:23:02.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:02.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj114 /dev/null 2024-09-18T15:23:02.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:02.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj115 /dev/null 2024-09-18T15:23:02.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:02.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj116 /dev/null 2024-09-18T15:23:03.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:03.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj117 /dev/null 2024-09-18T15:23:03.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:03.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj118 /dev/null 2024-09-18T15:23:03.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:03.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj119 /dev/null 2024-09-18T15:23:03.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:03.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj120 /dev/null 2024-09-18T15:23:03.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:03.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj121 /dev/null 2024-09-18T15:23:03.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:03.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj122 /dev/null 2024-09-18T15:23:03.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:03.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj123 /dev/null 2024-09-18T15:23:03.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:03.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj124 /dev/null 2024-09-18T15:23:04.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:04.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj125 /dev/null 2024-09-18T15:23:04.095 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:04.096 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj126 /dev/null 2024-09-18T15:23:04.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:04.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj127 /dev/null 2024-09-18T15:23:04.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:04.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj128 /dev/null 2024-09-18T15:23:04.389 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:04.389 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj129 /dev/null 2024-09-18T15:23:04.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:04.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj130 /dev/null 2024-09-18T15:23:04.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:04.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj131 /dev/null 2024-09-18T15:23:04.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:04.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj132 /dev/null 2024-09-18T15:23:04.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:04.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj133 /dev/null 2024-09-18T15:23:04.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:04.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj134 /dev/null 2024-09-18T15:23:05.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:05.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj135 /dev/null 2024-09-18T15:23:05.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:05.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj136 /dev/null 2024-09-18T15:23:05.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:05.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj137 /dev/null 2024-09-18T15:23:05.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:05.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj138 /dev/null 2024-09-18T15:23:05.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:05.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj139 /dev/null 2024-09-18T15:23:05.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:05.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj140 /dev/null 2024-09-18T15:23:05.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:05.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj141 /dev/null 2024-09-18T15:23:05.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:05.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj142 /dev/null 2024-09-18T15:23:05.861 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:05.862 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj143 /dev/null 2024-09-18T15:23:05.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:05.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj144 /dev/null 2024-09-18T15:23:06.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:06.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj145 /dev/null 2024-09-18T15:23:06.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:06.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj146 /dev/null 2024-09-18T15:23:06.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:06.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj147 /dev/null 2024-09-18T15:23:06.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:06.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj148 /dev/null 2024-09-18T15:23:06.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:06.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj149 /dev/null 2024-09-18T15:23:06.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:06.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj150 /dev/null 2024-09-18T15:23:06.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:06.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj151 /dev/null 2024-09-18T15:23:06.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:06.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj152 /dev/null 2024-09-18T15:23:06.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:06.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj153 /dev/null 2024-09-18T15:23:06.991 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:06.992 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj154 /dev/null 2024-09-18T15:23:07.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:07.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj155 /dev/null 2024-09-18T15:23:07.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:07.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj156 /dev/null 2024-09-18T15:23:07.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:07.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj157 /dev/null 2024-09-18T15:23:07.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:07.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj158 /dev/null 2024-09-18T15:23:07.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:07.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj159 /dev/null 2024-09-18T15:23:07.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:07.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj160 /dev/null 2024-09-18T15:23:07.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:07.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj161 /dev/null 2024-09-18T15:23:07.862 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:07.862 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj162 /dev/null 2024-09-18T15:23:07.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:07.938 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj163 /dev/null 2024-09-18T15:23:08.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:08.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj164 /dev/null 2024-09-18T15:23:08.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:08.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj165 /dev/null 2024-09-18T15:23:08.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:08.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj166 /dev/null 2024-09-18T15:23:08.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:08.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj167 /dev/null 2024-09-18T15:23:08.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:08.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj168 /dev/null 2024-09-18T15:23:08.549 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:08.549 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj169 /dev/null 2024-09-18T15:23:08.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:08.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj170 /dev/null 2024-09-18T15:23:08.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:08.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj171 /dev/null 2024-09-18T15:23:08.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:08.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj172 /dev/null 2024-09-18T15:23:08.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:08.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj173 /dev/null 2024-09-18T15:23:09.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:09.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj174 /dev/null 2024-09-18T15:23:09.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:09.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj175 /dev/null 2024-09-18T15:23:09.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:09.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj176 /dev/null 2024-09-18T15:23:09.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:09.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj177 /dev/null 2024-09-18T15:23:09.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:09.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj178 /dev/null 2024-09-18T15:23:10.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:10.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj179 /dev/null 2024-09-18T15:23:10.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:10.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj180 /dev/null 2024-09-18T15:23:10.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:10.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj181 /dev/null 2024-09-18T15:23:10.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:10.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj182 /dev/null 2024-09-18T15:23:10.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:10.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj183 /dev/null 2024-09-18T15:23:10.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:10.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj184 /dev/null 2024-09-18T15:23:10.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:10.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj185 /dev/null 2024-09-18T15:23:10.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:10.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj186 /dev/null 2024-09-18T15:23:10.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:10.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj187 /dev/null 2024-09-18T15:23:11.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:11.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj188 /dev/null 2024-09-18T15:23:11.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:11.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj189 /dev/null 2024-09-18T15:23:11.242 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:11.242 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj190 /dev/null 2024-09-18T15:23:11.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:11.343 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj191 /dev/null 2024-09-18T15:23:11.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:11.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj192 /dev/null 2024-09-18T15:23:11.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:11.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj193 /dev/null 2024-09-18T15:23:11.660 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:11.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj194 /dev/null 2024-09-18T15:23:11.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:11.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj195 /dev/null 2024-09-18T15:23:11.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:11.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj196 /dev/null 2024-09-18T15:23:12.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:12.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj197 /dev/null 2024-09-18T15:23:12.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:12.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj198 /dev/null 2024-09-18T15:23:12.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:12.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj199 /dev/null 2024-09-18T15:23:12.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-18T15:23:12.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj200 /dev/null 2024-09-18T15:23:12.398 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:483: TEST_recovery_multi: get_pg test obj1 2024-09-18T15:23:12.398 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-18T15:23:12.398 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-09-18T15:23:12.398 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-09-18T15:23:12.398 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:23:12.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:483: TEST_recovery_multi: local PG=1.0 2024-09-18T15:23:12.763 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:484: TEST_recovery_multi: get_not_primary test obj200 2024-09-18T15:23:12.763 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-09-18T15:23:12.763 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj200 2024-09-18T15:23:12.764 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj200 2024-09-18T15:23:12.764 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-18T15:23:12.764 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj200 2024-09-18T15:23:12.764 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj200 2024-09-18T15:23:12.764 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:23:13.132 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=0 2024-09-18T15:23:13.132 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj200 2024-09-18T15:23:13.132 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 0)) | .[0]' 2024-09-18T15:23:13.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:484: TEST_recovery_multi: local otherosd=2 2024-09-18T15:23:13.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:486: TEST_recovery_multi: ceph osd unset noout 2024-09-18T15:23:14.213 INFO:tasks.workunit.client.0.smithi047.stderr:noout is unset 2024-09-18T15:23:14.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:487: TEST_recovery_multi: ceph osd out osd.1 osd.2 2024-09-18T15:23:14.969 INFO:tasks.workunit.client.0.smithi047.stderr:osd.1 is already out. osd.2 is already out. 2024-09-18T15:23:14.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:488: TEST_recovery_multi: activate_osd td/osd-recovery-stats 1 2024-09-18T15:23:14.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-recovery-stats 2024-09-18T15:23:14.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:23:14.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:23:14.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:23:14.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-recovery-stats/1 2024-09-18T15:23:14.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:23:14.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:23:14.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:23:14.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:23:14.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-09-18T15:23:14.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-09-18T15:23:14.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:23:14.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:23:14.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:23:14.983 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:23:14.983 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:23:14.984 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:23:14.984 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:23:14.984 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:23:14.984 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:23:14.984 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:23:14.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:23:14.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:23:14.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:23:14.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:23:14.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:23:14.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:23:14.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:23:14.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:23:14.985 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:23:14.985 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:23:14.985 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-recovery-stats/1 2024-09-18T15:23:14.985 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T15:23:14.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:23:14.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:23:14.986 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-recovery-stats/1/whoami 2024-09-18T15:23:14.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:23:14.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:23:14.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:23:14.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:23:15.005 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:23:15.004+0000 7f5618635440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:23:15.111 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:23:15.109+0000 7f5618635440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:23:15.112 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:23:15.111+0000 7f5618635440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:23:15.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:23:15.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:23:15.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:23:15.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:23:15.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:23:15.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:15.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:23:15.431 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:23:15.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:15.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:23:15.630 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:23:15.629+0000 7f5618635440 -1 Falling back to public interface 2024-09-18T15:23:15.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:23:16.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:23:16.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:16.867 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:23:16.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:23:16.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:16.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:23:17.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:23:18.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:23:18.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:18.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:23:18.316 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:23:18.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:18.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:23:18.602 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:23:18.601+0000 7f5618635440 -1 osd.1 43 log_to_monitors true 2024-09-18T15:23:18.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:23:19.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:23:19.764 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:23:19.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:23:19.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:23:19.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:23:19.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:23:20.138 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up out weight 0 up_from 55 up_thru 42 down_at 44 last_clean_interval [10,43) [v2:127.0.0.1:6810/333620,v1:127.0.0.1:6811/333620] [v2:127.0.0.1:6812/333620,v1:127.0.0.1:6813/333620] exists,up 352d8ac3-c62f-4781-a7bb-0f34888ce050 2024-09-18T15:23:20.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:23:20.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:23:20.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:23:20.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:489: TEST_recovery_multi: sleep 3 2024-09-18T15:23:23.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:491: TEST_recovery_multi: ceph osd pool set test size 4 2024-09-18T15:23:24.048 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 1 size to 4 2024-09-18T15:23:24.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:492: TEST_recovery_multi: ceph osd unset norecover 2024-09-18T15:23:25.508 INFO:tasks.workunit.client.0.smithi047.stderr:norecover is unset 2024-09-18T15:23:25.521 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:493: TEST_recovery_multi: get_primary test obj1 2024-09-18T15:23:25.522 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-18T15:23:25.522 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T15:23:25.522 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-18T15:23:25.522 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:23:25.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:493: TEST_recovery_multi: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-18T15:23:26.086 INFO:tasks.workunit.client.0.smithi047.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-18T15:23:26.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:494: TEST_recovery_multi: sleep 2 2024-09-18T15:23:28.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:496: TEST_recovery_multi: wait_for_clean 2024-09-18T15:23:28.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:23:28.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:23:28.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:23:28.097 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:23:28.098 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:23:28.098 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:23:28.098 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:23:28.098 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:23:28.098 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:23:28.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:23:28.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:23:28.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:23:28.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:23:28.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:23:28.200 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:23:28.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:23:28.570 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:23:28.570 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T15:23:28.570 INFO:tasks.workunit.client.0.smithi047.stderr:3 2024-09-18T15:23:28.570 INFO:tasks.workunit.client.0.smithi047.stderr:4 2024-09-18T15:23:28.570 INFO:tasks.workunit.client.0.smithi047.stderr:5' 2024-09-18T15:23:28.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:23:28.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:23:28.570 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:23:28.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=201863462921 2024-09-18T15:23:28.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 201863462921 2024-09-18T15:23:28.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-201863462921' 2024-09-18T15:23:28.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:23:28.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:23:28.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=236223201284 2024-09-18T15:23:28.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 236223201284 2024-09-18T15:23:28.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-201863462921 1-236223201284' 2024-09-18T15:23:28.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:23:28.903 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:23:29.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509485 2024-09-18T15:23:29.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509485 2024-09-18T15:23:29.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-201863462921 1-236223201284 2-64424509485' 2024-09-18T15:23:29.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:23:29.012 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:23:29.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345956 2024-09-18T15:23:29.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345956 2024-09-18T15:23:29.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-201863462921 1-236223201284 2-64424509485 3-85899345956' 2024-09-18T15:23:29.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:23:29.120 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:23:29.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182425 2024-09-18T15:23:29.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182425 2024-09-18T15:23:29.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-201863462921 1-236223201284 2-64424509485 3-85899345956 4-107374182425' 2024-09-18T15:23:29.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:23:29.229 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:23:29.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018901 2024-09-18T15:23:29.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018901 2024-09-18T15:23:29.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-201863462921 1-236223201284 2-64424509485 3-85899345956 4-107374182425 5-128849018901' 2024-09-18T15:23:29.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:23:29.387 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-201863462921 2024-09-18T15:23:29.387 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:23:29.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:23:29.388 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-201863462921 2024-09-18T15:23:29.388 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:23:29.389 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=201863462921 2024-09-18T15:23:29.389 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 201863462921 2024-09-18T15:23:29.389 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 201863462921' 2024-09-18T15:23:29.390 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:23:29.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 201863462921 -lt 201863462921 2024-09-18T15:23:29.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:23:29.815 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-236223201284 2024-09-18T15:23:29.815 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:23:29.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:23:29.816 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-236223201284 2024-09-18T15:23:29.816 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:23:29.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=236223201284 2024-09-18T15:23:29.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 236223201284' 2024-09-18T15:23:29.817 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 236223201284 2024-09-18T15:23:29.818 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:23:30.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 236223201284 -lt 236223201284 2024-09-18T15:23:30.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:23:30.227 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509485 2024-09-18T15:23:30.227 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:23:30.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:23:30.229 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509485 2024-09-18T15:23:30.229 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:23:30.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509485 2024-09-18T15:23:30.230 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509485 2024-09-18T15:23:30.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509485' 2024-09-18T15:23:30.230 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:23:30.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509485 -lt 64424509485 2024-09-18T15:23:30.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:23:30.719 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345956 2024-09-18T15:23:30.719 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:23:30.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:23:30.720 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345956 2024-09-18T15:23:30.720 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:23:30.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345956 2024-09-18T15:23:30.722 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 85899345956 2024-09-18T15:23:30.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345956' 2024-09-18T15:23:30.722 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:23:31.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345956 -lt 85899345956 2024-09-18T15:23:31.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:23:31.205 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182425 2024-09-18T15:23:31.205 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:23:31.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:23:31.206 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182425 2024-09-18T15:23:31.206 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:23:31.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182425 2024-09-18T15:23:31.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182425' 2024-09-18T15:23:31.208 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.4 seq 107374182425 2024-09-18T15:23:31.208 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:23:31.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182425 -lt 107374182425 2024-09-18T15:23:31.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:23:31.738 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018901 2024-09-18T15:23:31.738 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:23:31.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:23:31.739 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018901 2024-09-18T15:23:31.739 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:23:31.740 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018901 2024-09-18T15:23:31.740 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.5 seq 128849018901 2024-09-18T15:23:31.741 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018901' 2024-09-18T15:23:31.741 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:23:32.172 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018901 -lt 128849018901 2024-09-18T15:23:32.172 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:23:32.172 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:23:32.172 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:23:32.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:23:32.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:23:32.703 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:23:32.703 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:23:32.703 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:23:32.703 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:23:32.703 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:23:32.703 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:23:33.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:23:33.072 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:23:33.072 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:23:33.072 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:23:33.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:23:33.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-18T15:23:33.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T15:23:33.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-18T15:23:33.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:23:33.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:23:33.740 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:23:33.741 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:23:33.741 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:23:33.741 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:23:33.741 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:23:33.741 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:23:33.741 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:23:33.741 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:23:34.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:23:34.198 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:23:34.198 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:23:34.198 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:23:34.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:23:34.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:23:34.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:23:34.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:23:34.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:23:34.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:23:34.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:23:34.757 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:23:34.757 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:23:35.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-18T15:23:35.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-18T15:23:35.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:23:35.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:23:35.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:23:35.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:23:35.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:23:35.462 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:23:35.462 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:23:35.462 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:23:35.462 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:23:35.462 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:23:35.462 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:23:35.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:23:35.823 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:23:35.824 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:23:35.824 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:23:36.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:23:36.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:23:36.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:23:36.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:23:36.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:23:36.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:23:36.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:23:36.427 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:23:36.427 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:23:37.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-18T15:23:37.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-18T15:23:37.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:23:37.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:23:37.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:23:37.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:23:37.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:23:37.115 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:23:37.116 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:23:37.116 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:23:37.116 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:23:37.116 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:23:37.116 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:23:37.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:23:37.510 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:23:37.510 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:23:37.510 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:23:38.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:23:38.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:23:38.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:23:38.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:23:38.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:23:38.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:23:38.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:23:38.032 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:23:38.032 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:23:38.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-18T15:23:38.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-18T15:23:38.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:23:38.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:23:38.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:23:38.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:23:38.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:23:38.682 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:23:38.682 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:23:38.682 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:23:38.682 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:23:38.683 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:23:38.683 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:23:39.052 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:23:39.052 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:23:39.052 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:23:39.053 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:23:39.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:23:39.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:23:39.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:23:39.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:23:39.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:23:39.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:23:39.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:23:39.634 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:23:39.634 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:23:40.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-09-18T15:23:40.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-09-18T15:23:40.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:23:40.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:23:40.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:23:40.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:23:40.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:23:40.324 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:23:40.324 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:23:40.324 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:23:40.324 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:23:40.324 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:23:40.324 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:23:40.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:23:40.682 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:23:40.682 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:23:40.682 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:23:41.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:23:41.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:23:41.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:23:41.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:23:41.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:23:41.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:23:41.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:23:41.246 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:23:41.246 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:23:41.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-18T15:23:41.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-18T15:23:41.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:23:41.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:23:41.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:23:41.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:23:41.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:23:41.856 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:23:41.856 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:23:41.856 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:23:41.856 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:23:41.856 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:23:41.856 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:23:42.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:23:42.224 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:23:42.224 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:23:42.224 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:23:42.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:23:42.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:23:42.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:23:42.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:23:42.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:23:42.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:23:42.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:23:42.725 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:23:42.725 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:23:43.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-18T15:23:43.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-18T15:23:43.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:23:43.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:23:43.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:23:43.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:23:43.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:23:43.379 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:23:43.379 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:23:43.379 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:23:43.379 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:23:43.379 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:23:43.379 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:23:43.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:23:43.752 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:23:43.752 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:23:43.752 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:23:44.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:23:44.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:23:44.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:23:44.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:23:44.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:23:44.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:23:44.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:23:44.404 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:23:44.404 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:23:44.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=7 2024-09-18T15:23:44.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 7 '!=' null 2024-09-18T15:23:44.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:23:44.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:23:44.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:23:45.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:23:45.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:23:45.041 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:23:45.041 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:23:45.041 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:23:45.041 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:23:45.041 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:23:45.042 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:23:45.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:23:45.468 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:23:45.469 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:23:45.469 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:23:46.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:23:46.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:23:46.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:23:46.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:23:46.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:23:46.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:23:46.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:23:46.018 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:23:46.018 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:23:46.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-09-18T15:23:46.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-09-18T15:23:46.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:23:46.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:23:46.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:23:46.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:23:46.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:23:46.814 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:23:46.814 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:23:46.814 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:23:46.814 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:23:46.814 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:23:46.814 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:23:47.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:23:47.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:23:47.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:23:47.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:23:47.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:23:47.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:23:47.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:23:47.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:23:47.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:23:47.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:23:47.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:23:47.880 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:23:47.880 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:23:48.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=9 2024-09-18T15:23:48.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 9 '!=' null 2024-09-18T15:23:48.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:23:48.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:23:48.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:23:48.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:23:48.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:23:48.526 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:23:48.526 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:23:48.526 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:23:48.526 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:23:48.526 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:23:48.527 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:23:48.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:23:48.887 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:23:48.888 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:23:48.888 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:23:49.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:23:49.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:23:49.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:23:49.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:23:49.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:23:49.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:23:49.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:23:49.553 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:23:49.553 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:23:50.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-18T15:23:50.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-18T15:23:50.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:23:50.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:23:50.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:23:50.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:23:50.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:23:50.331 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:23:50.331 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:23:50.331 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:23:50.332 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:23:50.332 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:23:50.332 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:23:50.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:23:50.795 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:23:50.795 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:23:50.795 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:23:51.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:23:51.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:23:51.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:23:51.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:23:51.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:23:51.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:23:51.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:23:51.393 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:23:51.393 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:23:51.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-18T15:23:51.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-18T15:23:51.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:23:51.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:23:51.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:23:52.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:23:52.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:23:52.059 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:23:52.059 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:23:52.059 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:23:52.059 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:23:52.060 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:23:52.060 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:23:52.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:23:52.418 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:23:52.418 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:23:52.419 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:23:52.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:23:52.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:23:52.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:23:52.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:23:52.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:23:52.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:23:52.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:23:52.987 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:23:52.987 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:23:53.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11 2024-09-18T15:23:53.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11 '!=' null 2024-09-18T15:23:53.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:23:53.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:23:53.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:23:53.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:23:53.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:23:53.689 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:23:53.689 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:23:53.689 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:23:53.689 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:23:53.689 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:23:53.689 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:23:54.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:23:54.047 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:23:54.047 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:23:54.047 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:23:54.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:23:54.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:23:54.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:23:54.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:23:54.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:23:54.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:23:54.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:23:54.540 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:23:54.540 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:23:55.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11 2024-09-18T15:23:55.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11 '!=' null 2024-09-18T15:23:55.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:23:55.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:23:55.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:23:55.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:23:55.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:23:55.158 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:23:55.159 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:23:55.159 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:23:55.159 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:23:55.159 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:23:55.159 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:23:55.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:23:55.518 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:23:55.518 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:23:55.518 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:23:56.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:23:56.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:23:56.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:23:56.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:23:56.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:23:56.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:23:56.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:23:56.020 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:23:56.020 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:23:56.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-09-18T15:23:56.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-09-18T15:23:56.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:23:56.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:23:56.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:23:56.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:23:56.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:23:56.658 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:23:56.658 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:23:56.658 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:23:56.658 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:23:56.658 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:23:56.658 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:23:57.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:23:57.017 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:23:57.018 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:23:57.018 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:23:57.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:23:57.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:23:57.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:23:57.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:23:57.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:23:57.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:23:57.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:23:57.510 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:23:57.510 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:23:58.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-09-18T15:23:58.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-09-18T15:23:58.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:23:58.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:23:58.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:23:58.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:23:58.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:23:58.134 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:23:58.134 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:23:58.134 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:23:58.134 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:23:58.134 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:23:58.135 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:23:58.491 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:23:58.491 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:23:58.492 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:23:58.492 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:23:59.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:23:59.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:23:59.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:23:59.005 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:499: TEST_recovery_multi: get_primary test obj1 2024-09-18T15:23:59.005 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-18T15:23:59.005 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T15:23:59.005 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-18T15:23:59.005 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:23:59.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:499: TEST_recovery_multi: primary=0 2024-09-18T15:23:59.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:501: TEST_recovery_multi: local log=td/osd-recovery-stats/osd.0.log 2024-09-18T15:23:59.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:502: TEST_recovery_multi: check td/osd-recovery-stats 1.0 0 replicated 399 0 300 0 99 0 2024-09-18T15:23:59.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-09-18T15:23:59.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-09-18T15:23:59.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=0 2024-09-18T15:23:59.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2024-09-18T15:23:59.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=399 2024-09-18T15:23:59.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-09-18T15:23:59.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=300 2024-09-18T15:23:59.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-09-18T15:23:59.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start=99 2024-09-18T15:23:59.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end=0 2024-09-18T15:23:59.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.0.log 2024-09-18T15:23:59.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-09-18T15:23:59.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2024-09-18T15:23:59.409 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-09-18T15:23:59.409 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.0.log 2024-09-18T15:23:59.409 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-09-18T15:23:59.410 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-09-18T15:23:59.437 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[0,4,3,5]/[0,3]' 2024-09-18T15:23:59.437 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-09-18T15:23:59.438 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-18T15:23:59.438 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.0.log 2024-09-18T15:23:59.438 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [0,4,3,5]/[0,3] ' 2024-09-18T15:23:59.438 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-09-18T15:23:59.438 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-09-18T15:23:59.438 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-18T15:23:59.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=399 2024-09-18T15:23:59.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 399 399 2024-09-18T15:23:59.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=399 2024-09-18T15:23:59.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-18T15:23:59.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=399 2024-09-18T15:23:59.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-18T15:23:59.444 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-18T15:23:59.444 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.0.log 2024-09-18T15:23:59.444 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-09-18T15:23:59.444 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-18T15:23:59.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-09-18T15:23:59.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-09-18T15:23:59.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-18T15:23:59.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-18T15:23:59.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-18T15:23:59.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-18T15:23:59.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-09-18T15:23:59.456 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-18T15:23:59.457 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.0.log 2024-09-18T15:23:59.457 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [0,4,3,5]/[0,3] ' 2024-09-18T15:23:59.457 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-09-18T15:23:59.457 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-09-18T15:23:59.457 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-18T15:23:59.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=300 2024-09-18T15:23:59.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 300 300 2024-09-18T15:23:59.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=300 2024-09-18T15:23:59.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-18T15:23:59.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=300 2024-09-18T15:23:59.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-18T15:23:59.462 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-18T15:23:59.462 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.0.log 2024-09-18T15:23:59.462 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-09-18T15:23:59.463 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-18T15:23:59.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-09-18T15:23:59.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-09-18T15:23:59.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-18T15:23:59.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-18T15:23:59.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-18T15:23:59.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-18T15:23:59.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n 99 ']' 2024-09-18T15:23:59.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:102: check: which='shard 0' 2024-09-18T15:23:59.474 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-18T15:23:59.474 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 0 ' td/osd-recovery-stats/osd.0.log 2024-09-18T15:23:59.475 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -F ' [0,4,3,5]/[0,3] ' 2024-09-18T15:23:59.475 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -v est 2024-09-18T15:23:59.475 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: head -1 2024-09-18T15:23:59.475 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-18T15:23:59.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: FIRST=99 2024-09-18T15:23:59.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:104: check: below_margin 99 99 2024-09-18T15:23:59.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=99 2024-09-18T15:23:59.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-18T15:23:59.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=99 2024-09-18T15:23:59.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-18T15:23:59.480 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-18T15:23:59.480 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 0 ' td/osd-recovery-stats/osd.0.log 2024-09-18T15:23:59.481 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: tail -1 2024-09-18T15:23:59.481 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-18T15:23:59.492 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: LAST=0 2024-09-18T15:23:59.492 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:106: check: above_margin 0 0 2024-09-18T15:23:59.492 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-18T15:23:59.492 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-18T15:23:59.492 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-18T15:23:59.492 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-18T15:23:59.492 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:504: TEST_recovery_multi: delete_pool test 2024-09-18T15:23:59.492 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-18T15:23:59.492 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-18T15:24:00.056 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test' does not exist 2024-09-18T15:24:00.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:505: TEST_recovery_multi: kill_daemons td/osd-recovery-stats 2024-09-18T15:24:00.066 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:24:00.066 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:24:00.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:24:00.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:24:00.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:24:05.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:24:05.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-09-18T15:24:05.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-18T15:24:05.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:24:05.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-18T15:24:05.384 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:24:05.384 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:24:05.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:24:05.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:24:05.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:24:05.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:24:05.393 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:24:05.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:24:05.394 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:24:05.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:24:05.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:24:05.396 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:24:05.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:24:05.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:24:05.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:24:05.397 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:24:05.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:24:05.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:24:05.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-18T15:24:05.442 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:24:05.442 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:24:05.443 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:24:05.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.288083 2024-09-18T15:24:05.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:24:05.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:24:05.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-09-18T15:24:05.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-09-18T15:24:05.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-09-18T15:24:05.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-09-18T15:24:05.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-18T15:24:05.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:24:05.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-18T15:24:05.445 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:24:05.445 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:24:05.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:24:05.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:24:05.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:24:05.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:24:05.447 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:24:05.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:24:05.448 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:24:05.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:24:05.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:24:05.449 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:24:05.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:24:05.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:24:05.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:24:05.451 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:24:05.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:24:05.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:24:05.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-18T15:24:05.454 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:24:05.454 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:24:05.454 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:24:05.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.288083 2024-09-18T15:24:05.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:24:05.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:24:05.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-09-18T15:24:05.457 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T15:24:05.457 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:24:05.457 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:24:05.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.288083 2024-09-18T15:24:05.459 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T15:24:05.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T15:24:05.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T15:24:05.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-09-18T15:24:05.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_replicated_out1 td/osd-recovery-stats 2024-09-18T15:24:05.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:167: TEST_recovery_replicated_out1: local dir=td/osd-recovery-stats 2024-09-18T15:24:05.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:169: TEST_recovery_replicated_out1: do_recovery_out1 td/osd-recovery-stats replicated 2024-09-18T15:24:05.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:117: do_recovery_out1: local dir=td/osd-recovery-stats 2024-09-18T15:24:05.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:118: do_recovery_out1: shift 2024-09-18T15:24:05.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:119: do_recovery_out1: local type=replicated 2024-09-18T15:24:05.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:121: do_recovery_out1: run_mon td/osd-recovery-stats a 2024-09-18T15:24:05.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-09-18T15:24:05.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T15:24:05.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T15:24:05.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T15:24:05.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-09-18T15:24:05.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-09-18T15:24:05.814 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T15:24:05.815 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:24:05.815 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:24:05.815 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:24:05.815 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:24:05.815 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:24:05.815 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:24:05.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T15:24:06.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T15:24:06.099 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T15:24:06.099 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:24:06.099 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:24:06.100 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T15:24:06.100 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T15:24:06.100 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:24:06.100 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:24:06.101 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:24:06.101 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:24:06.101 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:24:06.101 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:24:06.101 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.288083/ceph-mon.a.asok 2024-09-18T15:24:06.101 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:24:06.102 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.288083/ceph-mon.a.asok config get fsid 2024-09-18T15:24:06.186 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T15:24:06.186 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:24:06.186 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:24:06.186 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T15:24:06.186 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T15:24:06.187 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:24:06.187 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:24:06.187 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:24:06.187 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:24:06.188 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:24:06.188 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:24:06.188 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.288083/ceph-mon.a.asok 2024-09-18T15:24:06.188 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:24:06.188 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.288083/ceph-mon.a.asok config get mon_host 2024-09-18T15:24:06.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:122: do_recovery_out1: run_mgr td/osd-recovery-stats x 2024-09-18T15:24:06.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-09-18T15:24:06.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T15:24:06.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T15:24:06.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T15:24:06.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-09-18T15:24:06.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T15:24:07.120 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T15:24:07.120 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:24:07.121 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:24:07.121 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:24:07.121 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:24:07.121 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:24:07.121 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:24:07.121 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:24:07.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:24:07.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:123: do_recovery_out1: run_osd td/osd-recovery-stats 0 2024-09-18T15:24:07.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:24:07.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:24:07.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T15:24:07.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:24:07.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-09-18T15:24:07.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:24:07.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:24:07.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:24:07.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:24:07.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-09-18T15:24:07.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-09-18T15:24:07.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:24:07.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:24:07.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:24:07.149 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:24:07.149 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:24:07.149 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:24:07.149 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:24:07.149 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:24:07.149 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:24:07.149 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:24:07.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:24:07.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:24:07.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:24:07.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:24:07.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:24:07.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:24:07.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:24:07.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:24:07.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:24:07.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:24:07.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:24:07.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:24:07.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-09-18T15:24:07.151 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:24:07.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c6407b5b-79cb-40e5-b9e4-482798493fa3 2024-09-18T15:24:07.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c6407b5b-79cb-40e5-b9e4-482798493fa3' 2024-09-18T15:24:07.151 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 c6407b5b-79cb-40e5-b9e4-482798493fa3 2024-09-18T15:24:07.152 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:24:07.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCX8OpmRoi1CRAAlVUBB29xbwJ5o7JPuh4CbQ== 2024-09-18T15:24:07.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCX8OpmRoi1CRAAlVUBB29xbwJ5o7JPuh4CbQ=="}' 2024-09-18T15:24:07.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c6407b5b-79cb-40e5-b9e4-482798493fa3 -i td/osd-recovery-stats/0/new.json 2024-09-18T15:24:07.463 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:24:07.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-09-18T15:24:07.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCX8OpmRoi1CRAAlVUBB29xbwJ5o7JPuh4CbQ== --osd-uuid c6407b5b-79cb-40e5-b9e4-482798493fa3 2024-09-18T15:24:07.497 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:24:07.496+0000 7f707be69440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:24:07.507 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:24:07.506+0000 7f707be69440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:24:07.509 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:24:07.507+0000 7f707be69440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:24:07.509 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:24:07.508+0000 7f707be69440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-18T15:24:07.509 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:24:07.508+0000 7f707be69440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-18T15:24:07.509 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:24:07.508+0000 7f707be69440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-18T15:24:07.539 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:24:07.538+0000 7f707be69440 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-09-18T15:24:12.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-09-18T15:24:12.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:24:12.971 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T15:24:12.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T15:24:12.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:24:13.488 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T15:24:13.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T15:24:13.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:24:13.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:24:13.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:24:13.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:24:13.511 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:24:13.509+0000 7f62c1846440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:24:13.513 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:24:13.512+0000 7f62c1846440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:24:13.514 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:24:13.513+0000 7f62c1846440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:24:13.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T15:24:13.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:24:13.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:24:13.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:24:13.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:24:13.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:13.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:24:13.857 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:24:13.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:13.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:24:14.032 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:24:14.031+0000 7f62c1846440 -1 Falling back to public interface 2024-09-18T15:24:14.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:15.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:15.218 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:24:15.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:15.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:24:15.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:15.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:24:15.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:16.501 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:24:16.499+0000 7f62c1846440 -1 osd.0 0 log_to_monitors true 2024-09-18T15:24:16.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:16.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:16.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:24:16.638 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:24:16.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:16.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:24:17.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:18.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:18.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:18.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:24:18.252 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:24:18.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:18.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:24:18.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:19.648 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:19.648 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:19.649 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:24:19.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:24:19.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:19.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:24:20.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:21.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:21.027 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:24:21.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:21.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:24:21.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:21.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:24:21.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:22.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:22.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:22.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:24:22.520 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:24:22.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:22.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:24:22.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:23.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:23.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:23.903 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:24:23.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:24:23.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:23.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:24:24.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:25.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:25.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:25.352 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:24:25.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:24:25.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:25.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:24:25.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:26.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:26.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:26.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:24:26.771 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:24:26.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:26.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:24:27.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:28.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:28.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:28.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:24:28.209 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:24:28.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:28.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:24:29.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:30.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:30.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:30.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:24:30.005 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:24:30.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:30.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:24:30.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:31.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:31.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:31.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:24:31.440 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:24:31.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:31.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:24:32.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:33.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:33.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:33.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:24:33.091 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:24:33.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:33.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:24:33.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:34.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:34.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:34.721 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:24:34.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:24:34.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:34.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:24:35.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:36.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:36.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:36.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:24:36.103 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:24:36.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:36.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:24:36.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:37.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:37.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:37.842 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:24:37.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:24:37.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:37.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:24:38.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:39.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:39.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:39.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:24:39.474 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T15:24:39.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:39.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:24:39.862 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/337190,v1:127.0.0.1:6803/337190] [v2:127.0.0.1:6804/337190,v1:127.0.0.1:6805/337190] exists,up c6407b5b-79cb-40e5-b9e4-482798493fa3 2024-09-18T15:24:39.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:24:39.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:24:39.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:24:39.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:124: do_recovery_out1: run_osd td/osd-recovery-stats 1 2024-09-18T15:24:39.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:24:39.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:24:39.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T15:24:39.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:24:39.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-09-18T15:24:39.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:24:39.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:24:39.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:24:39.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:24:39.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-09-18T15:24:39.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-09-18T15:24:39.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:24:39.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:24:39.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:24:39.868 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:24:39.868 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:24:39.868 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:24:39.868 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:24:39.868 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:24:39.868 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:24:39.868 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:24:39.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:24:39.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:24:39.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:24:39.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:24:39.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:24:39.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:24:39.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:24:39.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:24:39.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:24:39.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:24:39.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:24:39.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:24:39.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-09-18T15:24:39.869 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:24:39.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=034db177-4965-4e7d-b647-e5be97ac976d 2024-09-18T15:24:39.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 034db177-4965-4e7d-b647-e5be97ac976d' 2024-09-18T15:24:39.869 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 034db177-4965-4e7d-b647-e5be97ac976d 2024-09-18T15:24:39.870 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:24:39.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC38OpmZ5V0NBAA/ZYn/444lwmZ87eL4vLGsQ== 2024-09-18T15:24:39.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC38OpmZ5V0NBAA/ZYn/444lwmZ87eL4vLGsQ=="}' 2024-09-18T15:24:39.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 034db177-4965-4e7d-b647-e5be97ac976d -i td/osd-recovery-stats/1/new.json 2024-09-18T15:24:40.402 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:24:40.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-09-18T15:24:40.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC38OpmZ5V0NBAA/ZYn/444lwmZ87eL4vLGsQ== --osd-uuid 034db177-4965-4e7d-b647-e5be97ac976d 2024-09-18T15:24:40.438 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:24:40.437+0000 7f661ce08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:24:40.440 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:24:40.439+0000 7f661ce08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:24:40.441 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:24:40.440+0000 7f661ce08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:24:40.442 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:24:40.440+0000 7f661ce08440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-18T15:24:40.442 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:24:40.440+0000 7f661ce08440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-18T15:24:40.442 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:24:40.440+0000 7f661ce08440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-18T15:24:40.527 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:24:40.525+0000 7f661ce08440 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-09-18T15:24:46.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-09-18T15:24:46.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:24:46.410 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T15:24:46.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T15:24:46.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:24:46.921 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T15:24:46.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T15:24:46.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:24:46.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:24:46.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:24:46.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:24:46.944 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:24:46.943+0000 7fb7e7410440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:24:46.947 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:24:46.945+0000 7fb7e7410440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:24:46.947 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:24:46.946+0000 7fb7e7410440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:24:47.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T15:24:47.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:24:47.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:24:47.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:24:47.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:24:47.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:47.308 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:24:47.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:24:47.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:47.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:24:47.466 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:24:47.465+0000 7fb7e7410440 -1 Falling back to public interface 2024-09-18T15:24:47.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:48.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:48.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:48.672 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:24:48.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:24:48.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:48.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:24:49.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:49.933 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:24:49.932+0000 7fb7e7410440 -1 osd.1 0 log_to_monitors true 2024-09-18T15:24:50.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:50.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:50.069 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:24:50.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:24:50.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:50.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:24:50.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:51.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:51.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:51.594 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:24:51.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:24:51.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:51.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:24:51.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:52.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:52.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:52.977 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:24:52.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:24:52.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:52.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:24:53.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:54.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:54.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:54.450 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:24:54.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:24:54.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:54.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:24:54.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:55.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:55.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:55.879 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:24:55.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:24:55.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:55.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:24:56.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:57.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:57.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:57.270 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:24:57.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:24:57.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:57.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:24:57.826 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:24:58.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:24:58.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:24:58.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:24:58.828 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:24:58.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:24:58.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:24:59.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:00.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:00.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:00.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:25:00.246 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:25:00.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:00.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:25:00.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:01.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:01.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:01.698 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:25:01.699 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:25:01.699 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:01.699 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:25:02.247 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:03.248 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:03.248 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:03.248 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:25:03.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:25:03.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:03.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:25:03.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:04.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:04.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:04.891 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:25:04.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:25:04.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:04.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:25:05.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:06.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:06.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:06.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:25:06.417 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:25:06.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:06.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:25:07.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:08.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:08.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:08.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:25:08.089 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:25:08.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:08.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:25:08.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:09.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:09.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:09.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:25:09.871 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:25:09.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:09.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:25:10.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:11.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:11.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:11.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:25:11.425 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:25:11.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:11.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:25:12.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:13.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:13.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:13.019 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T15:25:13.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:25:13.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:13.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:25:13.396 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/338944,v1:127.0.0.1:6811/338944] [v2:127.0.0.1:6812/338944,v1:127.0.0.1:6813/338944] exists,up 034db177-4965-4e7d-b647-e5be97ac976d 2024-09-18T15:25:13.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:25:13.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:25:13.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:25:13.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:125: do_recovery_out1: run_osd td/osd-recovery-stats 2 2024-09-18T15:25:13.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:25:13.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:25:13.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T15:25:13.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:25:13.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-09-18T15:25:13.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:25:13.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:25:13.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:25:13.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:25:13.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-09-18T15:25:13.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-09-18T15:25:13.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:25:13.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:25:13.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:25:13.401 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:25:13.401 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:25:13.401 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:25:13.401 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:25:13.401 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:25:13.401 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:25:13.402 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:25:13.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:25:13.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:25:13.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:25:13.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:25:13.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:25:13.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:25:13.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:25:13.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:25:13.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:25:13.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:25:13.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:25:13.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:25:13.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-09-18T15:25:13.403 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:25:13.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=62f03d87-b2c1-44f2-9568-3fc6d8bb1ff2 2024-09-18T15:25:13.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 62f03d87-b2c1-44f2-9568-3fc6d8bb1ff2' 2024-09-18T15:25:13.403 INFO:tasks.workunit.client.0.smithi047.stdout:add osd2 62f03d87-b2c1-44f2-9568-3fc6d8bb1ff2 2024-09-18T15:25:13.403 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:25:13.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDZ8OpmqIWjGBAAhSOPd1QFf40KALScfhI2Dw== 2024-09-18T15:25:13.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDZ8OpmqIWjGBAAhSOPd1QFf40KALScfhI2Dw=="}' 2024-09-18T15:25:13.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 62f03d87-b2c1-44f2-9568-3fc6d8bb1ff2 -i td/osd-recovery-stats/2/new.json 2024-09-18T15:25:13.961 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:25:13.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-09-18T15:25:13.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDZ8OpmqIWjGBAAhSOPd1QFf40KALScfhI2Dw== --osd-uuid 62f03d87-b2c1-44f2-9568-3fc6d8bb1ff2 2024-09-18T15:25:13.997 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:25:13.996+0000 7f82aaa0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:25:13.999 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:25:13.998+0000 7f82aaa0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:25:14.000 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:25:13.999+0000 7f82aaa0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:25:14.000 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:25:13.999+0000 7f82aaa0b440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-18T15:25:14.000 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:25:13.999+0000 7f82aaa0b440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-18T15:25:14.000 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:25:13.999+0000 7f82aaa0b440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-18T15:25:14.043 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:25:14.042+0000 7f82aaa0b440 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-09-18T15:25:19.993 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-09-18T15:25:19.993 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:25:19.994 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd2 key to auth repository 2024-09-18T15:25:19.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T15:25:19.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:25:20.508 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T15:25:20.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T15:25:20.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:25:20.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:25:20.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:25:20.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:25:20.531 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:25:20.530+0000 7f5f51250440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:25:20.533 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:25:20.532+0000 7f5f51250440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:25:20.534 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:25:20.533+0000 7f5f51250440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:25:20.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T15:25:20.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:25:20.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:25:20.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:25:20.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:25:20.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:20.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:25:20.874 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:25:20.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:20.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:25:21.050 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:25:21.049+0000 7f5f51250440 -1 Falling back to public interface 2024-09-18T15:25:21.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:22.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:22.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:22.239 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:25:22.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:25:22.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:22.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:25:22.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:23.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:23.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:23.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:25:23.612 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:25:23.612 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:23.612 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:25:23.637 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:25:23.635+0000 7f5f51250440 -1 osd.2 0 log_to_monitors true 2024-09-18T15:25:24.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:25.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:25.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:25.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:25:25.122 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:25:25.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:25.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:25:25.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:26.551 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:26.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:26.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:25:26.552 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:25:26.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:26.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:25:26.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:27.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:27.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:27.965 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:25:27.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:25:27.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:27.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:25:28.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:29.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:29.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:29.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:25:29.429 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:25:29.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:29.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:25:29.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:30.826 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:30.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:30.827 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:25:30.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:25:30.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:30.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:25:31.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:32.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:32.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:32.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:25:32.404 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:25:32.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:32.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:25:32.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:33.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:33.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:33.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:25:33.814 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:25:33.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:33.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:25:34.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:35.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:35.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:35.228 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:25:35.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:25:35.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:35.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:25:35.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:36.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:36.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:36.625 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:25:36.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:25:36.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:36.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:25:37.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:38.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:38.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:38.126 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:25:38.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:25:38.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:38.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:25:38.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:39.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:39.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:39.935 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:25:39.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:25:39.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:39.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:25:40.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:41.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:41.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:41.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:25:41.697 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:25:41.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:41.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:25:42.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:43.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:43.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:43.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:25:43.341 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:25:43.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:43.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:25:44.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:45.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:45.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:45.010 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:25:45.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:25:45.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:45.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:25:45.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:46.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:46.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:46.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:25:46.848 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T15:25:46.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:46.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:25:47.228 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/340699,v1:127.0.0.1:6819/340699] [v2:127.0.0.1:6820/340699,v1:127.0.0.1:6821/340699] exists,up 62f03d87-b2c1-44f2-9568-3fc6d8bb1ff2 2024-09-18T15:25:47.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:25:47.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:25:47.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:25:47.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:126: do_recovery_out1: run_osd td/osd-recovery-stats 3 2024-09-18T15:25:47.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:25:47.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:25:47.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-18T15:25:47.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:25:47.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-09-18T15:25:47.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:25:47.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:25:47.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:25:47.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:25:47.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-09-18T15:25:47.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-09-18T15:25:47.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:25:47.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:25:47.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:25:47.233 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:25:47.234 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:25:47.234 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:25:47.234 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:25:47.234 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:25:47.234 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:25:47.234 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:25:47.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:25:47.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:25:47.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:25:47.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:25:47.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:25:47.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:25:47.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:25:47.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:25:47.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:25:47.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:25:47.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:25:47.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:25:47.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-09-18T15:25:47.235 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:25:47.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b2a4315b-a981-4b74-9b77-5f938110cace 2024-09-18T15:25:47.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 b2a4315b-a981-4b74-9b77-5f938110cace' 2024-09-18T15:25:47.235 INFO:tasks.workunit.client.0.smithi047.stdout:add osd3 b2a4315b-a981-4b74-9b77-5f938110cace 2024-09-18T15:25:47.236 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:25:47.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD78OpmXUijDhAA5TlnwkB3YtXoSmZqpHKMvQ== 2024-09-18T15:25:47.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD78OpmXUijDhAA5TlnwkB3YtXoSmZqpHKMvQ=="}' 2024-09-18T15:25:47.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b2a4315b-a981-4b74-9b77-5f938110cace -i td/osd-recovery-stats/3/new.json 2024-09-18T15:25:47.757 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:25:47.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-09-18T15:25:47.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD78OpmXUijDhAA5TlnwkB3YtXoSmZqpHKMvQ== --osd-uuid b2a4315b-a981-4b74-9b77-5f938110cace 2024-09-18T15:25:47.794 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:25:47.793+0000 7f9242039440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:25:47.796 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:25:47.795+0000 7f9242039440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:25:47.797 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:25:47.796+0000 7f9242039440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:25:47.798 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:25:47.796+0000 7f9242039440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-18T15:25:47.798 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:25:47.796+0000 7f9242039440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-18T15:25:47.798 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:25:47.796+0000 7f9242039440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-18T15:25:47.873 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:25:47.871+0000 7f9242039440 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-09-18T15:25:53.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-09-18T15:25:53.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:25:53.465 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd3 key to auth repository 2024-09-18T15:25:53.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-18T15:25:53.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:25:53.976 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.3 2024-09-18T15:25:53.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-18T15:25:53.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:25:53.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:25:53.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:25:53.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:25:54.000 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:25:53.998+0000 7f6916d66440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:25:54.002 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:25:54.000+0000 7f6916d66440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:25:54.003 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:25:54.002+0000 7f6916d66440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:25:54.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-18T15:25:54.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:25:54.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T15:25:54.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:25:54.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:25:54.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:54.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:25:54.341 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:25:54.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:54.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:25:54.520 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:25:54.519+0000 7f6916d66440 -1 Falling back to public interface 2024-09-18T15:25:54.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:55.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:55.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:55.745 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:25:55.746 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:25:55.746 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:55.746 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:25:56.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:57.047 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:25:57.046+0000 7f6916d66440 -1 osd.3 0 log_to_monitors true 2024-09-18T15:25:57.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:57.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:57.164 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:25:57.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:25:57.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:57.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:25:57.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:25:58.826 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:25:58.826 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:25:58.826 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:25:58.826 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:25:58.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:25:58.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:25:59.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:00.231 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:00.231 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:00.231 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:26:00.231 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:26:00.231 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:00.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:26:00.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:01.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:01.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:01.621 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:26:01.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:26:01.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:01.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:26:02.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:03.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:03.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:03.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:26:03.039 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:26:03.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:03.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:26:03.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:04.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:04.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:04.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:26:04.464 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:26:04.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:04.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:26:04.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:05.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:05.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:05.881 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:26:05.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:26:05.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:05.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:26:06.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:07.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:07.285 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:07.285 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:26:07.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:26:07.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:07.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:26:07.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:08.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:08.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:08.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:26:08.887 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:26:08.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:08.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:26:09.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:10.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:10.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:10.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:26:10.802 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:26:10.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:10.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:26:11.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:12.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:12.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:12.314 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:26:12.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:26:12.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:12.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:26:13.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:14.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:14.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:14.360 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:26:14.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:26:14.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:14.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:26:15.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:16.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:16.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:16.033 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:26:16.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:26:16.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:16.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:26:16.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:17.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:17.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:17.970 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:26:17.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:26:17.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:17.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:26:18.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:19.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:19.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:19.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:26:19.721 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:26:19.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:19.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:26:20.095 INFO:tasks.workunit.client.0.smithi047.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/342453,v1:127.0.0.1:6827/342453] [v2:127.0.0.1:6828/342453,v1:127.0.0.1:6829/342453] exists,up b2a4315b-a981-4b74-9b77-5f938110cace 2024-09-18T15:26:20.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:26:20.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:26:20.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:26:20.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:127: do_recovery_out1: run_osd td/osd-recovery-stats 4 2024-09-18T15:26:20.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:26:20.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:26:20.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-18T15:26:20.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:26:20.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2024-09-18T15:26:20.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:26:20.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:26:20.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:26:20.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:26:20.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2024-09-18T15:26:20.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2024-09-18T15:26:20.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:26:20.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:26:20.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:26:20.100 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:26:20.100 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:26:20.100 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:26:20.100 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:26:20.100 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:26:20.100 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:26:20.100 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:26:20.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:26:20.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:26:20.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:26:20.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:26:20.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:26:20.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:26:20.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:26:20.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:26:20.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:26:20.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:26:20.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:26:20.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:26:20.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2024-09-18T15:26:20.101 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:26:20.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2f475542-0201-45c5-b8c7-75175ec49f28 2024-09-18T15:26:20.102 INFO:tasks.workunit.client.0.smithi047.stdout:add osd4 2f475542-0201-45c5-b8c7-75175ec49f28 2024-09-18T15:26:20.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 2f475542-0201-45c5-b8c7-75175ec49f28' 2024-09-18T15:26:20.102 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:26:20.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAc8epmJHyyBhAAUe7BZ4tOc3CzOiUOgh+gPw== 2024-09-18T15:26:20.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAc8epmJHyyBhAAUe7BZ4tOc3CzOiUOgh+gPw=="}' 2024-09-18T15:26:20.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2f475542-0201-45c5-b8c7-75175ec49f28 -i td/osd-recovery-stats/4/new.json 2024-09-18T15:26:20.788 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:26:20.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2024-09-18T15:26:20.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAc8epmJHyyBhAAUe7BZ4tOc3CzOiUOgh+gPw== --osd-uuid 2f475542-0201-45c5-b8c7-75175ec49f28 2024-09-18T15:26:20.825 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:26:20.824+0000 7f217444f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:26:20.827 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:26:20.826+0000 7f217444f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:26:20.828 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:26:20.827+0000 7f217444f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:26:20.828 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:26:20.827+0000 7f217444f440 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-18T15:26:20.828 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:26:20.827+0000 7f217444f440 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-18T15:26:20.828 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:26:20.827+0000 7f217444f440 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-18T15:26:20.912 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:26:20.911+0000 7f217444f440 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2024-09-18T15:26:26.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2024-09-18T15:26:26.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:26:26.753 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd4 key to auth repository 2024-09-18T15:26:26.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-18T15:26:26.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:26:27.267 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.4 2024-09-18T15:26:27.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-18T15:26:27.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:26:27.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:26:27.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:26:27.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:26:27.291 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:26:27.290+0000 7fa0df426440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:26:27.312 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:26:27.311+0000 7fa0df426440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:26:27.313 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:26:27.312+0000 7fa0df426440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:26:27.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-18T15:26:27.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:26:27.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-18T15:26:27.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:26:27.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:26:27.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:27.718 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:26:27.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:26:27.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:27.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:26:27.846 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:26:27.845+0000 7fa0df426440 -1 Falling back to public interface 2024-09-18T15:26:28.095 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:29.096 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:29.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:29.097 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:26:29.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:26:29.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:29.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:26:29.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:30.334 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:26:30.332+0000 7fa0df426440 -1 osd.4 0 log_to_monitors true 2024-09-18T15:26:30.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:30.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:30.480 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:26:30.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:26:30.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:30.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:26:31.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:32.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:32.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:32.112 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:26:32.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:26:32.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:32.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:26:32.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:33.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:33.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:33.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:26:33.509 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:26:33.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:33.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:26:33.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:34.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:34.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:34.974 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:26:34.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:26:34.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:34.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:26:35.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:36.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:36.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:36.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:26:36.530 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:26:36.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:36.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:26:36.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:37.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:37.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:37.927 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:26:37.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:26:37.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:37.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:26:38.355 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:39.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:39.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:39.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:26:39.357 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:26:39.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:39.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:26:39.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:40.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:40.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:40.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:26:40.808 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:26:40.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:40.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:26:41.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:42.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:42.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:42.322 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:26:42.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:26:42.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:42.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:26:42.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:43.714 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:43.714 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:43.714 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:26:43.715 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:26:43.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:43.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:26:44.406 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:45.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:45.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:45.408 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:26:45.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:26:45.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:45.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:26:46.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:47.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:47.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:47.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:26:47.207 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:26:47.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:47.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:26:47.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:48.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:48.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:48.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:26:48.975 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:26:48.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:48.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:26:49.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:50.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:50.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:50.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:26:50.404 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:26:50.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:50.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:26:51.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:52.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:52.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:52.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:26:52.279 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:26:52.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:52.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:26:52.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:26:53.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:26:53.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:26:53.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:26:53.999 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T15:26:54.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:26:54.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:26:54.394 INFO:tasks.workunit.client.0.smithi047.stdout:osd.4 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/344182,v1:127.0.0.1:6835/344182] [v2:127.0.0.1:6836/344182,v1:127.0.0.1:6837/344182] exists,up 2f475542-0201-45c5-b8c7-75175ec49f28 2024-09-18T15:26:54.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:26:54.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:26:54.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:26:54.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:128: do_recovery_out1: run_osd td/osd-recovery-stats 5 2024-09-18T15:26:54.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:26:54.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:26:54.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-18T15:26:54.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:26:54.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2024-09-18T15:26:54.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:26:54.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:26:54.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:26:54.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:26:54.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2024-09-18T15:26:54.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2024-09-18T15:26:54.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:26:54.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:26:54.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:26:54.399 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:26:54.399 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:26:54.399 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:26:54.399 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:26:54.399 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:26:54.399 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:26:54.399 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:26:54.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:26:54.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:26:54.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:26:54.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:26:54.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:26:54.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:26:54.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:26:54.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:26:54.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:26:54.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:26:54.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:26:54.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:26:54.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2024-09-18T15:26:54.400 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:26:54.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a1fe142c-2311-4da3-a7af-0c70c0d0aa2c 2024-09-18T15:26:54.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 a1fe142c-2311-4da3-a7af-0c70c0d0aa2c' 2024-09-18T15:26:54.401 INFO:tasks.workunit.client.0.smithi047.stdout:add osd5 a1fe142c-2311-4da3-a7af-0c70c0d0aa2c 2024-09-18T15:26:54.401 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:26:54.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA+8epm/3aHGBAAlGaeiQnP9s9ZCOyKBlJaIg== 2024-09-18T15:26:54.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA+8epm/3aHGBAAlGaeiQnP9s9ZCOyKBlJaIg=="}' 2024-09-18T15:26:54.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a1fe142c-2311-4da3-a7af-0c70c0d0aa2c -i td/osd-recovery-stats/5/new.json 2024-09-18T15:26:54.919 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:26:54.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2024-09-18T15:26:54.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA+8epm/3aHGBAAlGaeiQnP9s9ZCOyKBlJaIg== --osd-uuid a1fe142c-2311-4da3-a7af-0c70c0d0aa2c 2024-09-18T15:26:54.957 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:26:54.955+0000 7f43a8c75440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:26:54.960 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:26:54.959+0000 7f43a8c75440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:26:54.961 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:26:54.960+0000 7f43a8c75440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:26:54.961 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:26:54.960+0000 7f43a8c75440 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-18T15:26:54.962 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:26:54.960+0000 7f43a8c75440 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-18T15:26:54.962 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:26:54.960+0000 7f43a8c75440 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-18T15:26:55.060 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:26:55.059+0000 7f43a8c75440 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2024-09-18T15:27:01.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2024-09-18T15:27:01.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:27:01.087 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd5 key to auth repository 2024-09-18T15:27:01.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-18T15:27:01.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:27:01.601 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.5 2024-09-18T15:27:01.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-18T15:27:01.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:27:01.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:27:01.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:27:01.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:27:01.624 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:27:01.623+0000 7f8b2028d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:27:01.646 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:27:01.645+0000 7f8b2028d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:27:01.647 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:27:01.646+0000 7f8b2028d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:27:01.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-18T15:27:01.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:27:01.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-18T15:27:01.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:27:01.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:27:01.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:27:01.996 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:27:01.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:27:01.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:27:01.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:27:02.176 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:27:02.175+0000 7f8b2028d440 -1 Falling back to public interface 2024-09-18T15:27:02.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:27:03.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:27:03.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:27:03.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:27:03.405 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:27:03.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:27:03.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:27:03.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:27:04.694 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:27:04.693+0000 7f8b2028d440 -1 osd.5 0 log_to_monitors true 2024-09-18T15:27:04.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:27:04.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:27:04.823 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:27:04.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:27:04.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:27:04.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:27:05.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:27:06.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:27:06.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:27:06.306 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:27:06.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:27:06.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:27:06.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:27:06.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:27:07.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:27:07.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:27:07.770 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:27:07.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:27:07.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:27:07.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:27:08.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:27:09.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:27:09.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:27:09.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:27:09.188 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:27:09.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:27:09.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:27:09.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:27:10.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:27:10.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:27:10.574 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:27:10.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:27:10.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:27:10.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:27:10.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:27:11.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:27:11.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:27:11.981 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:27:11.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:27:11.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:27:11.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:27:12.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:27:13.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:27:13.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:27:13.370 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:27:13.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:27:13.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:27:13.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:27:13.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:27:14.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:27:14.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:27:14.769 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:27:14.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:27:14.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:27:14.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:27:15.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:27:16.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:27:16.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:27:16.200 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:27:16.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:27:16.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:27:16.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:27:16.949 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:27:17.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:27:17.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:27:17.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:27:17.951 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:27:17.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:27:17.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:27:18.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:27:19.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:27:19.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:27:19.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:27:19.850 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:27:19.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:27:19.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:27:20.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:27:21.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:27:21.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:27:21.602 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:27:21.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:27:21.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:27:21.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:27:22.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:27:23.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:27:23.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:27:23.654 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:27:23.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:27:23.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:27:23.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:27:24.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:27:25.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:27:25.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:27:25.535 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:27:25.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:27:25.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:27:25.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:27:26.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:27:27.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:27:27.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:27:27.286 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:27:27.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:27:27.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:27:27.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:27:27.710 INFO:tasks.workunit.client.0.smithi047.stdout:osd.5 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/345936,v1:127.0.0.1:6843/345936] [v2:127.0.0.1:6844/345936,v1:127.0.0.1:6845/345936] exists,up a1fe142c-2311-4da3-a7af-0c70c0d0aa2c 2024-09-18T15:27:27.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:27:27.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:27:27.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:27:27.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:130: do_recovery_out1: '[' replicated = erasure ']' 2024-09-18T15:27:27.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:135: do_recovery_out1: create_pool test 1 1 replicated 2024-09-18T15:27:27.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 replicated 2024-09-18T15:27:28.315 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test' already exists 2024-09-18T15:27:28.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:27:29.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:138: do_recovery_out1: wait_for_clean 2024-09-18T15:27:29.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:27:29.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:27:29.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:27:29.328 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:27:29.329 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:27:29.329 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:27:29.329 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:27:29.329 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:27:29.329 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:27:29.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:27:29.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:27:29.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:27:29.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:27:29.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:27:29.430 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:27:29.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:27:29.913 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:27:29.913 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T15:27:29.913 INFO:tasks.workunit.client.0.smithi047.stderr:3 2024-09-18T15:27:29.914 INFO:tasks.workunit.client.0.smithi047.stderr:4 2024-09-18T15:27:29.914 INFO:tasks.workunit.client.0.smithi047.stderr:5' 2024-09-18T15:27:29.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:27:29.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:29.914 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:27:30.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836522 2024-09-18T15:27:30.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836522 2024-09-18T15:27:30.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836522' 2024-09-18T15:27:30.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:30.121 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:27:30.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672996 2024-09-18T15:27:30.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672996 2024-09-18T15:27:30.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836522 1-42949672996' 2024-09-18T15:27:30.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:30.229 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:27:30.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509466 2024-09-18T15:27:30.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509466 2024-09-18T15:27:30.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836522 1-42949672996 2-64424509466' 2024-09-18T15:27:30.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:30.397 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:27:30.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378642 2024-09-18T15:27:30.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378642 2024-09-18T15:27:30.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836522 1-42949672996 2-64424509466 3-81604378642' 2024-09-18T15:27:30.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:30.505 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:27:30.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=103079215116 2024-09-18T15:27:30.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 103079215116 2024-09-18T15:27:30.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836522 1-42949672996 2-64424509466 3-81604378642 4-103079215116' 2024-09-18T15:27:30.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:27:30.613 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:27:30.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=124554051591 2024-09-18T15:27:30.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 124554051591 2024-09-18T15:27:30.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836522 1-42949672996 2-64424509466 3-81604378642 4-103079215116 5-124554051591' 2024-09-18T15:27:30.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:30.721 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836522 2024-09-18T15:27:30.721 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:30.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:27:30.722 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836522 2024-09-18T15:27:30.723 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:30.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836522 2024-09-18T15:27:30.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836522' 2024-09-18T15:27:30.724 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836522 2024-09-18T15:27:30.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:27:31.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836522 -lt 21474836522 2024-09-18T15:27:31.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:31.090 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672996 2024-09-18T15:27:31.090 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:31.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:27:31.091 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672996 2024-09-18T15:27:31.091 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:31.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672996 2024-09-18T15:27:31.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672996' 2024-09-18T15:27:31.093 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672996 2024-09-18T15:27:31.093 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:27:31.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672996 -lt 42949672996 2024-09-18T15:27:31.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:31.457 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509466 2024-09-18T15:27:31.457 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:31.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:27:31.458 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509466 2024-09-18T15:27:31.458 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:31.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509466 2024-09-18T15:27:31.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509466' 2024-09-18T15:27:31.460 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509466 2024-09-18T15:27:31.460 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:27:31.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509466 -lt 64424509466 2024-09-18T15:27:31.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:31.863 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-81604378642 2024-09-18T15:27:31.863 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:31.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:27:31.865 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-81604378642 2024-09-18T15:27:31.865 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:31.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378642 2024-09-18T15:27:31.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 81604378642' 2024-09-18T15:27:31.866 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 81604378642 2024-09-18T15:27:31.866 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:27:32.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378643 -lt 81604378642 2024-09-18T15:27:32.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:32.282 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-103079215116 2024-09-18T15:27:32.282 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:32.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:27:32.283 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-103079215116 2024-09-18T15:27:32.283 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:32.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=103079215116 2024-09-18T15:27:32.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 103079215116' 2024-09-18T15:27:32.285 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.4 seq 103079215116 2024-09-18T15:27:32.285 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:27:32.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 103079215116 -lt 103079215116 2024-09-18T15:27:32.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:27:32.652 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-124554051591 2024-09-18T15:27:32.653 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:27:32.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:27:32.654 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-124554051591 2024-09-18T15:27:32.654 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:27:32.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=124554051591 2024-09-18T15:27:32.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 124554051591' 2024-09-18T15:27:32.655 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.5 seq 124554051591 2024-09-18T15:27:32.655 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:27:33.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051591 -lt 124554051591 2024-09-18T15:27:33.093 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:27:33.093 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:27:33.094 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:27:33.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:27:33.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:27:33.586 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:27:33.586 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:27:33.586 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:27:33.587 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:27:33.587 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:27:33.587 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:27:33.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:27:33.971 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:27:33.971 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:27:33.971 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:27:34.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:27:34.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:27:34.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:27:34.507 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: seq 1 200 2024-09-18T15:27:34.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:34.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj1 /dev/null 2024-09-18T15:27:34.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:34.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj2 /dev/null 2024-09-18T15:27:34.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:34.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj3 /dev/null 2024-09-18T15:27:34.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:34.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj4 /dev/null 2024-09-18T15:27:35.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:35.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj5 /dev/null 2024-09-18T15:27:35.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:35.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj6 /dev/null 2024-09-18T15:27:35.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:35.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj7 /dev/null 2024-09-18T15:27:35.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:35.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj8 /dev/null 2024-09-18T15:27:35.483 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:35.483 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj9 /dev/null 2024-09-18T15:27:35.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:35.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj10 /dev/null 2024-09-18T15:27:35.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:35.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj11 /dev/null 2024-09-18T15:27:35.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:35.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj12 /dev/null 2024-09-18T15:27:35.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:35.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj13 /dev/null 2024-09-18T15:27:36.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:36.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj14 /dev/null 2024-09-18T15:27:36.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:36.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj15 /dev/null 2024-09-18T15:27:36.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:36.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj16 /dev/null 2024-09-18T15:27:36.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:36.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj17 /dev/null 2024-09-18T15:27:36.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:36.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj18 /dev/null 2024-09-18T15:27:36.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:36.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj19 /dev/null 2024-09-18T15:27:36.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:36.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj20 /dev/null 2024-09-18T15:27:36.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:36.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj21 /dev/null 2024-09-18T15:27:36.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:36.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj22 /dev/null 2024-09-18T15:27:37.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:37.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj23 /dev/null 2024-09-18T15:27:37.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:37.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj24 /dev/null 2024-09-18T15:27:37.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:37.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj25 /dev/null 2024-09-18T15:27:37.419 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:37.419 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj26 /dev/null 2024-09-18T15:27:37.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:37.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj27 /dev/null 2024-09-18T15:27:37.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:37.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj28 /dev/null 2024-09-18T15:27:37.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:37.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj29 /dev/null 2024-09-18T15:27:37.862 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:37.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj30 /dev/null 2024-09-18T15:27:38.021 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:38.022 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj31 /dev/null 2024-09-18T15:27:38.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:38.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj32 /dev/null 2024-09-18T15:27:38.248 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:38.248 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj33 /dev/null 2024-09-18T15:27:38.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:38.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj34 /dev/null 2024-09-18T15:27:38.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:38.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj35 /dev/null 2024-09-18T15:27:38.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:38.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj36 /dev/null 2024-09-18T15:27:38.666 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:38.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj37 /dev/null 2024-09-18T15:27:38.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:38.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj38 /dev/null 2024-09-18T15:27:38.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:38.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj39 /dev/null 2024-09-18T15:27:39.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:39.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj40 /dev/null 2024-09-18T15:27:39.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:39.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj41 /dev/null 2024-09-18T15:27:39.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:39.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj42 /dev/null 2024-09-18T15:27:39.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:39.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj43 /dev/null 2024-09-18T15:27:39.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:39.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj44 /dev/null 2024-09-18T15:27:39.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:39.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj45 /dev/null 2024-09-18T15:27:39.714 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:39.714 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj46 /dev/null 2024-09-18T15:27:39.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:39.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj47 /dev/null 2024-09-18T15:27:39.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:39.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj48 /dev/null 2024-09-18T15:27:40.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:40.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj49 /dev/null 2024-09-18T15:27:40.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:40.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj50 /dev/null 2024-09-18T15:27:40.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:40.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj51 /dev/null 2024-09-18T15:27:40.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:40.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj52 /dev/null 2024-09-18T15:27:40.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:40.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj53 /dev/null 2024-09-18T15:27:40.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:40.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj54 /dev/null 2024-09-18T15:27:40.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:40.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj55 /dev/null 2024-09-18T15:27:40.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:40.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj56 /dev/null 2024-09-18T15:27:40.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:40.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj57 /dev/null 2024-09-18T15:27:41.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:41.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj58 /dev/null 2024-09-18T15:27:41.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:41.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj59 /dev/null 2024-09-18T15:27:41.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:41.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj60 /dev/null 2024-09-18T15:27:41.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:41.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj61 /dev/null 2024-09-18T15:27:41.599 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:41.599 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj62 /dev/null 2024-09-18T15:27:41.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:41.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj63 /dev/null 2024-09-18T15:27:41.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:41.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj64 /dev/null 2024-09-18T15:27:41.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:41.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj65 /dev/null 2024-09-18T15:27:42.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:42.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj66 /dev/null 2024-09-18T15:27:42.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:42.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj67 /dev/null 2024-09-18T15:27:42.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:42.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj68 /dev/null 2024-09-18T15:27:42.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:42.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj69 /dev/null 2024-09-18T15:27:42.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:42.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj70 /dev/null 2024-09-18T15:27:42.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:42.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj71 /dev/null 2024-09-18T15:27:42.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:42.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj72 /dev/null 2024-09-18T15:27:42.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:42.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj73 /dev/null 2024-09-18T15:27:43.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:43.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj74 /dev/null 2024-09-18T15:27:43.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:43.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj75 /dev/null 2024-09-18T15:27:43.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:43.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj76 /dev/null 2024-09-18T15:27:43.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:43.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj77 /dev/null 2024-09-18T15:27:43.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:43.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj78 /dev/null 2024-09-18T15:27:43.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:43.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj79 /dev/null 2024-09-18T15:27:43.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:43.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj80 /dev/null 2024-09-18T15:27:44.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:44.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj81 /dev/null 2024-09-18T15:27:44.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:44.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj82 /dev/null 2024-09-18T15:27:44.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:44.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj83 /dev/null 2024-09-18T15:27:44.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:44.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj84 /dev/null 2024-09-18T15:27:44.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:44.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj85 /dev/null 2024-09-18T15:27:44.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:44.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj86 /dev/null 2024-09-18T15:27:44.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:44.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj87 /dev/null 2024-09-18T15:27:44.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:44.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj88 /dev/null 2024-09-18T15:27:44.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:44.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj89 /dev/null 2024-09-18T15:27:45.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:45.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj90 /dev/null 2024-09-18T15:27:45.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:45.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj91 /dev/null 2024-09-18T15:27:45.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:45.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj92 /dev/null 2024-09-18T15:27:45.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:45.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj93 /dev/null 2024-09-18T15:27:45.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:45.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj94 /dev/null 2024-09-18T15:27:45.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:45.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj95 /dev/null 2024-09-18T15:27:45.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:45.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj96 /dev/null 2024-09-18T15:27:45.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:45.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj97 /dev/null 2024-09-18T15:27:46.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:46.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj98 /dev/null 2024-09-18T15:27:46.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:46.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj99 /dev/null 2024-09-18T15:27:46.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:46.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj100 /dev/null 2024-09-18T15:27:46.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:46.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj101 /dev/null 2024-09-18T15:27:46.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:46.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj102 /dev/null 2024-09-18T15:27:46.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:46.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj103 /dev/null 2024-09-18T15:27:46.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:46.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj104 /dev/null 2024-09-18T15:27:46.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:46.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj105 /dev/null 2024-09-18T15:27:46.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:46.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj106 /dev/null 2024-09-18T15:27:47.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:47.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj107 /dev/null 2024-09-18T15:27:47.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:47.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj108 /dev/null 2024-09-18T15:27:47.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:47.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj109 /dev/null 2024-09-18T15:27:47.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:47.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj110 /dev/null 2024-09-18T15:27:47.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:47.551 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj111 /dev/null 2024-09-18T15:27:47.648 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:47.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj112 /dev/null 2024-09-18T15:27:47.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:47.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj113 /dev/null 2024-09-18T15:27:47.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:47.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj114 /dev/null 2024-09-18T15:27:48.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:48.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj115 /dev/null 2024-09-18T15:27:48.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:48.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj116 /dev/null 2024-09-18T15:27:48.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:48.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj117 /dev/null 2024-09-18T15:27:48.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:48.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj118 /dev/null 2024-09-18T15:27:48.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:48.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj119 /dev/null 2024-09-18T15:27:48.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:48.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj120 /dev/null 2024-09-18T15:27:48.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:48.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj121 /dev/null 2024-09-18T15:27:48.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:48.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj122 /dev/null 2024-09-18T15:27:48.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:48.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj123 /dev/null 2024-09-18T15:27:49.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:49.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj124 /dev/null 2024-09-18T15:27:49.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:49.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj125 /dev/null 2024-09-18T15:27:49.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:49.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj126 /dev/null 2024-09-18T15:27:49.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:49.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj127 /dev/null 2024-09-18T15:27:49.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:49.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj128 /dev/null 2024-09-18T15:27:49.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:49.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj129 /dev/null 2024-09-18T15:27:49.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:49.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj130 /dev/null 2024-09-18T15:27:49.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:49.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj131 /dev/null 2024-09-18T15:27:50.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:50.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj132 /dev/null 2024-09-18T15:27:50.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:50.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj133 /dev/null 2024-09-18T15:27:50.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:50.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj134 /dev/null 2024-09-18T15:27:50.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:50.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj135 /dev/null 2024-09-18T15:27:50.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:50.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj136 /dev/null 2024-09-18T15:27:50.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:50.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj137 /dev/null 2024-09-18T15:27:50.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:50.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj138 /dev/null 2024-09-18T15:27:50.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:50.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj139 /dev/null 2024-09-18T15:27:50.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:50.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj140 /dev/null 2024-09-18T15:27:51.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:51.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj141 /dev/null 2024-09-18T15:27:51.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:51.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj142 /dev/null 2024-09-18T15:27:51.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:51.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj143 /dev/null 2024-09-18T15:27:51.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:51.419 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj144 /dev/null 2024-09-18T15:27:51.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:51.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj145 /dev/null 2024-09-18T15:27:51.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:51.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj146 /dev/null 2024-09-18T15:27:51.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:51.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj147 /dev/null 2024-09-18T15:27:51.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:51.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj148 /dev/null 2024-09-18T15:27:51.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:51.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj149 /dev/null 2024-09-18T15:27:52.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:52.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj150 /dev/null 2024-09-18T15:27:52.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:52.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj151 /dev/null 2024-09-18T15:27:52.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:52.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj152 /dev/null 2024-09-18T15:27:52.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:52.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj153 /dev/null 2024-09-18T15:27:52.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:52.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj154 /dev/null 2024-09-18T15:27:52.666 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:52.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj155 /dev/null 2024-09-18T15:27:52.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:52.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj156 /dev/null 2024-09-18T15:27:52.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:52.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj157 /dev/null 2024-09-18T15:27:53.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:53.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj158 /dev/null 2024-09-18T15:27:53.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:53.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj159 /dev/null 2024-09-18T15:27:53.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:53.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj160 /dev/null 2024-09-18T15:27:53.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:53.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj161 /dev/null 2024-09-18T15:27:53.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:53.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj162 /dev/null 2024-09-18T15:27:53.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:53.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj163 /dev/null 2024-09-18T15:27:53.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:53.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj164 /dev/null 2024-09-18T15:27:53.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:53.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj165 /dev/null 2024-09-18T15:27:54.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:54.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj166 /dev/null 2024-09-18T15:27:54.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:54.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj167 /dev/null 2024-09-18T15:27:54.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:54.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj168 /dev/null 2024-09-18T15:27:54.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:54.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj169 /dev/null 2024-09-18T15:27:54.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:54.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj170 /dev/null 2024-09-18T15:27:54.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:54.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj171 /dev/null 2024-09-18T15:27:54.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:54.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj172 /dev/null 2024-09-18T15:27:54.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:54.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj173 /dev/null 2024-09-18T15:27:54.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:54.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj174 /dev/null 2024-09-18T15:27:55.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:55.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj175 /dev/null 2024-09-18T15:27:55.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:55.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj176 /dev/null 2024-09-18T15:27:55.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:55.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj177 /dev/null 2024-09-18T15:27:55.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:55.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj178 /dev/null 2024-09-18T15:27:55.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:55.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj179 /dev/null 2024-09-18T15:27:55.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:55.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj180 /dev/null 2024-09-18T15:27:55.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:55.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj181 /dev/null 2024-09-18T15:27:55.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:55.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj182 /dev/null 2024-09-18T15:27:55.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:55.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj183 /dev/null 2024-09-18T15:27:56.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:56.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj184 /dev/null 2024-09-18T15:27:56.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:56.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj185 /dev/null 2024-09-18T15:27:56.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:56.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj186 /dev/null 2024-09-18T15:27:56.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:56.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj187 /dev/null 2024-09-18T15:27:56.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:56.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj188 /dev/null 2024-09-18T15:27:56.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:56.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj189 /dev/null 2024-09-18T15:27:56.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:56.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj190 /dev/null 2024-09-18T15:27:56.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:56.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj191 /dev/null 2024-09-18T15:27:57.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:57.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj192 /dev/null 2024-09-18T15:27:57.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:57.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj193 /dev/null 2024-09-18T15:27:57.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:57.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj194 /dev/null 2024-09-18T15:27:57.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:57.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj195 /dev/null 2024-09-18T15:27:57.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:57.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj196 /dev/null 2024-09-18T15:27:57.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:57.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj197 /dev/null 2024-09-18T15:27:57.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:57.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj198 /dev/null 2024-09-18T15:27:57.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:57.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj199 /dev/null 2024-09-18T15:27:57.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-18T15:27:57.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj200 /dev/null 2024-09-18T15:27:58.054 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: get_primary test obj1 2024-09-18T15:27:58.054 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-18T15:27:58.055 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T15:27:58.055 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-18T15:27:58.055 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:27:58.420 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: local primary=1 2024-09-18T15:27:58.420 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: get_pg test obj1 2024-09-18T15:27:58.420 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-18T15:27:58.420 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-09-18T15:27:58.420 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-09-18T15:27:58.421 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:27:58.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: local PG=1.0 2024-09-18T15:27:58.834 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: get_not_primary test obj1 2024-09-18T15:27:58.835 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-09-18T15:27:58.835 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-09-18T15:27:58.835 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-09-18T15:27:58.835 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-18T15:27:58.835 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T15:27:58.835 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-18T15:27:58.835 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:27:59.277 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-18T15:27:59.277 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-09-18T15:27:59.277 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-18T15:27:59.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: local otherosd=0 2024-09-18T15:27:59.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:150: do_recovery_out1: ceph osd set norecover 2024-09-18T15:28:00.310 INFO:tasks.workunit.client.0.smithi047.stderr:norecover is set 2024-09-18T15:28:00.322 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: cat td/osd-recovery-stats/osd.0.pid 2024-09-18T15:28:00.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: kill 337190 2024-09-18T15:28:00.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:152: do_recovery_out1: ceph osd down osd.0 2024-09-18T15:28:00.964 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already down. 2024-09-18T15:28:00.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:153: do_recovery_out1: ceph osd out osd.0 2024-09-18T15:28:01.621 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already out. 2024-09-18T15:28:01.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:154: do_recovery_out1: ceph osd unset norecover 2024-09-18T15:28:03.221 INFO:tasks.workunit.client.0.smithi047.stderr:norecover is unset 2024-09-18T15:28:03.234 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: get_primary test obj1 2024-09-18T15:28:03.234 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-18T15:28:03.234 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T15:28:03.234 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-18T15:28:03.234 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:28:03.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-18T15:28:03.751 INFO:tasks.workunit.client.0.smithi047.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-18T15:28:03.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:156: do_recovery_out1: sleep 2 2024-09-18T15:28:05.761 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:158: do_recovery_out1: wait_for_clean 2024-09-18T15:28:05.761 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:28:05.761 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:28:05.761 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:28:05.761 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:28:05.762 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:28:05.762 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:28:05.762 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:28:05.762 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:28:05.762 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:28:05.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:28:05.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:28:05.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:28:05.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:28:05.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:28:05.864 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:28:06.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:28:06.234 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:28:06.234 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T15:28:06.234 INFO:tasks.workunit.client.0.smithi047.stderr:3 2024-09-18T15:28:06.234 INFO:tasks.workunit.client.0.smithi047.stderr:4 2024-09-18T15:28:06.234 INFO:tasks.workunit.client.0.smithi047.stderr:5' 2024-09-18T15:28:06.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:28:06.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:06.234 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:28:06.334 INFO:tasks.workunit.client.0.smithi047.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-18T15:28:06.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq= 2024-09-18T15:28:06.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z '' 2024-09-18T15:28:06.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: flush_pg_stats: continue 2024-09-18T15:28:06.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:06.337 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:28:06.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673005 2024-09-18T15:28:06.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673005 2024-09-18T15:28:06.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 1-42949673005' 2024-09-18T15:28:06.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:06.446 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:28:06.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509475 2024-09-18T15:28:06.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509475 2024-09-18T15:28:06.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 1-42949673005 2-64424509475' 2024-09-18T15:28:06.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:06.576 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:28:06.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378652 2024-09-18T15:28:06.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378652 2024-09-18T15:28:06.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 1-42949673005 2-64424509475 3-81604378652' 2024-09-18T15:28:06.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:06.685 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:28:06.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=103079215125 2024-09-18T15:28:06.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 103079215125 2024-09-18T15:28:06.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 1-42949673005 2-64424509475 3-81604378652 4-103079215125' 2024-09-18T15:28:06.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:28:06.793 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:28:06.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=124554051600 2024-09-18T15:28:06.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 124554051600 2024-09-18T15:28:06.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 1-42949673005 2-64424509475 3-81604378652 4-103079215125 5-124554051600' 2024-09-18T15:28:06.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:06.902 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673005 2024-09-18T15:28:06.902 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:06.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:28:06.903 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673005 2024-09-18T15:28:06.903 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:06.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673005 2024-09-18T15:28:06.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673005' 2024-09-18T15:28:06.905 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949673005 2024-09-18T15:28:06.905 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:28:07.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673005 -lt 42949673005 2024-09-18T15:28:07.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:07.309 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509475 2024-09-18T15:28:07.310 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:07.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:28:07.311 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509475 2024-09-18T15:28:07.311 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:07.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509475 2024-09-18T15:28:07.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509475' 2024-09-18T15:28:07.312 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509475 2024-09-18T15:28:07.313 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:28:07.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509475 -lt 64424509475 2024-09-18T15:28:07.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:07.811 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-81604378652 2024-09-18T15:28:07.811 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:07.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:28:07.812 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-81604378652 2024-09-18T15:28:07.812 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:07.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378652 2024-09-18T15:28:07.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 81604378652' 2024-09-18T15:28:07.814 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 81604378652 2024-09-18T15:28:07.814 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:28:08.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378652 -lt 81604378652 2024-09-18T15:28:08.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:08.181 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-103079215125 2024-09-18T15:28:08.181 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:08.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:28:08.182 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-103079215125 2024-09-18T15:28:08.182 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:08.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=103079215125 2024-09-18T15:28:08.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 103079215125' 2024-09-18T15:28:08.184 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.4 seq 103079215125 2024-09-18T15:28:08.184 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:28:08.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 103079215125 -lt 103079215125 2024-09-18T15:28:08.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:28:08.585 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-124554051600 2024-09-18T15:28:08.586 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:28:08.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:28:08.587 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-124554051600 2024-09-18T15:28:08.587 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:28:08.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=124554051600 2024-09-18T15:28:08.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 124554051600' 2024-09-18T15:28:08.589 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.5 seq 124554051600 2024-09-18T15:28:08.589 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:28:09.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051601 -lt 124554051600 2024-09-18T15:28:09.117 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:28:09.117 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:28:09.118 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:28:09.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:28:09.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:28:09.706 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:28:09.706 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:28:09.706 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:28:09.706 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:28:09.706 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:28:09.706 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:28:10.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:28:10.062 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:28:10.062 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:28:10.062 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:28:10.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:28:10.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-18T15:28:10.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T15:28:10.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-18T15:28:10.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:28:10.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:28:10.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:28:10.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:28:10.661 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:28:10.662 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:28:10.662 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:28:10.662 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:28:10.662 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:28:10.662 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:28:11.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:28:11.018 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:28:11.018 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:28:11.018 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:28:11.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:28:11.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:28:11.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:28:11.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:28:11.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:28:11.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:28:11.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:28:11.567 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:28:11.567 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:28:12.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=9 2024-09-18T15:28:12.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 9 '!=' null 2024-09-18T15:28:12.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:28:12.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:28:12.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:28:12.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:28:12.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:28:12.239 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:28:12.240 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:28:12.240 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:28:12.240 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:28:12.240 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:28:12.240 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:28:12.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:28:12.611 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:28:12.611 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:28:12.611 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:28:13.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:28:13.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:28:13.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:28:13.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:28:13.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:28:13.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:28:13.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:28:13.196 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:28:13.196 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:28:13.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15 2024-09-18T15:28:13.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15 '!=' null 2024-09-18T15:28:13.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:28:13.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:28:13.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:28:13.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:28:13.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:28:13.883 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:28:13.883 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:28:13.883 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:28:13.883 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:28:13.883 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:28:13.883 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:28:14.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:28:14.246 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:28:14.247 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:28:14.247 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:28:14.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:28:14.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:28:14.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:28:14.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:28:14.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:28:14.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:28:14.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:28:14.905 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:28:14.905 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:28:15.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13 2024-09-18T15:28:15.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13 '!=' null 2024-09-18T15:28:15.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:28:15.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:28:15.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:28:15.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:28:15.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:28:15.516 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:28:15.516 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:28:15.516 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:28:15.516 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:28:15.516 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:28:15.516 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:28:15.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:28:15.968 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:28:15.968 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:28:15.968 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:28:16.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:28:16.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:28:16.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:28:16.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:28:16.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:28:16.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:28:16.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:28:16.564 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:28:16.564 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:28:17.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=9 2024-09-18T15:28:17.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 9 '!=' null 2024-09-18T15:28:17.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:28:17.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:28:17.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:28:17.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:28:17.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:28:17.185 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:28:17.185 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:28:17.185 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:28:17.185 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:28:17.185 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:28:17.186 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:28:17.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:28:17.540 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:28:17.540 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:28:17.540 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:28:18.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:28:18.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:28:18.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:28:18.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:28:18.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:28:18.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:28:18.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:28:18.038 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:28:18.039 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:28:18.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=9 2024-09-18T15:28:18.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 9 '!=' null 2024-09-18T15:28:18.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:28:18.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:28:18.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:28:18.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:28:18.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:28:18.672 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:28:18.672 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:28:18.672 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:28:18.672 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:28:18.672 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:28:18.672 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:28:19.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:28:19.030 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:28:19.030 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:28:19.031 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:28:19.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:28:19.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:28:19.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:28:19.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:160: do_recovery_out1: check td/osd-recovery-stats 1.0 1 replicated 200 0 0 0 2024-09-18T15:28:19.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-09-18T15:28:19.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-09-18T15:28:19.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=1 2024-09-18T15:28:19.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2024-09-18T15:28:19.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=200 2024-09-18T15:28:19.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-09-18T15:28:19.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=0 2024-09-18T15:28:19.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-09-18T15:28:19.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2024-09-18T15:28:19.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2024-09-18T15:28:19.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.1.log 2024-09-18T15:28:19.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-09-18T15:28:19.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2024-09-18T15:28:19.524 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-09-18T15:28:19.524 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.1.log 2024-09-18T15:28:19.524 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-09-18T15:28:19.524 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-09-18T15:28:19.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[1,2,4]/[1,2]' 2024-09-18T15:28:19.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-09-18T15:28:19.536 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-18T15:28:19.536 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-09-18T15:28:19.536 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [1,2,4]/[1,2] ' 2024-09-18T15:28:19.536 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-09-18T15:28:19.536 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-09-18T15:28:19.536 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-18T15:28:19.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=200 2024-09-18T15:28:19.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 200 200 2024-09-18T15:28:19.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-09-18T15:28:19.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-18T15:28:19.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-09-18T15:28:19.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-18T15:28:19.542 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-18T15:28:19.542 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-09-18T15:28:19.542 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-09-18T15:28:19.542 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-18T15:28:19.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-09-18T15:28:19.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-09-18T15:28:19.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-18T15:28:19.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-18T15:28:19.551 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-18T15:28:19.551 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-18T15:28:19.551 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-09-18T15:28:19.551 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-18T15:28:19.551 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-09-18T15:28:19.551 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [1,2,4]/[1,2] ' 2024-09-18T15:28:19.551 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-09-18T15:28:19.551 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-09-18T15:28:19.551 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-18T15:28:19.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=0 2024-09-18T15:28:19.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 0 0 2024-09-18T15:28:19.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=0 2024-09-18T15:28:19.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-18T15:28:19.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=0 2024-09-18T15:28:19.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-18T15:28:19.557 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-18T15:28:19.557 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-09-18T15:28:19.557 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-09-18T15:28:19.557 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-18T15:28:19.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-09-18T15:28:19.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-09-18T15:28:19.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-18T15:28:19.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-18T15:28:19.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-18T15:28:19.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-18T15:28:19.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2024-09-18T15:28:19.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:162: do_recovery_out1: delete_pool test 2024-09-18T15:28:19.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-18T15:28:19.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-18T15:28:20.206 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test' does not exist 2024-09-18T15:28:20.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:163: do_recovery_out1: kill_daemons td/osd-recovery-stats 2024-09-18T15:28:20.217 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:28:20.217 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:28:20.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:28:20.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:28:20.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:28:25.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:28:25.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-09-18T15:28:25.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-18T15:28:25.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:28:25.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-18T15:28:25.535 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:28:25.535 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:28:25.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:28:25.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:28:25.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:28:25.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:28:25.545 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:28:25.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:28:25.546 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:28:25.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:28:25.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:28:25.547 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:28:25.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:28:25.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:28:25.549 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:28:25.549 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:28:25.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:28:25.551 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:28:25.551 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-18T15:28:25.589 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:28:25.589 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:28:25.589 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:28:25.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.288083 2024-09-18T15:28:25.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:28:25.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:28:25.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-09-18T15:28:25.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-09-18T15:28:25.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-09-18T15:28:25.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-09-18T15:28:25.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-18T15:28:25.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:28:25.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-18T15:28:25.592 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:28:25.592 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:28:25.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:28:25.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:28:25.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:28:25.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:28:25.593 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:28:25.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:28:25.594 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:28:25.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:28:25.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:28:25.596 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:28:25.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:28:25.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:28:25.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:28:25.597 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:28:25.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:28:25.599 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:28:25.599 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-18T15:28:25.600 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:28:25.601 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:28:25.601 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:28:25.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.288083 2024-09-18T15:28:25.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:28:25.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:28:25.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-09-18T15:28:25.603 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T15:28:25.603 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:28:25.603 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:28:25.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.288083 2024-09-18T15:28:25.605 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T15:28:25.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T15:28:25.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T15:28:25.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-09-18T15:28:25.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_sizedown td/osd-recovery-stats 2024-09-18T15:28:25.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:241: TEST_recovery_sizedown: local dir=td/osd-recovery-stats 2024-09-18T15:28:25.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:243: TEST_recovery_sizedown: run_mon td/osd-recovery-stats a 2024-09-18T15:28:25.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-09-18T15:28:25.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T15:28:25.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T15:28:25.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T15:28:25.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-09-18T15:28:25.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-09-18T15:28:25.965 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T15:28:25.966 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:28:25.966 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:28:25.966 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:28:25.966 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:28:25.966 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:28:25.966 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:28:25.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T15:28:26.197 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T15:28:26.198 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T15:28:26.198 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:28:26.198 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:28:26.198 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T15:28:26.199 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T15:28:26.199 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:28:26.199 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:28:26.199 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:28:26.200 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:28:26.200 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:28:26.200 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:28:26.200 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.288083/ceph-mon.a.asok 2024-09-18T15:28:26.200 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:28:26.200 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.288083/ceph-mon.a.asok config get fsid 2024-09-18T15:28:26.286 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T15:28:26.286 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:28:26.286 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:28:26.286 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T15:28:26.286 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T15:28:26.286 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:28:26.287 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:28:26.287 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:28:26.287 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:28:26.287 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:28:26.288 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:28:26.288 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.288083/ceph-mon.a.asok 2024-09-18T15:28:26.288 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:28:26.288 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.288083/ceph-mon.a.asok config get mon_host 2024-09-18T15:28:26.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:244: TEST_recovery_sizedown: run_mgr td/osd-recovery-stats x 2024-09-18T15:28:26.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-09-18T15:28:26.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T15:28:26.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T15:28:26.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T15:28:26.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-09-18T15:28:26.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T15:28:27.209 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T15:28:27.209 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:28:27.209 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:28:27.210 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:28:27.210 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:28:27.210 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:28:27.210 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:28:27.210 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:28:27.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:28:27.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:245: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 0 2024-09-18T15:28:27.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:28:27.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:28:27.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T15:28:27.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:28:27.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-09-18T15:28:27.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:28:27.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:28:27.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:28:27.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:28:27.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-09-18T15:28:27.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-09-18T15:28:27.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:28:27.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:28:27.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:28:27.237 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:28:27.237 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:28:27.237 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:28:27.238 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:28:27.238 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:28:27.238 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:28:27.238 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:28:27.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:28:27.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:28:27.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:28:27.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:28:27.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:28:27.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:28:27.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:28:27.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:28:27.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:28:27.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:28:27.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:28:27.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:28:27.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-09-18T15:28:27.239 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:28:27.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=060009dd-3d65-4107-bda7-74c5ccf69a97 2024-09-18T15:28:27.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 060009dd-3d65-4107-bda7-74c5ccf69a97' 2024-09-18T15:28:27.240 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 060009dd-3d65-4107-bda7-74c5ccf69a97 2024-09-18T15:28:27.240 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:28:27.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCb8epmo3j4DhAAqXRuURnGbuXdU9d+Az6qwQ== 2024-09-18T15:28:27.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCb8epmo3j4DhAAqXRuURnGbuXdU9d+Az6qwQ=="}' 2024-09-18T15:28:27.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 060009dd-3d65-4107-bda7-74c5ccf69a97 -i td/osd-recovery-stats/0/new.json 2024-09-18T15:28:27.553 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:28:27.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-09-18T15:28:27.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCb8epmo3j4DhAAqXRuURnGbuXdU9d+Az6qwQ== --osd-uuid 060009dd-3d65-4107-bda7-74c5ccf69a97 2024-09-18T15:28:27.588 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:28:27.586+0000 7f6760275440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:28:27.597 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:28:27.596+0000 7f6760275440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:28:27.598 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:28:27.597+0000 7f6760275440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:28:27.599 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:28:27.598+0000 7f6760275440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-18T15:28:27.599 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:28:27.598+0000 7f6760275440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-18T15:28:27.599 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:28:27.598+0000 7f6760275440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-18T15:28:27.637 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:28:27.636+0000 7f6760275440 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-09-18T15:28:33.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-09-18T15:28:33.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:28:33.056 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T15:28:33.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T15:28:33.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:28:33.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T15:28:33.575 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T15:28:33.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:28:33.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:28:33.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:28:33.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:28:33.597 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:28:33.596+0000 7fd901321440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:28:33.619 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:28:33.618+0000 7fd901321440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:28:33.620 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:28:33.619+0000 7fd901321440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:28:33.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T15:28:33.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:28:33.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:28:33.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:28:33.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:28:33.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:33.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:28:33.977 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:28:33.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:33.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:28:34.143 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:28:34.142+0000 7fd901321440 -1 Falling back to public interface 2024-09-18T15:28:34.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:35.343 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:35.343 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:35.343 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:28:35.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:28:35.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:35.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:28:35.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:36.623 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:28:36.622+0000 7fd901321440 -1 osd.0 0 log_to_monitors true 2024-09-18T15:28:36.760 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:36.760 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:36.760 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:28:36.761 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:28:36.761 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:36.761 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:28:37.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:38.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:38.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:38.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:28:38.217 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:28:38.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:38.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:28:38.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:39.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:39.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:39.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:28:39.641 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:28:39.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:39.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:28:40.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:41.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:41.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:41.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:28:41.018 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:28:41.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:41.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:28:41.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:42.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:42.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:42.510 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:28:42.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:28:42.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:42.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:28:42.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:43.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:43.904 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:28:43.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:43.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:28:43.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:43.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:28:44.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:45.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:45.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:45.309 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:28:45.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:28:45.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:45.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:28:45.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:46.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:46.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:46.718 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:28:46.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:28:46.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:46.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:28:47.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:48.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:48.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:48.132 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:28:48.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:28:48.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:48.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:28:48.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:49.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:49.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:49.771 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:28:49.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:28:49.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:49.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:28:50.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:51.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:51.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:51.291 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:28:51.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:28:51.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:51.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:28:52.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:53.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:53.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:53.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:28:53.170 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:28:53.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:53.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:28:53.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:54.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:54.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:54.848 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:28:54.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:28:54.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:54.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:28:55.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:56.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:56.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:56.637 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:28:56.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:28:56.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:56.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:28:57.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:28:58.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:28:58.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:28:58.251 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:28:58.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:28:58.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:28:58.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:28:58.690 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/353656,v1:127.0.0.1:6803/353656] [v2:127.0.0.1:6804/353656,v1:127.0.0.1:6805/353656] exists,up 060009dd-3d65-4107-bda7-74c5ccf69a97 2024-09-18T15:28:58.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:28:58.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:28:58.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:28:58.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:246: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 1 2024-09-18T15:28:58.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:28:58.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:28:58.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T15:28:58.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:28:58.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-09-18T15:28:58.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:28:58.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:28:58.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:28:58.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:28:58.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-09-18T15:28:58.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-09-18T15:28:58.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:28:58.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:28:58.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:28:58.695 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:28:58.695 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:28:58.695 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:28:58.695 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:28:58.695 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:28:58.695 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:28:58.695 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:28:58.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:28:58.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:28:58.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:28:58.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:28:58.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:28:58.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:28:58.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:28:58.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:28:58.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:28:58.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:28:58.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:28:58.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:28:58.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-09-18T15:28:58.696 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:28:58.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7c3b3a3d-ec8e-46c8-b973-7a69f777284a 2024-09-18T15:28:58.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 7c3b3a3d-ec8e-46c8-b973-7a69f777284a' 2024-09-18T15:28:58.697 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 7c3b3a3d-ec8e-46c8-b973-7a69f777284a 2024-09-18T15:28:58.697 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:28:58.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC68epmZEs7KhAAfDoHVKaUXVVER0vO0WMA4A== 2024-09-18T15:28:58.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC68epmZEs7KhAAfDoHVKaUXVVER0vO0WMA4A=="}' 2024-09-18T15:28:58.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7c3b3a3d-ec8e-46c8-b973-7a69f777284a -i td/osd-recovery-stats/1/new.json 2024-09-18T15:28:59.264 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:28:59.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-09-18T15:28:59.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC68epmZEs7KhAAfDoHVKaUXVVER0vO0WMA4A== --osd-uuid 7c3b3a3d-ec8e-46c8-b973-7a69f777284a 2024-09-18T15:28:59.301 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:28:59.300+0000 7f64d024f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:28:59.302 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:28:59.302+0000 7f64d024f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:28:59.304 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:28:59.303+0000 7f64d024f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:28:59.304 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:28:59.303+0000 7f64d024f440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-18T15:28:59.304 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:28:59.303+0000 7f64d024f440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-18T15:28:59.304 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:28:59.303+0000 7f64d024f440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-18T15:28:59.380 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:28:59.379+0000 7f64d024f440 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-09-18T15:29:05.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-09-18T15:29:05.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:29:05.147 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T15:29:05.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T15:29:05.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:29:05.655 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T15:29:05.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T15:29:05.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:29:05.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:29:05.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:29:05.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:29:05.678 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:29:05.677+0000 7f0aac11d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:29:05.703 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:29:05.702+0000 7f0aac11d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:29:05.704 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:29:05.703+0000 7f0aac11d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:29:06.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T15:29:06.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:29:06.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:29:06.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:29:06.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:29:06.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:06.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:29:06.109 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:29:06.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:06.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:29:06.241 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:29:06.240+0000 7f0aac11d440 -1 Falling back to public interface 2024-09-18T15:29:06.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:07.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:07.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:07.518 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:29:07.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:29:07.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:07.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:29:07.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:08.725 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:29:08.723+0000 7f0aac11d440 -1 osd.1 0 log_to_monitors true 2024-09-18T15:29:08.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:08.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:08.899 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:29:08.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:29:08.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:08.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:29:09.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:10.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:10.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:10.378 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:29:10.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:29:10.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:10.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:29:10.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:11.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:11.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:11.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:29:11.808 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:29:11.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:11.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:29:12.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:13.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:13.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:13.204 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:29:13.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:29:13.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:13.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:29:13.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:14.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:14.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:14.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:29:14.738 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:29:14.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:14.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:29:15.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:16.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:16.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:16.134 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:29:16.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:29:16.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:16.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:29:16.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:17.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:17.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:17.595 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:29:17.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:29:17.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:17.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:29:18.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:19.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:19.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:19.004 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:29:19.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:29:19.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:19.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:29:19.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:20.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:20.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:20.504 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:29:20.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:29:20.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:20.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:29:21.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:22.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:22.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:22.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:29:22.194 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:29:22.194 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:22.194 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:29:22.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:23.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:23.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:23.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:29:23.810 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:29:23.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:23.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:29:24.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:25.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:25.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:25.503 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:29:25.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:29:25.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:25.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:29:26.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:27.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:27.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:27.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:29:27.500 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:29:27.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:27.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:29:28.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:29.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:29.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:29.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:29:29.161 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:29:29.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:29.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:29:29.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:30.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:30.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:30.822 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:29:30.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:29:30.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:30.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:29:31.218 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/355385,v1:127.0.0.1:6811/355385] [v2:127.0.0.1:6812/355385,v1:127.0.0.1:6813/355385] exists,up 7c3b3a3d-ec8e-46c8-b973-7a69f777284a 2024-09-18T15:29:31.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:29:31.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:29:31.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:29:31.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:247: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 2 2024-09-18T15:29:31.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:29:31.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:29:31.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T15:29:31.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:29:31.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-09-18T15:29:31.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:29:31.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:29:31.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:29:31.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:29:31.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-09-18T15:29:31.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-09-18T15:29:31.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:29:31.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:29:31.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:29:31.224 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:29:31.224 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:29:31.224 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:29:31.224 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:29:31.224 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:29:31.224 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:29:31.224 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:29:31.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:29:31.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:29:31.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:29:31.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:29:31.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:29:31.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:29:31.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:29:31.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:29:31.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:29:31.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:29:31.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:29:31.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:29:31.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-09-18T15:29:31.225 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:29:31.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bcc3b41e-cca5-40ea-92e0-7e8521fe20e5 2024-09-18T15:29:31.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 bcc3b41e-cca5-40ea-92e0-7e8521fe20e5' 2024-09-18T15:29:31.226 INFO:tasks.workunit.client.0.smithi047.stdout:add osd2 bcc3b41e-cca5-40ea-92e0-7e8521fe20e5 2024-09-18T15:29:31.226 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:29:31.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDb8epmMpMaDhAA/Kq6+m5mWhrdF0hE5cS3+A== 2024-09-18T15:29:31.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDb8epmMpMaDhAA/Kq6+m5mWhrdF0hE5cS3+A=="}' 2024-09-18T15:29:31.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bcc3b41e-cca5-40ea-92e0-7e8521fe20e5 -i td/osd-recovery-stats/2/new.json 2024-09-18T15:29:31.884 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:29:31.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-09-18T15:29:31.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDb8epmMpMaDhAA/Kq6+m5mWhrdF0hE5cS3+A== --osd-uuid bcc3b41e-cca5-40ea-92e0-7e8521fe20e5 2024-09-18T15:29:31.920 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:29:31.919+0000 7fb916620440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:29:31.950 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:29:31.949+0000 7fb916620440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:29:31.951 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:29:31.950+0000 7fb916620440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:29:31.952 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:29:31.951+0000 7fb916620440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-18T15:29:31.952 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:29:31.951+0000 7fb916620440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-18T15:29:31.952 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:29:31.951+0000 7fb916620440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-18T15:29:32.028 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:29:32.024+0000 7fb916620440 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-09-18T15:29:37.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-09-18T15:29:37.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:29:37.936 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd2 key to auth repository 2024-09-18T15:29:37.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T15:29:37.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:29:38.450 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T15:29:38.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T15:29:38.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:29:38.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:29:38.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:29:38.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:29:38.472 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:29:38.471+0000 7fdd1f17b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:29:38.500 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:29:38.498+0000 7fdd1f17b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:29:38.501 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:29:38.500+0000 7fdd1f17b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:29:38.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T15:29:38.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:29:38.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:29:38.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:29:38.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:29:38.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:38.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:29:38.906 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:29:38.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:38.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:29:39.031 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:29:39.030+0000 7fdd1f17b440 -1 Falling back to public interface 2024-09-18T15:29:39.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:40.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:40.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:40.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:29:40.278 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:29:40.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:40.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:29:40.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:41.647 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:29:41.645+0000 7fdd1f17b440 -1 osd.2 0 log_to_monitors true 2024-09-18T15:29:41.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:41.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:41.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:29:41.654 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:29:41.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:41.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:29:42.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:43.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:43.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:43.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:29:43.030 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:29:43.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:43.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:29:43.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:44.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:44.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:44.539 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:29:44.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:29:44.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:44.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:29:44.938 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:45.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:45.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:45.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:29:45.940 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:29:45.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:45.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:29:46.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:47.476 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:47.477 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:47.477 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:29:47.477 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:29:47.477 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:47.477 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:29:47.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:48.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:48.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:48.933 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:29:48.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:29:48.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:48.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:29:49.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:50.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:50.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:50.465 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:29:50.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:29:50.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:50.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:29:50.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:51.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:51.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:51.873 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:29:51.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:29:51.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:51.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:29:52.265 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:53.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:53.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:53.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:29:53.267 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:29:53.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:53.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:29:54.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:55.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:55.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:55.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:29:55.088 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:29:55.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:55.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:29:55.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:56.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:56.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:56.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:29:56.797 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:29:56.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:56.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:29:57.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:58.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:58.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:58.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:29:58.196 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:29:58.197 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:58.197 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:29:58.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:29:59.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:29:59.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:29:59.927 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:29:59.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:29:59.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:29:59.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:30:00.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:01.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:01.605 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:30:01.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:01.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:30:01.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:01.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:30:02.355 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:03.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:03.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:03.357 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:30:03.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:30:03.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:03.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:30:04.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:05.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:05.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:05.138 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T15:30:05.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:30:05.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:05.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:30:05.512 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/357114,v1:127.0.0.1:6819/357114] [v2:127.0.0.1:6820/357114,v1:127.0.0.1:6821/357114] exists,up bcc3b41e-cca5-40ea-92e0-7e8521fe20e5 2024-09-18T15:30:05.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:30:05.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:30:05.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:30:05.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:248: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 3 2024-09-18T15:30:05.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:30:05.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:30:05.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-18T15:30:05.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:30:05.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-09-18T15:30:05.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:30:05.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:30:05.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:30:05.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:30:05.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-09-18T15:30:05.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-09-18T15:30:05.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:30:05.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:30:05.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:30:05.517 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:30:05.517 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:30:05.517 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:30:05.517 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:30:05.517 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:30:05.518 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:30:05.518 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:30:05.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:30:05.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:30:05.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:30:05.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:30:05.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:30:05.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:30:05.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:30:05.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:30:05.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:30:05.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:30:05.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:30:05.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:30:05.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-09-18T15:30:05.519 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:30:05.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5100a69a-d3cf-4864-bc3d-4368976adbc9 2024-09-18T15:30:05.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 5100a69a-d3cf-4864-bc3d-4368976adbc9' 2024-09-18T15:30:05.519 INFO:tasks.workunit.client.0.smithi047.stdout:add osd3 5100a69a-d3cf-4864-bc3d-4368976adbc9 2024-09-18T15:30:05.519 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:30:05.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD98epmCwiUHxAA80T5KRIF9DX8E9V12nN4xw== 2024-09-18T15:30:05.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD98epmCwiUHxAA80T5KRIF9DX8E9V12nN4xw=="}' 2024-09-18T15:30:05.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5100a69a-d3cf-4864-bc3d-4368976adbc9 -i td/osd-recovery-stats/3/new.json 2024-09-18T15:30:06.020 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:30:06.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-09-18T15:30:06.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD98epmCwiUHxAA80T5KRIF9DX8E9V12nN4xw== --osd-uuid 5100a69a-d3cf-4864-bc3d-4368976adbc9 2024-09-18T15:30:06.057 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:30:06.055+0000 7fbaa1357440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:30:06.066 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:30:06.065+0000 7fbaa1357440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:30:06.067 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:30:06.066+0000 7fbaa1357440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:30:06.067 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:30:06.066+0000 7fbaa1357440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-18T15:30:06.067 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:30:06.066+0000 7fbaa1357440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-18T15:30:06.067 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:30:06.066+0000 7fbaa1357440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-18T15:30:06.106 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:30:06.105+0000 7fbaa1357440 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-09-18T15:30:11.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-09-18T15:30:11.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:30:11.772 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd3 key to auth repository 2024-09-18T15:30:11.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-18T15:30:11.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:30:12.283 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.3 2024-09-18T15:30:12.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-18T15:30:12.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:30:12.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:30:12.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:30:12.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:30:12.306 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:30:12.305+0000 7fa2a3015440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:30:12.332 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:30:12.330+0000 7fa2a3015440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:30:12.332 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:30:12.331+0000 7fa2a3015440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:30:12.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-18T15:30:12.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:30:12.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T15:30:12.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:30:12.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:30:12.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:12.690 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:30:12.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:30:12.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:12.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:30:12.874 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:30:12.873+0000 7fa2a3015440 -1 Falling back to public interface 2024-09-18T15:30:13.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:14.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:14.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:14.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:30:14.076 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:30:14.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:14.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:30:14.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:15.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:15.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:15.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:30:15.453 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:30:15.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:15.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:30:15.511 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:30:15.510+0000 7fa2a3015440 -1 osd.3 0 log_to_monitors true 2024-09-18T15:30:15.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:16.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:16.847 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:30:16.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:16.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:30:16.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:16.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:30:17.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:18.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:18.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:18.328 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:30:18.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:30:18.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:18.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:30:18.740 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:19.741 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:19.741 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:19.741 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:30:19.742 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:30:19.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:19.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:30:20.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:21.119 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:21.119 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:21.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:30:21.120 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:30:21.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:21.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:30:21.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:22.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:22.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:22.516 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:30:22.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:30:22.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:22.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:30:22.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:23.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:23.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:23.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:30:23.900 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:30:23.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:23.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:30:24.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:25.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:25.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:25.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:30:25.389 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:30:25.389 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:25.389 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:30:25.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:26.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:26.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:26.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:30:26.845 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:30:26.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:26.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:30:27.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:28.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:28.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:28.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:30:28.240 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:30:28.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:28.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:30:28.949 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:29.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:29.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:29.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:30:29.951 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:30:29.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:29.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:30:30.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:31.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:31.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:31.939 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:30:31.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:30:31.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:31.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:30:32.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:33.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:33.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:33.575 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:30:33.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:30:33.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:33.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:30:34.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:35.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:35.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:35.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:30:35.026 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:30:35.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:35.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:30:36.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:37.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:37.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:37.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:30:37.116 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:30:37.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:37.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:30:37.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:38.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:38.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:38.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:30:38.711 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T15:30:38.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:38.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:30:39.087 INFO:tasks.workunit.client.0.smithi047.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/358869,v1:127.0.0.1:6827/358869] [v2:127.0.0.1:6828/358869,v1:127.0.0.1:6829/358869] exists,up 5100a69a-d3cf-4864-bc3d-4368976adbc9 2024-09-18T15:30:39.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:30:39.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:30:39.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:30:39.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:249: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 4 2024-09-18T15:30:39.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:30:39.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:30:39.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-18T15:30:39.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:30:39.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2024-09-18T15:30:39.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:30:39.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:30:39.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:30:39.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:30:39.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2024-09-18T15:30:39.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2024-09-18T15:30:39.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:30:39.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:30:39.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:30:39.091 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:30:39.092 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:30:39.092 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:30:39.092 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:30:39.092 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:30:39.092 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:30:39.092 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:30:39.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:30:39.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:30:39.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:30:39.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:30:39.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:30:39.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:30:39.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:30:39.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:30:39.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:30:39.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:30:39.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:30:39.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:30:39.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2024-09-18T15:30:39.093 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:30:39.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cb2f6750-53cd-41e5-b4a0-b74cbeae5953 2024-09-18T15:30:39.093 INFO:tasks.workunit.client.0.smithi047.stdout:add osd4 cb2f6750-53cd-41e5-b4a0-b74cbeae5953 2024-09-18T15:30:39.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 cb2f6750-53cd-41e5-b4a0-b74cbeae5953' 2024-09-18T15:30:39.094 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:30:39.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAf8upm/4I4BhAATFeRGw/20lzrfO5ocWVg+A== 2024-09-18T15:30:39.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAf8upm/4I4BhAATFeRGw/20lzrfO5ocWVg+A=="}' 2024-09-18T15:30:39.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cb2f6750-53cd-41e5-b4a0-b74cbeae5953 -i td/osd-recovery-stats/4/new.json 2024-09-18T15:30:39.699 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:30:39.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2024-09-18T15:30:39.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAf8upm/4I4BhAATFeRGw/20lzrfO5ocWVg+A== --osd-uuid cb2f6750-53cd-41e5-b4a0-b74cbeae5953 2024-09-18T15:30:39.735 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:30:39.734+0000 7ff10d235440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:30:39.748 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:30:39.747+0000 7ff10d235440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:30:39.749 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:30:39.748+0000 7ff10d235440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:30:39.750 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:30:39.749+0000 7ff10d235440 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-18T15:30:39.750 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:30:39.749+0000 7ff10d235440 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-18T15:30:39.750 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:30:39.749+0000 7ff10d235440 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-18T15:30:39.841 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:30:39.839+0000 7ff10d235440 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2024-09-18T15:30:45.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2024-09-18T15:30:45.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:30:45.373 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd4 key to auth repository 2024-09-18T15:30:45.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-18T15:30:45.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:30:45.882 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.4 2024-09-18T15:30:45.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-18T15:30:45.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:30:45.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:30:45.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:30:45.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:30:45.902 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:30:45.901+0000 7f4e28c08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:30:45.904 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:30:45.903+0000 7f4e28c08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:30:45.905 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:30:45.904+0000 7f4e28c08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:30:46.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-18T15:30:46.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:30:46.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-18T15:30:46.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:30:46.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:30:46.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:46.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:30:46.289 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:30:46.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:46.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:30:46.422 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:30:46.421+0000 7f4e28c08440 -1 Falling back to public interface 2024-09-18T15:30:46.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:47.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:47.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:47.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:30:47.698 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:30:47.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:47.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:30:48.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:48.922 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:30:48.920+0000 7f4e28c08440 -1 osd.4 0 log_to_monitors true 2024-09-18T15:30:49.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:49.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:49.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:30:49.117 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:30:49.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:49.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:30:49.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:50.514 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:50.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:50.515 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:30:50.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:30:50.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:50.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:30:50.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:51.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:51.938 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:51.938 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:30:51.938 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:30:51.938 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:51.938 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:30:52.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:53.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:53.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:53.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:30:53.352 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:30:53.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:53.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:30:53.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:54.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:54.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:54.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:30:54.767 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:30:54.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:54.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:30:55.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:56.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:56.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:56.182 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:30:56.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:30:56.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:56.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:30:56.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:57.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:57.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:57.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:30:57.623 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:30:57.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:57.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:30:58.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:30:59.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:30:59.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:30:59.008 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:30:59.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:30:59.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:30:59.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:30:59.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:00.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:00.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:00.383 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:31:00.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:31:00.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:00.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:31:00.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:01.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:01.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:01.843 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:31:01.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:31:01.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:01.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:31:02.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:03.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:03.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:03.814 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:31:03.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:31:03.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:03.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:31:04.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:05.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:05.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:05.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:31:05.605 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:31:05.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:05.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:31:06.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:07.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:07.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:07.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:31:07.346 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:31:07.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:07.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:31:08.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:09.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:09.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:09.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:31:09.087 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:31:09.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:09.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:31:10.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:11.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:11.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:11.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:31:11.496 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:31:11.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:11.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:31:11.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:12.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:12.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:12.969 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T15:31:12.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:31:12.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:12.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:31:13.418 INFO:tasks.workunit.client.0.smithi047.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/360624,v1:127.0.0.1:6835/360624] [v2:127.0.0.1:6836/360624,v1:127.0.0.1:6837/360624] exists,up cb2f6750-53cd-41e5-b4a0-b74cbeae5953 2024-09-18T15:31:13.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:31:13.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:31:13.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:31:13.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:250: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 5 2024-09-18T15:31:13.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:31:13.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:31:13.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-18T15:31:13.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:31:13.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2024-09-18T15:31:13.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:31:13.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:31:13.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:31:13.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:31:13.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2024-09-18T15:31:13.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2024-09-18T15:31:13.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:31:13.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:31:13.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:31:13.423 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:31:13.423 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:31:13.423 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:31:13.423 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:31:13.423 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:31:13.423 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:31:13.423 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:31:13.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:31:13.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:31:13.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:31:13.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:31:13.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:31:13.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:31:13.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:31:13.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:31:13.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:31:13.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:31:13.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:31:13.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:31:13.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2024-09-18T15:31:13.424 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:31:13.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=be3a1d93-4d3a-4a1b-97b8-7cb262bb3328 2024-09-18T15:31:13.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 be3a1d93-4d3a-4a1b-97b8-7cb262bb3328' 2024-09-18T15:31:13.425 INFO:tasks.workunit.client.0.smithi047.stdout:add osd5 be3a1d93-4d3a-4a1b-97b8-7cb262bb3328 2024-09-18T15:31:13.425 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:31:13.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBB8upmrPT7GRAAtTa69pTW3XA4TTf+ZMe57w== 2024-09-18T15:31:13.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBB8upmrPT7GRAAtTa69pTW3XA4TTf+ZMe57w=="}' 2024-09-18T15:31:13.437 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new be3a1d93-4d3a-4a1b-97b8-7cb262bb3328 -i td/osd-recovery-stats/5/new.json 2024-09-18T15:31:13.920 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:31:13.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2024-09-18T15:31:13.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBB8upmrPT7GRAAtTa69pTW3XA4TTf+ZMe57w== --osd-uuid be3a1d93-4d3a-4a1b-97b8-7cb262bb3328 2024-09-18T15:31:13.955 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:31:13.954+0000 7fc2d071d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:31:13.957 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:31:13.956+0000 7fc2d071d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:31:13.958 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:31:13.957+0000 7fc2d071d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:31:13.958 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:31:13.957+0000 7fc2d071d440 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-18T15:31:13.958 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:31:13.957+0000 7fc2d071d440 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-18T15:31:13.958 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:31:13.957+0000 7fc2d071d440 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-18T15:31:14.068 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:31:14.067+0000 7fc2d071d440 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2024-09-18T15:31:19.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2024-09-18T15:31:19.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:31:19.970 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd5 key to auth repository 2024-09-18T15:31:19.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-18T15:31:19.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:31:20.481 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.5 2024-09-18T15:31:20.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-18T15:31:20.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:31:20.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:31:20.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:31:20.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:31:20.504 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:31:20.503+0000 7f69a814a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:31:20.506 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:31:20.505+0000 7f69a814a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:31:20.507 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:31:20.506+0000 7f69a814a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:31:20.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-18T15:31:20.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:31:20.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-18T15:31:20.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:31:20.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:31:20.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:20.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:31:20.848 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:31:20.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:20.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:31:21.023 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:31:21.022+0000 7f69a814a440 -1 Falling back to public interface 2024-09-18T15:31:21.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:22.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:22.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:22.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:31:22.253 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:31:22.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:22.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:31:22.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:23.634 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:31:23.633+0000 7f69a814a440 -1 osd.5 0 log_to_monitors true 2024-09-18T15:31:23.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:23.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:23.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:31:23.679 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:31:23.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:23.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:31:24.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:25.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:25.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:25.289 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:31:25.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:31:25.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:25.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:31:25.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:26.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:26.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:26.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:31:26.720 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:31:26.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:26.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:31:27.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:28.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:28.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:28.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:31:28.201 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:31:28.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:28.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:31:28.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:29.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:29.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:29.707 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:31:29.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:31:29.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:29.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:31:30.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:31.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:31.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:31.122 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:31:31.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:31:31.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:31.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:31:31.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:32.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:32.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:32.562 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:31:32.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:31:32.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:32.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:31:32.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:33.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:33.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:33.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:31:33.966 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:31:33.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:33.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:31:34.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:35.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:35.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:35.393 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:31:35.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:31:35.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:35.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:31:36.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:37.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:37.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:37.175 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:31:37.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:31:37.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:37.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:31:37.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:38.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:38.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:38.562 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:31:38.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:31:38.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:38.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:31:39.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:40.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:40.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:40.150 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:31:40.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:31:40.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:40.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:31:40.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:41.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:41.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:41.913 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:31:41.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:31:41.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:41.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:31:42.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:43.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:43.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:43.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:31:43.564 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:31:43.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:43.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:31:44.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:45.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:45.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:45.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:31:45.320 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:31:45.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:45.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:31:46.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:31:47.119 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:31:47.119 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:31:47.119 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:31:47.120 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T15:31:47.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:31:47.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:31:47.495 INFO:tasks.workunit.client.0.smithi047.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/362378,v1:127.0.0.1:6843/362378] [v2:127.0.0.1:6844/362378,v1:127.0.0.1:6845/362378] exists,up be3a1d93-4d3a-4a1b-97b8-7cb262bb3328 2024-09-18T15:31:47.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:31:47.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:31:47.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:31:47.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:252: TEST_recovery_sizedown: create_pool test 1 1 2024-09-18T15:31:47.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-18T15:31:48.025 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test' already exists 2024-09-18T15:31:48.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:31:49.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:253: TEST_recovery_sizedown: ceph osd pool set test size 4 2024-09-18T15:31:49.699 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 1 size to 4 2024-09-18T15:31:49.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:255: TEST_recovery_sizedown: wait_for_clean 2024-09-18T15:31:49.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:31:49.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:31:49.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:31:49.711 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:31:49.711 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:31:49.711 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:31:49.711 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:31:49.711 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:31:49.712 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:31:49.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:31:49.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:31:49.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:31:49.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:31:49.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:31:49.802 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:31:50.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:31:50.271 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:31:50.271 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T15:31:50.271 INFO:tasks.workunit.client.0.smithi047.stderr:3 2024-09-18T15:31:50.271 INFO:tasks.workunit.client.0.smithi047.stderr:4 2024-09-18T15:31:50.271 INFO:tasks.workunit.client.0.smithi047.stderr:5' 2024-09-18T15:31:50.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:31:50.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:31:50.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:31:50.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836517 2024-09-18T15:31:50.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836517 2024-09-18T15:31:50.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836517' 2024-09-18T15:31:50.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:31:50.434 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:31:50.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672992 2024-09-18T15:31:50.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672992 2024-09-18T15:31:50.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836517 1-42949672992' 2024-09-18T15:31:50.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:31:50.548 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:31:50.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509469 2024-09-18T15:31:50.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509469 2024-09-18T15:31:50.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836517 1-42949672992 2-64424509469' 2024-09-18T15:31:50.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:31:50.657 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:31:50.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345941 2024-09-18T15:31:50.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345941 2024-09-18T15:31:50.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836517 1-42949672992 2-64424509469 3-85899345941' 2024-09-18T15:31:50.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:31:50.765 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:31:50.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182414 2024-09-18T15:31:50.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182414 2024-09-18T15:31:50.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836517 1-42949672992 2-64424509469 3-85899345941 4-107374182414' 2024-09-18T15:31:50.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:31:50.874 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:31:50.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018887 2024-09-18T15:31:50.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018887 2024-09-18T15:31:50.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836517 1-42949672992 2-64424509469 3-85899345941 4-107374182414 5-128849018887' 2024-09-18T15:31:50.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:31:50.983 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836517 2024-09-18T15:31:50.983 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:31:50.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:31:50.984 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836517 2024-09-18T15:31:50.984 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:31:50.985 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836517 2024-09-18T15:31:50.985 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836517 2024-09-18T15:31:50.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836517' 2024-09-18T15:31:50.986 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:31:51.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836517 -lt 21474836517 2024-09-18T15:31:51.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:31:51.349 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672992 2024-09-18T15:31:51.350 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:31:51.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:31:51.351 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672992 2024-09-18T15:31:51.351 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:31:51.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672992 2024-09-18T15:31:51.352 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672992 2024-09-18T15:31:51.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672992' 2024-09-18T15:31:51.352 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:31:51.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672992 -lt 42949672992 2024-09-18T15:31:51.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:31:51.773 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509469 2024-09-18T15:31:51.773 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:31:51.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:31:51.774 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509469 2024-09-18T15:31:51.775 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:31:51.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509469 2024-09-18T15:31:51.776 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509469 2024-09-18T15:31:51.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509469' 2024-09-18T15:31:51.777 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:31:52.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509469 -lt 64424509469 2024-09-18T15:31:52.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:31:52.224 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345941 2024-09-18T15:31:52.224 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:31:52.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:31:52.225 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345941 2024-09-18T15:31:52.225 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:31:52.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345941 2024-09-18T15:31:52.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345941' 2024-09-18T15:31:52.227 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 85899345941 2024-09-18T15:31:52.227 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:31:52.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345941 -lt 85899345941 2024-09-18T15:31:52.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:31:52.595 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182414 2024-09-18T15:31:52.595 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:31:52.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:31:52.597 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182414 2024-09-18T15:31:52.597 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:31:52.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182414 2024-09-18T15:31:52.598 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.4 seq 107374182414 2024-09-18T15:31:52.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182414' 2024-09-18T15:31:52.598 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:31:52.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182414 -lt 107374182414 2024-09-18T15:31:52.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:31:52.971 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018887 2024-09-18T15:31:52.971 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:31:52.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:31:52.972 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018887 2024-09-18T15:31:52.973 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:31:52.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018887 2024-09-18T15:31:52.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018887' 2024-09-18T15:31:52.974 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.5 seq 128849018887 2024-09-18T15:31:52.974 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:31:53.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018888 -lt 128849018887 2024-09-18T15:31:53.340 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:31:53.341 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:31:53.341 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:31:53.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:31:53.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:31:53.835 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:31:53.835 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:31:53.835 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:31:53.835 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:31:53.835 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:31:53.835 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:31:54.197 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:31:54.197 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:31:54.197 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:31:54.198 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:31:54.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:31:54.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:31:54.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:31:54.687 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: seq 1 200 2024-09-18T15:31:54.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:54.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj1 /dev/null 2024-09-18T15:31:54.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:54.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj2 /dev/null 2024-09-18T15:31:55.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:55.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj3 /dev/null 2024-09-18T15:31:55.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:55.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj4 /dev/null 2024-09-18T15:31:55.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:55.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj5 /dev/null 2024-09-18T15:31:55.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:55.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj6 /dev/null 2024-09-18T15:31:55.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:55.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj7 /dev/null 2024-09-18T15:31:55.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:55.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj8 /dev/null 2024-09-18T15:31:55.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:55.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj9 /dev/null 2024-09-18T15:31:55.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:55.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj10 /dev/null 2024-09-18T15:31:56.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:56.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj11 /dev/null 2024-09-18T15:31:56.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:56.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj12 /dev/null 2024-09-18T15:31:56.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:56.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj13 /dev/null 2024-09-18T15:31:56.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:56.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj14 /dev/null 2024-09-18T15:31:56.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:56.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj15 /dev/null 2024-09-18T15:31:56.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:56.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj16 /dev/null 2024-09-18T15:31:56.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:56.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj17 /dev/null 2024-09-18T15:31:57.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:57.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj18 /dev/null 2024-09-18T15:31:57.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:57.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj19 /dev/null 2024-09-18T15:31:57.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:57.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj20 /dev/null 2024-09-18T15:31:57.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:57.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj21 /dev/null 2024-09-18T15:31:57.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:57.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj22 /dev/null 2024-09-18T15:31:57.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:57.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj23 /dev/null 2024-09-18T15:31:57.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:57.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj24 /dev/null 2024-09-18T15:31:58.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:58.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj25 /dev/null 2024-09-18T15:31:58.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:58.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj26 /dev/null 2024-09-18T15:31:58.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:58.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj27 /dev/null 2024-09-18T15:31:58.419 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:58.419 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj28 /dev/null 2024-09-18T15:31:58.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:58.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj29 /dev/null 2024-09-18T15:31:58.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:58.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj30 /dev/null 2024-09-18T15:31:58.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:58.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj31 /dev/null 2024-09-18T15:31:58.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:58.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj32 /dev/null 2024-09-18T15:31:59.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:59.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj33 /dev/null 2024-09-18T15:31:59.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:59.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj34 /dev/null 2024-09-18T15:31:59.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:59.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj35 /dev/null 2024-09-18T15:31:59.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:59.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj36 /dev/null 2024-09-18T15:31:59.676 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:59.676 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj37 /dev/null 2024-09-18T15:31:59.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:59.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj38 /dev/null 2024-09-18T15:31:59.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:31:59.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj39 /dev/null 2024-09-18T15:32:00.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:00.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj40 /dev/null 2024-09-18T15:32:00.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:00.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj41 /dev/null 2024-09-18T15:32:00.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:00.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj42 /dev/null 2024-09-18T15:32:00.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:00.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj43 /dev/null 2024-09-18T15:32:00.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:00.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj44 /dev/null 2024-09-18T15:32:00.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:00.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj45 /dev/null 2024-09-18T15:32:00.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:00.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj46 /dev/null 2024-09-18T15:32:01.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:01.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj47 /dev/null 2024-09-18T15:32:01.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:01.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj48 /dev/null 2024-09-18T15:32:01.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:01.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj49 /dev/null 2024-09-18T15:32:01.419 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:01.420 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj50 /dev/null 2024-09-18T15:32:01.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:01.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj51 /dev/null 2024-09-18T15:32:01.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:01.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj52 /dev/null 2024-09-18T15:32:01.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:01.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj53 /dev/null 2024-09-18T15:32:01.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:01.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj54 /dev/null 2024-09-18T15:32:02.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:02.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj55 /dev/null 2024-09-18T15:32:02.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:02.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj56 /dev/null 2024-09-18T15:32:02.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:02.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj57 /dev/null 2024-09-18T15:32:02.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:02.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj58 /dev/null 2024-09-18T15:32:02.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:02.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj59 /dev/null 2024-09-18T15:32:02.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:02.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj60 /dev/null 2024-09-18T15:32:02.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:02.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj61 /dev/null 2024-09-18T15:32:02.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:02.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj62 /dev/null 2024-09-18T15:32:03.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:03.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj63 /dev/null 2024-09-18T15:32:03.265 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:03.265 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj64 /dev/null 2024-09-18T15:32:03.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:03.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj65 /dev/null 2024-09-18T15:32:03.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:03.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj66 /dev/null 2024-09-18T15:32:03.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:03.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj67 /dev/null 2024-09-18T15:32:03.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:03.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj68 /dev/null 2024-09-18T15:32:03.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:03.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj69 /dev/null 2024-09-18T15:32:04.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:04.095 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj70 /dev/null 2024-09-18T15:32:04.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:04.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj71 /dev/null 2024-09-18T15:32:04.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:04.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj72 /dev/null 2024-09-18T15:32:04.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:04.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj73 /dev/null 2024-09-18T15:32:04.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:04.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj74 /dev/null 2024-09-18T15:32:04.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:04.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj75 /dev/null 2024-09-18T15:32:04.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:04.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj76 /dev/null 2024-09-18T15:32:04.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:04.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj77 /dev/null 2024-09-18T15:32:05.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:05.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj78 /dev/null 2024-09-18T15:32:05.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:05.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj79 /dev/null 2024-09-18T15:32:05.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:05.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj80 /dev/null 2024-09-18T15:32:05.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:05.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj81 /dev/null 2024-09-18T15:32:05.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:05.714 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj82 /dev/null 2024-09-18T15:32:05.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:05.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj83 /dev/null 2024-09-18T15:32:05.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:05.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj84 /dev/null 2024-09-18T15:32:06.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:06.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj85 /dev/null 2024-09-18T15:32:06.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:06.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj86 /dev/null 2024-09-18T15:32:06.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:06.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj87 /dev/null 2024-09-18T15:32:06.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:06.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj88 /dev/null 2024-09-18T15:32:06.660 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:06.660 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj89 /dev/null 2024-09-18T15:32:06.993 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:06.993 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj90 /dev/null 2024-09-18T15:32:07.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:07.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj91 /dev/null 2024-09-18T15:32:07.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:07.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj92 /dev/null 2024-09-18T15:32:07.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:07.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj93 /dev/null 2024-09-18T15:32:07.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:07.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj94 /dev/null 2024-09-18T15:32:07.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:07.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj95 /dev/null 2024-09-18T15:32:07.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:07.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj96 /dev/null 2024-09-18T15:32:08.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:08.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj97 /dev/null 2024-09-18T15:32:08.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:08.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj98 /dev/null 2024-09-18T15:32:08.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:08.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj99 /dev/null 2024-09-18T15:32:08.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:08.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj100 /dev/null 2024-09-18T15:32:08.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:08.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj101 /dev/null 2024-09-18T15:32:08.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:08.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj102 /dev/null 2024-09-18T15:32:08.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:08.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj103 /dev/null 2024-09-18T15:32:08.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:08.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj104 /dev/null 2024-09-18T15:32:09.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:09.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj105 /dev/null 2024-09-18T15:32:09.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:09.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj106 /dev/null 2024-09-18T15:32:09.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:09.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj107 /dev/null 2024-09-18T15:32:09.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:09.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj108 /dev/null 2024-09-18T15:32:09.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:09.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj109 /dev/null 2024-09-18T15:32:09.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:09.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj110 /dev/null 2024-09-18T15:32:09.862 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:09.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj111 /dev/null 2024-09-18T15:32:10.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:10.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj112 /dev/null 2024-09-18T15:32:10.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:10.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj113 /dev/null 2024-09-18T15:32:10.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:10.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj114 /dev/null 2024-09-18T15:32:10.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:10.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj115 /dev/null 2024-09-18T15:32:10.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:10.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj116 /dev/null 2024-09-18T15:32:10.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:10.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj117 /dev/null 2024-09-18T15:32:10.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:10.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj118 /dev/null 2024-09-18T15:32:10.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:10.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj119 /dev/null 2024-09-18T15:32:11.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:11.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj120 /dev/null 2024-09-18T15:32:11.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:11.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj121 /dev/null 2024-09-18T15:32:11.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:11.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj122 /dev/null 2024-09-18T15:32:11.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:11.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj123 /dev/null 2024-09-18T15:32:11.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:11.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj124 /dev/null 2024-09-18T15:32:11.826 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:11.826 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj125 /dev/null 2024-09-18T15:32:11.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:11.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj126 /dev/null 2024-09-18T15:32:12.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:12.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj127 /dev/null 2024-09-18T15:32:12.194 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:12.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj128 /dev/null 2024-09-18T15:32:12.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:12.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj129 /dev/null 2024-09-18T15:32:12.437 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:12.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj130 /dev/null 2024-09-18T15:32:12.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:12.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj131 /dev/null 2024-09-18T15:32:12.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:12.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj132 /dev/null 2024-09-18T15:32:12.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:12.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj133 /dev/null 2024-09-18T15:32:12.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:12.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj134 /dev/null 2024-09-18T15:32:13.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:13.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj135 /dev/null 2024-09-18T15:32:13.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:13.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj136 /dev/null 2024-09-18T15:32:13.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:13.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj137 /dev/null 2024-09-18T15:32:13.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:13.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj138 /dev/null 2024-09-18T15:32:13.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:13.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj139 /dev/null 2024-09-18T15:32:13.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:13.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj140 /dev/null 2024-09-18T15:32:13.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:13.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj141 /dev/null 2024-09-18T15:32:14.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:14.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj142 /dev/null 2024-09-18T15:32:14.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:14.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj143 /dev/null 2024-09-18T15:32:14.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:14.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj144 /dev/null 2024-09-18T15:32:14.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:14.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj145 /dev/null 2024-09-18T15:32:14.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:14.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj146 /dev/null 2024-09-18T15:32:14.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:14.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj147 /dev/null 2024-09-18T15:32:14.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:14.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj148 /dev/null 2024-09-18T15:32:15.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:15.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj149 /dev/null 2024-09-18T15:32:15.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:15.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj150 /dev/null 2024-09-18T15:32:15.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:15.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj151 /dev/null 2024-09-18T15:32:15.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:15.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj152 /dev/null 2024-09-18T15:32:15.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:15.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj153 /dev/null 2024-09-18T15:32:15.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:15.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj154 /dev/null 2024-09-18T15:32:15.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:15.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj155 /dev/null 2024-09-18T15:32:15.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:15.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj156 /dev/null 2024-09-18T15:32:16.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:16.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj157 /dev/null 2024-09-18T15:32:16.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:16.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj158 /dev/null 2024-09-18T15:32:16.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:16.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj159 /dev/null 2024-09-18T15:32:16.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:16.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj160 /dev/null 2024-09-18T15:32:16.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:16.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj161 /dev/null 2024-09-18T15:32:16.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:16.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj162 /dev/null 2024-09-18T15:32:16.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:16.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj163 /dev/null 2024-09-18T15:32:17.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:17.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj164 /dev/null 2024-09-18T15:32:17.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:17.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj165 /dev/null 2024-09-18T15:32:17.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:17.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj166 /dev/null 2024-09-18T15:32:17.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:17.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj167 /dev/null 2024-09-18T15:32:17.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:17.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj168 /dev/null 2024-09-18T15:32:17.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:17.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj169 /dev/null 2024-09-18T15:32:17.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:17.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj170 /dev/null 2024-09-18T15:32:18.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:18.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj171 /dev/null 2024-09-18T15:32:18.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:18.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj172 /dev/null 2024-09-18T15:32:18.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:18.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj173 /dev/null 2024-09-18T15:32:18.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:18.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj174 /dev/null 2024-09-18T15:32:18.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:18.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj175 /dev/null 2024-09-18T15:32:18.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:18.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj176 /dev/null 2024-09-18T15:32:18.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:18.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj177 /dev/null 2024-09-18T15:32:18.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:18.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj178 /dev/null 2024-09-18T15:32:19.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:19.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj179 /dev/null 2024-09-18T15:32:19.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:19.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj180 /dev/null 2024-09-18T15:32:19.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:19.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj181 /dev/null 2024-09-18T15:32:19.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:19.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj182 /dev/null 2024-09-18T15:32:19.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:19.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj183 /dev/null 2024-09-18T15:32:19.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:19.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj184 /dev/null 2024-09-18T15:32:19.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:19.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj185 /dev/null 2024-09-18T15:32:20.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:20.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj186 /dev/null 2024-09-18T15:32:20.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:20.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj187 /dev/null 2024-09-18T15:32:20.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:20.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj188 /dev/null 2024-09-18T15:32:20.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:20.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj189 /dev/null 2024-09-18T15:32:20.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:20.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj190 /dev/null 2024-09-18T15:32:20.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:20.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj191 /dev/null 2024-09-18T15:32:20.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:20.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj192 /dev/null 2024-09-18T15:32:20.988 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:20.988 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj193 /dev/null 2024-09-18T15:32:21.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:21.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj194 /dev/null 2024-09-18T15:32:21.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:21.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj195 /dev/null 2024-09-18T15:32:21.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:21.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj196 /dev/null 2024-09-18T15:32:21.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:21.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj197 /dev/null 2024-09-18T15:32:21.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:21.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj198 /dev/null 2024-09-18T15:32:21.861 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:21.861 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj199 /dev/null 2024-09-18T15:32:21.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-18T15:32:21.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj200 /dev/null 2024-09-18T15:32:22.113 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:262: TEST_recovery_sizedown: get_primary test obj1 2024-09-18T15:32:22.113 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-18T15:32:22.113 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T15:32:22.113 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-18T15:32:22.113 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:32:22.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:262: TEST_recovery_sizedown: local primary=1 2024-09-18T15:32:22.486 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:263: TEST_recovery_sizedown: get_pg test obj1 2024-09-18T15:32:22.487 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-18T15:32:22.487 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-09-18T15:32:22.487 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-09-18T15:32:22.487 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:32:22.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:263: TEST_recovery_sizedown: local PG=1.0 2024-09-18T15:32:22.855 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:265: TEST_recovery_sizedown: get_osds test obj1 2024-09-18T15:32:22.856 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test 2024-09-18T15:32:22.856 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj1 2024-09-18T15:32:22.856 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test obj1 2024-09-18T15:32:22.856 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:32:23.260 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-09-18T15:32:23.260 INFO:tasks.workunit.client.0.smithi047.stderr:0 2024-09-18T15:32:23.260 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T15:32:23.260 INFO:tasks.workunit.client.0.smithi047.stderr:4' 2024-09-18T15:32:23.260 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2 4 2024-09-18T15:32:23.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:265: TEST_recovery_sizedown: local 'allosds=1 0 2 4' 2024-09-18T15:32:23.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:267: TEST_recovery_sizedown: ceph osd set norecover 2024-09-18T15:32:23.917 INFO:tasks.workunit.client.0.smithi047.stderr:norecover is set 2024-09-18T15:32:23.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2024-09-18T15:32:23.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.1 2024-09-18T15:32:24.546 INFO:tasks.workunit.client.0.smithi047.stderr:osd.1 is already out. 2024-09-18T15:32:24.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2024-09-18T15:32:24.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.0 2024-09-18T15:32:25.308 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already out. 2024-09-18T15:32:25.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2024-09-18T15:32:25.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.2 2024-09-18T15:32:25.897 INFO:tasks.workunit.client.0.smithi047.stderr:osd.2 is already out. 2024-09-18T15:32:25.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2024-09-18T15:32:25.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.4 2024-09-18T15:32:27.195 INFO:tasks.workunit.client.0.smithi047.stderr:osd.4 is already out. 2024-09-18T15:32:27.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:273: TEST_recovery_sizedown: ceph osd pool set test size 2 2024-09-18T15:32:28.315 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 1 size to 2 2024-09-18T15:32:28.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:274: TEST_recovery_sizedown: ceph osd unset norecover 2024-09-18T15:32:30.050 INFO:tasks.workunit.client.0.smithi047.stderr:norecover is unset 2024-09-18T15:32:30.065 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:275: TEST_recovery_sizedown: get_primary test obj1 2024-09-18T15:32:30.066 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-18T15:32:30.066 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T15:32:30.066 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-18T15:32:30.066 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:32:30.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:275: TEST_recovery_sizedown: ceph tell osd.3 debug kick_recovery_wq 0 2024-09-18T15:32:30.657 INFO:tasks.workunit.client.0.smithi047.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-18T15:32:30.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:276: TEST_recovery_sizedown: sleep 2 2024-09-18T15:32:32.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:278: TEST_recovery_sizedown: wait_for_clean 2024-09-18T15:32:32.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:32:32.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:32:32.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:32:32.667 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:32:32.668 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:32:32.668 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:32:32.668 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:32:32.668 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:32:32.668 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:32:32.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:32:32.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:32:32.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:32:32.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:32:32.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:32:32.773 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:32:33.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:32:33.145 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:32:33.145 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T15:32:33.145 INFO:tasks.workunit.client.0.smithi047.stderr:3 2024-09-18T15:32:33.145 INFO:tasks.workunit.client.0.smithi047.stderr:4 2024-09-18T15:32:33.145 INFO:tasks.workunit.client.0.smithi047.stderr:5' 2024-09-18T15:32:33.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:32:33.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:32:33.145 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:32:33.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836527 2024-09-18T15:32:33.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836527 2024-09-18T15:32:33.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836527' 2024-09-18T15:32:33.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:32:33.296 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:32:33.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673003 2024-09-18T15:32:33.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673003 2024-09-18T15:32:33.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836527 1-42949673003' 2024-09-18T15:32:33.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:32:33.404 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:32:33.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509479 2024-09-18T15:32:33.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509479 2024-09-18T15:32:33.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836527 1-42949673003 2-64424509479' 2024-09-18T15:32:33.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:32:33.538 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:32:33.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345951 2024-09-18T15:32:33.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345951 2024-09-18T15:32:33.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836527 1-42949673003 2-64424509479 3-85899345951' 2024-09-18T15:32:33.648 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:32:33.648 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:32:33.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182425 2024-09-18T15:32:33.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182425 2024-09-18T15:32:33.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836527 1-42949673003 2-64424509479 3-85899345951 4-107374182425' 2024-09-18T15:32:33.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:32:33.756 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:32:33.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018898 2024-09-18T15:32:33.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018898 2024-09-18T15:32:33.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836527 1-42949673003 2-64424509479 3-85899345951 4-107374182425 5-128849018898' 2024-09-18T15:32:33.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:32:33.864 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836527 2024-09-18T15:32:33.864 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:32:33.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:32:33.865 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836527 2024-09-18T15:32:33.865 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:32:33.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836527 2024-09-18T15:32:33.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836527' 2024-09-18T15:32:33.867 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836527 2024-09-18T15:32:33.867 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:32:34.231 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836525 -lt 21474836527 2024-09-18T15:32:34.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:32:35.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:32:35.233 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:32:35.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836528 -lt 21474836527 2024-09-18T15:32:35.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:32:35.652 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673003 2024-09-18T15:32:35.652 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:32:35.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:32:35.653 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673003 2024-09-18T15:32:35.653 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:32:35.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673003 2024-09-18T15:32:35.654 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949673003 2024-09-18T15:32:35.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673003' 2024-09-18T15:32:35.655 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:32:36.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673003 -lt 42949673003 2024-09-18T15:32:36.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:32:36.148 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509479 2024-09-18T15:32:36.149 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:32:36.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:32:36.150 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509479 2024-09-18T15:32:36.150 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:32:36.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509479 2024-09-18T15:32:36.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509479' 2024-09-18T15:32:36.152 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509479 2024-09-18T15:32:36.152 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:32:36.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509479 -lt 64424509479 2024-09-18T15:32:36.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:32:36.542 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345951 2024-09-18T15:32:36.542 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:32:36.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:32:36.544 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345951 2024-09-18T15:32:36.544 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:32:36.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345951 2024-09-18T15:32:36.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345951' 2024-09-18T15:32:36.545 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 85899345951 2024-09-18T15:32:36.545 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:32:37.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345952 -lt 85899345951 2024-09-18T15:32:37.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:32:37.055 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182425 2024-09-18T15:32:37.055 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:32:37.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:32:37.056 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182425 2024-09-18T15:32:37.056 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:32:37.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182425 2024-09-18T15:32:37.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182425' 2024-09-18T15:32:37.057 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.4 seq 107374182425 2024-09-18T15:32:37.057 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:32:37.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182425 -lt 107374182425 2024-09-18T15:32:37.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:32:37.491 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018898 2024-09-18T15:32:37.491 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:32:37.491 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:32:37.492 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018898 2024-09-18T15:32:37.492 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:32:37.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018898 2024-09-18T15:32:37.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018898' 2024-09-18T15:32:37.494 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.5 seq 128849018898 2024-09-18T15:32:37.494 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:32:37.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018898 -lt 128849018898 2024-09-18T15:32:37.866 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:32:37.867 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:32:37.867 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:32:38.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:32:38.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:32:38.798 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:32:38.798 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:32:38.799 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:32:38.799 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:32:38.799 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:32:38.799 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:32:39.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:32:39.158 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:32:39.158 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:32:39.159 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:32:39.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:32:39.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-18T15:32:39.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T15:32:39.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-18T15:32:39.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:32:39.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:32:39.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:32:39.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:32:39.772 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:32:39.772 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:32:39.772 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:32:39.772 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:32:39.772 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:32:39.772 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:32:40.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:32:40.163 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:32:40.164 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:32:40.164 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:32:40.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:32:40.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:32:40.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:32:40.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:32:40.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:32:40.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:32:40.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:32:40.712 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:32:40.712 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:32:41.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-18T15:32:41.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-18T15:32:41.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:32:41.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:32:41.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:32:41.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:32:41.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:32:41.316 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:32:41.316 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:32:41.316 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:32:41.316 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:32:41.316 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:32:41.316 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:32:41.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:32:41.715 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:32:41.715 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:32:41.716 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:32:42.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:32:42.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:32:42.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:32:42.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:32:42.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:32:42.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:32:42.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:32:42.292 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:32:42.292 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:32:42.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-18T15:32:42.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-18T15:32:42.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:32:42.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:32:42.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:32:42.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:32:42.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:32:42.969 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:32:42.969 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:32:42.969 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:32:42.969 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:32:42.969 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:32:42.970 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:32:43.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:32:43.385 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:32:43.385 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:32:43.385 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:32:43.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:32:43.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:32:43.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:32:43.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:32:43.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:32:43.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:32:43.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:32:43.975 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:32:43.975 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:32:44.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-09-18T15:32:44.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-09-18T15:32:44.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:32:44.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:32:44.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:32:44.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:32:44.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:32:44.596 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:32:44.596 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:32:44.596 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:32:44.596 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:32:44.596 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:32:44.596 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:32:44.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:32:44.968 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:32:44.968 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:32:44.968 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:32:45.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:32:45.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:32:45.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:32:45.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:32:45.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:32:45.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:32:45.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:32:45.536 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:32:45.536 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:32:46.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=9 2024-09-18T15:32:46.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 9 '!=' null 2024-09-18T15:32:46.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:32:46.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:32:46.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:32:46.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:32:46.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:32:46.164 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:32:46.164 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:32:46.164 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:32:46.165 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:32:46.165 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:32:46.165 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:32:46.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:32:46.552 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:32:46.553 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:32:46.553 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:32:47.172 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:32:47.172 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:32:47.172 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:32:47.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:32:47.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:32:47.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:32:47.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:32:47.173 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:32:47.173 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:32:47.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-18T15:32:47.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-18T15:32:47.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:32:47.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:32:47.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:32:47.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:32:47.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:32:47.935 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:32:47.935 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:32:47.935 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:32:47.935 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:32:47.935 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:32:47.935 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:32:48.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:32:48.313 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:32:48.314 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:32:48.314 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:32:48.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:32:48.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:32:48.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:32:48.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:32:48.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:32:48.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:32:48.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:32:48.813 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:32:48.813 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:32:49.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-18T15:32:49.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-18T15:32:49.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:32:49.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:32:49.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:32:49.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:32:49.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:32:49.562 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:32:49.562 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:32:49.562 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:32:49.563 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:32:49.563 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:32:49.563 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:32:49.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:32:49.944 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:32:49.944 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:32:49.944 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:32:50.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:32:50.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:32:50.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:32:50.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:32:50.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:32:50.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:32:50.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:32:50.519 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:32:50.519 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:32:51.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-18T15:32:51.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-18T15:32:51.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:32:51.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:32:51.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:32:51.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:32:51.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:32:51.315 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:32:51.316 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:32:51.316 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:32:51.316 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:32:51.316 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:32:51.316 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:32:51.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:32:51.723 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:32:51.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:32:51.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:32:52.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:32:52.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:32:52.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:32:52.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:32:52.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:32:52.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:32:52.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:32:52.302 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:32:52.302 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:32:52.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=10 2024-09-18T15:32:52.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 10 '!=' null 2024-09-18T15:32:52.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:32:52.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:32:52.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:32:52.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:32:52.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:32:52.995 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:32:52.995 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:32:52.995 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:32:52.995 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:32:52.995 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:32:52.996 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:32:53.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:32:53.391 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:32:53.392 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:32:53.392 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:32:54.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:32:54.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:32:54.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:32:54.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:32:54.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:32:54.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:32:54.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:32:54.143 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:32:54.143 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:32:54.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=7 2024-09-18T15:32:54.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 7 '!=' null 2024-09-18T15:32:54.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:32:54.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:32:54.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:32:54.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:32:54.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:32:54.739 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:32:54.739 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:32:54.739 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:32:54.740 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:32:54.740 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:32:54.740 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:32:55.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:32:55.106 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:32:55.106 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:32:55.106 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:32:55.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:32:55.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:32:55.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:32:55.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:32:55.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:32:55.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:32:55.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:32:55.797 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:32:55.797 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:32:56.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=10 2024-09-18T15:32:56.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 10 '!=' null 2024-09-18T15:32:56.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:32:56.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:32:56.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:32:56.432 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:32:56.432 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:32:56.432 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:32:56.432 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:32:56.432 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:32:56.432 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:32:56.432 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:32:56.432 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:32:56.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:32:56.791 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:32:56.792 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:32:56.792 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:32:57.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:32:57.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:32:57.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:32:57.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:32:57.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:32:57.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:32:57.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:32:57.316 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:32:57.316 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:32:57.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=10 2024-09-18T15:32:57.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 10 '!=' null 2024-09-18T15:32:57.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:32:57.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:32:57.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:32:57.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:32:57.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:32:57.911 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:32:57.911 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:32:57.911 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:32:57.911 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:32:57.911 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:32:57.911 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:32:58.264 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:32:58.264 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:32:58.265 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:32:58.265 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:32:58.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:32:58.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:32:58.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:32:58.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:32:58.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:32:58.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:32:58.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:32:58.767 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:32:58.767 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:32:59.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-18T15:32:59.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-18T15:32:59.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:32:59.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:32:59.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:32:59.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:32:59.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:32:59.395 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:32:59.395 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:32:59.396 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:32:59.396 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:32:59.396 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:32:59.396 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:32:59.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:32:59.752 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:32:59.753 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:32:59.753 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:33:00.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:33:00.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:33:00.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:33:00.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:33:00.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:33:00.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:33:00.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:33:00.246 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:33:00.246 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:33:00.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-18T15:33:00.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-18T15:33:00.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:33:00.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:33:00.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:33:00.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:33:00.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:33:00.889 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:33:00.890 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:33:00.890 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:33:00.890 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:33:00.890 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:33:00.890 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:33:01.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:33:01.251 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:33:01.251 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:33:01.251 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:33:01.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:33:01.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:33:01.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:33:01.766 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:281: TEST_recovery_sizedown: get_primary test obj1 2024-09-18T15:33:01.766 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-18T15:33:01.766 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T15:33:01.766 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-18T15:33:01.766 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:33:02.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:281: TEST_recovery_sizedown: primary=3 2024-09-18T15:33:02.134 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:283: TEST_recovery_sizedown: expr 200 '*' 2 2024-09-18T15:33:02.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:283: TEST_recovery_sizedown: local misplaced=400 2024-09-18T15:33:02.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:284: TEST_recovery_sizedown: local log=td/osd-recovery-stats/osd.3.log 2024-09-18T15:33:02.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:285: TEST_recovery_sizedown: check td/osd-recovery-stats 1.0 3 replicated 0 0 400 0 2024-09-18T15:33:02.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-09-18T15:33:02.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-09-18T15:33:02.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=3 2024-09-18T15:33:02.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2024-09-18T15:33:02.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=0 2024-09-18T15:33:02.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-09-18T15:33:02.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=400 2024-09-18T15:33:02.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-09-18T15:33:02.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2024-09-18T15:33:02.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2024-09-18T15:33:02.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.3.log 2024-09-18T15:33:02.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-09-18T15:33:02.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2024-09-18T15:33:02.138 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-09-18T15:33:02.138 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.3.log 2024-09-18T15:33:02.138 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-09-18T15:33:02.138 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-09-18T15:33:02.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[3,5]/[3]' 2024-09-18T15:33:02.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-09-18T15:33:02.154 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-18T15:33:02.154 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.3.log 2024-09-18T15:33:02.154 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [3,5]/[3] ' 2024-09-18T15:33:02.154 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-09-18T15:33:02.154 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-09-18T15:33:02.155 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-18T15:33:02.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=0 2024-09-18T15:33:02.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 0 0 2024-09-18T15:33:02.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=0 2024-09-18T15:33:02.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-18T15:33:02.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=0 2024-09-18T15:33:02.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-18T15:33:02.159 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-18T15:33:02.159 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.3.log 2024-09-18T15:33:02.159 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-09-18T15:33:02.159 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-18T15:33:02.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-09-18T15:33:02.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-09-18T15:33:02.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-18T15:33:02.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-18T15:33:02.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-18T15:33:02.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-18T15:33:02.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-09-18T15:33:02.167 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-18T15:33:02.167 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.3.log 2024-09-18T15:33:02.167 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [3,5]/[3] ' 2024-09-18T15:33:02.167 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-09-18T15:33:02.167 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-09-18T15:33:02.167 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-18T15:33:02.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=400 2024-09-18T15:33:02.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 400 400 2024-09-18T15:33:02.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=400 2024-09-18T15:33:02.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-18T15:33:02.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=400 2024-09-18T15:33:02.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-18T15:33:02.170 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-18T15:33:02.170 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.3.log 2024-09-18T15:33:02.171 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-09-18T15:33:02.171 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-18T15:33:02.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-09-18T15:33:02.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-09-18T15:33:02.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-18T15:33:02.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-18T15:33:02.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-18T15:33:02.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-18T15:33:02.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2024-09-18T15:33:02.178 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.3.log 2024-09-18T15:33:02.178 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: tail -1 2024-09-18T15:33:02.178 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-09-18T15:33:02.194 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: UPACT='[3,5]/[3]' 2024-09-18T15:33:02.195 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 3 ' td/osd-recovery-stats/osd.3.log 2024-09-18T15:33:02.195 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: grep -F ' [3,5]/[3] ' 2024-09-18T15:33:02.195 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: head -1 2024-09-18T15:33:02.195 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: sed 's/.* \([0-9]*\)$/\1/' 2024-09-18T15:33:02.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: FIRST=200 2024-09-18T15:33:02.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:291: TEST_recovery_sizedown: below_margin 200 200 2024-09-18T15:33:02.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-09-18T15:33:02.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-18T15:33:02.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-09-18T15:33:02.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-18T15:33:02.199 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 3 ' td/osd-recovery-stats/osd.3.log 2024-09-18T15:33:02.199 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: tail -1 2024-09-18T15:33:02.199 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: sed 's/.* \([0-9]*\)$/\1/' 2024-09-18T15:33:02.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: LAST=0 2024-09-18T15:33:02.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:293: TEST_recovery_sizedown: above_margin 0 0 2024-09-18T15:33:02.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-18T15:33:02.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-18T15:33:02.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-18T15:33:02.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-18T15:33:02.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:295: TEST_recovery_sizedown: delete_pool test 2024-09-18T15:33:02.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-18T15:33:02.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-18T15:33:02.785 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test' does not exist 2024-09-18T15:33:02.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:296: TEST_recovery_sizedown: kill_daemons td/osd-recovery-stats 2024-09-18T15:33:02.797 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:33:02.797 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:33:02.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:33:02.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:33:02.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:33:08.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:33:08.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-09-18T15:33:08.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-18T15:33:08.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:33:08.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-18T15:33:08.116 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:33:08.116 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:33:08.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:33:08.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:33:08.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:33:08.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:33:08.125 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:33:08.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:33:08.126 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:33:08.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:33:08.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:33:08.128 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:33:08.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:33:08.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:33:08.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:33:08.129 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:33:08.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:33:08.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:33:08.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-18T15:33:08.172 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:33:08.172 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:33:08.172 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:33:08.172 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.288083 2024-09-18T15:33:08.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:33:08.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:33:08.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-09-18T15:33:08.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-09-18T15:33:08.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-09-18T15:33:08.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-09-18T15:33:08.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-18T15:33:08.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:33:08.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-18T15:33:08.175 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:33:08.175 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:33:08.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:33:08.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:33:08.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:33:08.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:33:08.176 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:33:08.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:33:08.178 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:33:08.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:33:08.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:33:08.179 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:33:08.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:33:08.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:33:08.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:33:08.181 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:33:08.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:33:08.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:33:08.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-18T15:33:08.184 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:33:08.184 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:33:08.184 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:33:08.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.288083 2024-09-18T15:33:08.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:33:08.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:33:08.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-09-18T15:33:08.186 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T15:33:08.187 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:33:08.187 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:33:08.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.288083 2024-09-18T15:33:08.189 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T15:33:08.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T15:33:08.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T15:33:08.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-09-18T15:33:08.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_sizeup td/osd-recovery-stats 2024-09-18T15:33:08.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:186: TEST_recovery_sizeup: local dir=td/osd-recovery-stats 2024-09-18T15:33:08.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:188: TEST_recovery_sizeup: run_mon td/osd-recovery-stats a 2024-09-18T15:33:08.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-09-18T15:33:08.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T15:33:08.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T15:33:08.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T15:33:08.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-09-18T15:33:08.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-09-18T15:33:08.548 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T15:33:08.549 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:33:08.549 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:33:08.549 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:33:08.549 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:33:08.549 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:33:08.549 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:33:08.549 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T15:33:08.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T15:33:08.783 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T15:33:08.783 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:33:08.784 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:33:08.784 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T15:33:08.785 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T15:33:08.785 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:33:08.785 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:33:08.785 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:33:08.785 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:33:08.786 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:33:08.786 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:33:08.786 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.288083/ceph-mon.a.asok 2024-09-18T15:33:08.786 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:33:08.786 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.288083/ceph-mon.a.asok config get fsid 2024-09-18T15:33:08.867 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T15:33:08.867 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:33:08.867 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:33:08.867 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T15:33:08.867 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T15:33:08.867 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:33:08.867 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:33:08.867 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:33:08.867 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:33:08.868 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:33:08.868 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:33:08.868 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.288083/ceph-mon.a.asok 2024-09-18T15:33:08.868 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:33:08.868 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.288083/ceph-mon.a.asok config get mon_host 2024-09-18T15:33:08.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:189: TEST_recovery_sizeup: run_mgr td/osd-recovery-stats x 2024-09-18T15:33:08.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-09-18T15:33:08.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T15:33:08.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T15:33:08.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T15:33:08.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-09-18T15:33:08.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T15:33:09.788 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T15:33:09.788 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:33:09.788 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:33:09.788 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:33:09.788 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:33:09.788 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:33:09.788 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:33:09.789 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:33:09.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:33:09.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:190: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 0 2024-09-18T15:33:09.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:33:09.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:33:09.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T15:33:09.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:33:09.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-09-18T15:33:09.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:33:09.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:33:09.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:33:09.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:33:09.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-09-18T15:33:09.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-09-18T15:33:09.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:33:09.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:33:09.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:33:09.816 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:33:09.816 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:33:09.816 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:33:09.817 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:33:09.817 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:33:09.817 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:33:09.817 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:33:09.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:33:09.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:33:09.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:33:09.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:33:09.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:33:09.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:33:09.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:33:09.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:33:09.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:33:09.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:33:09.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:33:09.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:33:09.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-09-18T15:33:09.818 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:33:09.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=58d606e3-aa92-45ee-a59d-1a421e25e5e7 2024-09-18T15:33:09.818 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 58d606e3-aa92-45ee-a59d-1a421e25e5e7 2024-09-18T15:33:09.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 58d606e3-aa92-45ee-a59d-1a421e25e5e7' 2024-09-18T15:33:09.819 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:33:09.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC18upmBUqKMRAAMKXBcSUD1DVuGQ1j3OQS9w== 2024-09-18T15:33:09.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC18upmBUqKMRAAMKXBcSUD1DVuGQ1j3OQS9w=="}' 2024-09-18T15:33:09.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 58d606e3-aa92-45ee-a59d-1a421e25e5e7 -i td/osd-recovery-stats/0/new.json 2024-09-18T15:33:10.155 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:33:10.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-09-18T15:33:10.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC18upmBUqKMRAAMKXBcSUD1DVuGQ1j3OQS9w== --osd-uuid 58d606e3-aa92-45ee-a59d-1a421e25e5e7 2024-09-18T15:33:10.190 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:33:10.189+0000 7f805dc69440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:33:10.199 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:33:10.199+0000 7f805dc69440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:33:10.201 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:33:10.200+0000 7f805dc69440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:33:10.201 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:33:10.200+0000 7f805dc69440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-18T15:33:10.201 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:33:10.200+0000 7f805dc69440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-18T15:33:10.201 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:33:10.200+0000 7f805dc69440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-18T15:33:10.240 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:33:10.239+0000 7f805dc69440 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-09-18T15:33:15.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-09-18T15:33:15.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:33:15.837 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T15:33:15.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T15:33:15.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:33:16.349 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T15:33:16.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T15:33:16.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:33:16.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:33:16.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:33:16.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:33:16.373 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:33:16.372+0000 7ff61881a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:33:16.396 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:33:16.395+0000 7ff61881a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:33:16.397 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:33:16.397+0000 7ff61881a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:33:16.714 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T15:33:16.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:33:16.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:33:16.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:33:16.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:33:16.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:16.715 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:33:16.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:33:16.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:16.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:33:16.928 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:33:16.927+0000 7ff61881a440 -1 Falling back to public interface 2024-09-18T15:33:17.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:33:18.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:33:18.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:18.120 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:33:18.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:33:18.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:18.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:33:18.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:33:19.462 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:33:19.461+0000 7ff61881a440 -1 osd.0 0 log_to_monitors true 2024-09-18T15:33:19.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:33:19.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:19.546 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:33:19.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:33:19.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:19.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:33:20.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:33:21.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:33:21.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:21.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:33:21.076 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:33:21.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:21.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:33:21.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:33:22.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:33:22.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:22.524 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:33:22.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:33:22.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:22.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:33:22.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:33:23.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:33:23.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:23.903 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:33:23.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:33:23.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:23.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:33:24.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:33:25.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:33:25.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:25.334 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:33:25.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:33:25.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:25.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:33:25.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:33:26.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:33:26.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:26.731 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:33:26.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:33:26.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:26.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:33:27.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:33:28.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:33:28.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:28.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:33:28.221 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:33:28.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:28.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:33:28.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:33:29.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:33:29.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:29.594 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:33:29.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:33:29.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:29.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:33:29.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:33:30.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:33:30.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:30.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:33:30.975 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:33:30.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:30.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:33:31.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:33:32.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:33:32.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:32.728 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:33:32.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:33:32.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:32.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:33:33.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:33:34.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:33:34.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:34.191 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:33:34.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:33:34.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:34.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:33:34.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:33:35.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:33:35.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:35.801 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:33:35.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:33:35.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:35.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:33:36.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:33:37.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:33:37.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:37.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:33:37.638 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:33:37.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:37.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:33:38.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:33:39.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:33:39.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:39.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:33:39.598 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:33:39.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:39.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:33:40.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:33:41.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:33:41.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:41.085 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:33:41.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:33:41.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:41.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:33:41.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:33:42.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:33:42.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:42.583 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T15:33:42.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:33:42.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:42.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:33:42.965 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/370898,v1:127.0.0.1:6803/370898] [v2:127.0.0.1:6804/370898,v1:127.0.0.1:6805/370898] exists,up 58d606e3-aa92-45ee-a59d-1a421e25e5e7 2024-09-18T15:33:42.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:33:42.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:33:42.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:33:42.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:191: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 1 2024-09-18T15:33:42.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:33:42.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:33:42.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T15:33:42.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:33:42.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-09-18T15:33:42.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:33:42.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:33:42.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:33:42.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:33:42.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-09-18T15:33:42.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-09-18T15:33:42.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:33:42.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:33:42.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:33:42.970 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:33:42.970 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:33:42.970 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:33:42.971 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:33:42.971 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:33:42.971 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:33:42.971 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:33:42.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:33:42.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:33:42.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:33:42.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:33:42.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:33:42.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:33:42.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:33:42.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:33:42.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:33:42.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:33:42.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:33:42.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:33:42.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-09-18T15:33:42.972 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:33:42.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=24014b91-356d-42ac-913a-34203f790be8 2024-09-18T15:33:42.972 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 24014b91-356d-42ac-913a-34203f790be8 2024-09-18T15:33:42.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 24014b91-356d-42ac-913a-34203f790be8' 2024-09-18T15:33:42.973 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:33:42.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDW8upmFcCZOhAAwALyOeSVtRtoJC3+3zJQvw== 2024-09-18T15:33:42.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDW8upmFcCZOhAAwALyOeSVtRtoJC3+3zJQvw=="}' 2024-09-18T15:33:42.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 24014b91-356d-42ac-913a-34203f790be8 -i td/osd-recovery-stats/1/new.json 2024-09-18T15:33:43.506 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:33:43.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-09-18T15:33:43.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDW8upmFcCZOhAAwALyOeSVtRtoJC3+3zJQvw== --osd-uuid 24014b91-356d-42ac-913a-34203f790be8 2024-09-18T15:33:43.543 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:33:43.542+0000 7f9de8c0a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:33:43.544 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:33:43.543+0000 7f9de8c0a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:33:43.545 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:33:43.544+0000 7f9de8c0a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:33:43.546 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:33:43.545+0000 7f9de8c0a440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-18T15:33:43.546 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:33:43.545+0000 7f9de8c0a440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-18T15:33:43.546 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:33:43.545+0000 7f9de8c0a440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-18T15:33:43.621 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:33:43.620+0000 7f9de8c0a440 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-09-18T15:33:49.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-09-18T15:33:49.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:33:49.401 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T15:33:49.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T15:33:49.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:33:49.919 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T15:33:49.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T15:33:49.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:33:49.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:33:49.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:33:49.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:33:49.942 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:33:49.941+0000 7f56f8e8b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:33:49.944 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:33:49.943+0000 7f56f8e8b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:33:49.945 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:33:49.944+0000 7f56f8e8b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:33:50.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T15:33:50.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:33:50.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:33:50.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:33:50.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:33:50.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:50.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:33:50.325 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:33:50.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:50.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:33:50.464 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:33:50.463+0000 7f56f8e8b440 -1 Falling back to public interface 2024-09-18T15:33:50.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:33:51.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:33:51.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:51.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:33:51.734 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:33:51.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:51.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:33:52.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:33:52.957 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:33:52.956+0000 7f56f8e8b440 -1 osd.1 0 log_to_monitors true 2024-09-18T15:33:53.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:33:53.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:53.127 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:33:53.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:33:53.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:53.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:33:53.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:33:54.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:33:54.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:54.771 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:33:54.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:33:54.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:54.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:33:55.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:33:56.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:33:56.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:56.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:33:56.168 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:33:56.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:56.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:33:56.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:33:57.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:33:57.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:57.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:33:57.593 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:33:57.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:57.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:33:58.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:33:59.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:33:59.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:33:59.211 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:33:59.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:33:59.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:33:59.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:33:59.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:00.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:00.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:00.588 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:34:00.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:34:00.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:00.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:34:01.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:02.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:02.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:02.067 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:34:02.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:34:02.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:02.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:34:02.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:03.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:03.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:03.502 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:34:03.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:34:03.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:03.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:34:03.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:04.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:04.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:04.957 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:34:04.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:34:04.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:04.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:34:05.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:06.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:06.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:06.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:34:06.588 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:34:06.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:06.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:34:07.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:08.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:08.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:08.158 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:34:08.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:34:08.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:08.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:34:09.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:10.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:10.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:10.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:34:10.105 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:34:10.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:10.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:34:11.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:12.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:12.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:12.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:34:12.091 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:34:12.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:12.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:34:13.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:14.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:14.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:14.163 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:34:14.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:34:14.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:14.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:34:15.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:16.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:16.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:16.138 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:34:16.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:34:16.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:16.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:34:16.544 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/372653,v1:127.0.0.1:6811/372653] [v2:127.0.0.1:6812/372653,v1:127.0.0.1:6813/372653] exists,up 24014b91-356d-42ac-913a-34203f790be8 2024-09-18T15:34:16.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:34:16.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:34:16.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:34:16.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:192: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 2 2024-09-18T15:34:16.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:34:16.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:34:16.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T15:34:16.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:34:16.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-09-18T15:34:16.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:34:16.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:34:16.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:34:16.549 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:34:16.549 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-09-18T15:34:16.549 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-09-18T15:34:16.549 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:34:16.549 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:34:16.549 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:34:16.549 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:34:16.549 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:34:16.549 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:34:16.549 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:34:16.549 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:34:16.549 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:34:16.549 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:34:16.549 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:34:16.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:34:16.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:34:16.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:34:16.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:34:16.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:34:16.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:34:16.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:34:16.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:34:16.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:34:16.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:34:16.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:34:16.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-09-18T15:34:16.550 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:34:16.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6a1d2db9-1a2b-4bc7-9584-c892a30fe89a 2024-09-18T15:34:16.551 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 6a1d2db9-1a2b-4bc7-9584-c892a30fe89a' 2024-09-18T15:34:16.551 INFO:tasks.workunit.client.0.smithi047.stdout:add osd2 6a1d2db9-1a2b-4bc7-9584-c892a30fe89a 2024-09-18T15:34:16.551 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:34:16.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD48upmlzt5IRAAzykmslqCg2rlZPpfz10rDA== 2024-09-18T15:34:16.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD48upmlzt5IRAAzykmslqCg2rlZPpfz10rDA=="}' 2024-09-18T15:34:16.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6a1d2db9-1a2b-4bc7-9584-c892a30fe89a -i td/osd-recovery-stats/2/new.json 2024-09-18T15:34:17.052 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:34:17.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-09-18T15:34:17.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD48upmlzt5IRAAzykmslqCg2rlZPpfz10rDA== --osd-uuid 6a1d2db9-1a2b-4bc7-9584-c892a30fe89a 2024-09-18T15:34:17.088 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:34:17.087+0000 7fd8b7018440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:34:17.090 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:34:17.089+0000 7fd8b7018440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:34:17.091 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:34:17.090+0000 7fd8b7018440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:34:17.092 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:34:17.091+0000 7fd8b7018440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-18T15:34:17.092 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:34:17.091+0000 7fd8b7018440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-18T15:34:17.092 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:34:17.091+0000 7fd8b7018440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-18T15:34:17.167 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:34:17.167+0000 7fd8b7018440 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-09-18T15:34:23.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-09-18T15:34:23.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:34:23.315 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd2 key to auth repository 2024-09-18T15:34:23.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T15:34:23.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:34:23.837 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T15:34:23.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T15:34:23.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:34:23.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:34:23.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:34:23.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:34:23.860 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:34:23.859+0000 7fee95424440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:34:23.865 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:34:23.864+0000 7fee95424440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:34:23.866 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:34:23.865+0000 7fee95424440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:34:24.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T15:34:24.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:34:24.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:34:24.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:34:24.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:34:24.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:24.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:34:24.263 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:34:24.264 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:24.264 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:34:24.400 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:34:24.399+0000 7fee95424440 -1 Falling back to public interface 2024-09-18T15:34:24.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:25.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:25.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:25.673 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:34:25.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:34:25.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:25.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:34:26.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:26.912 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:34:26.912+0000 7fee95424440 -1 osd.2 0 log_to_monitors true 2024-09-18T15:34:27.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:27.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:27.089 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:34:27.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:34:27.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:27.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:34:27.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:28.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:28.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:28.590 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:34:28.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:34:28.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:28.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:34:28.985 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:29.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:29.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:29.986 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:34:29.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:34:29.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:29.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:34:30.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:31.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:31.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:31.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:34:31.402 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:34:31.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:31.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:34:31.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:32.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:32.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:32.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:34:32.865 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:34:32.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:32.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:34:33.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:34.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:34.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:34.262 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:34:34.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:34:34.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:34.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:34:34.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:35.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:35.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:35.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:34:35.687 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:34:35.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:35.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:34:36.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:37.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:37.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:37.126 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:34:37.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:34:37.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:37.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:34:37.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:38.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:38.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:38.530 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:34:38.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:34:38.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:38.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:34:38.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:39.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:39.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:39.944 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:34:39.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:34:39.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:39.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:34:40.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:41.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:41.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:41.690 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:34:41.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:34:41.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:41.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:34:42.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:43.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:43.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:43.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:34:43.461 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:34:43.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:43.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:34:44.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:45.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:45.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:45.308 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:34:45.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:34:45.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:45.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:34:45.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:46.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:46.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:46.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:34:46.975 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:34:46.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:46.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:34:47.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:48.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:48.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:48.610 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:34:48.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:34:48.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:48.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:34:49.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:50.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:50.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:50.270 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T15:34:50.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:34:50.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:50.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:34:50.692 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/374382,v1:127.0.0.1:6819/374382] [v2:127.0.0.1:6820/374382,v1:127.0.0.1:6821/374382] exists,up 6a1d2db9-1a2b-4bc7-9584-c892a30fe89a 2024-09-18T15:34:50.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:34:50.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:34:50.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:34:50.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:193: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 3 2024-09-18T15:34:50.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:34:50.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:34:50.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-18T15:34:50.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:34:50.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-09-18T15:34:50.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:34:50.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:34:50.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:34:50.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:34:50.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-09-18T15:34:50.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-09-18T15:34:50.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:34:50.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:34:50.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:34:50.697 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:34:50.697 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:34:50.697 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:34:50.697 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:34:50.697 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:34:50.697 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:34:50.697 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:34:50.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:34:50.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:34:50.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:34:50.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:34:50.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:34:50.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:34:50.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:34:50.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:34:50.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:34:50.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:34:50.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:34:50.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:34:50.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-09-18T15:34:50.698 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:34:50.699 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bb4bcea0-42b5-4891-86fa-2c0043b59a59 2024-09-18T15:34:50.699 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 bb4bcea0-42b5-4891-86fa-2c0043b59a59' 2024-09-18T15:34:50.699 INFO:tasks.workunit.client.0.smithi047.stdout:add osd3 bb4bcea0-42b5-4891-86fa-2c0043b59a59 2024-09-18T15:34:50.699 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:34:50.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAa8+pm3IdUKhAA7tuhtcKG6Vb7ikcYfrHU+w== 2024-09-18T15:34:50.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAa8+pm3IdUKhAA7tuhtcKG6Vb7ikcYfrHU+w=="}' 2024-09-18T15:34:50.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bb4bcea0-42b5-4891-86fa-2c0043b59a59 -i td/osd-recovery-stats/3/new.json 2024-09-18T15:34:51.242 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:34:51.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-09-18T15:34:51.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAa8+pm3IdUKhAA7tuhtcKG6Vb7ikcYfrHU+w== --osd-uuid bb4bcea0-42b5-4891-86fa-2c0043b59a59 2024-09-18T15:34:51.278 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:34:51.277+0000 7f6d0c926440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:34:51.280 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:34:51.279+0000 7f6d0c926440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:34:51.282 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:34:51.280+0000 7f6d0c926440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:34:51.282 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:34:51.281+0000 7f6d0c926440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-18T15:34:51.282 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:34:51.281+0000 7f6d0c926440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-18T15:34:51.282 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:34:51.281+0000 7f6d0c926440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-18T15:34:51.324 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:34:51.323+0000 7f6d0c926440 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-09-18T15:34:57.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-09-18T15:34:57.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:34:57.273 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd3 key to auth repository 2024-09-18T15:34:57.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-18T15:34:57.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:34:57.793 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.3 2024-09-18T15:34:57.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-18T15:34:57.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:34:57.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:34:57.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:34:57.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:34:57.816 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:34:57.815+0000 7fa3a9323440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:34:57.818 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:34:57.817+0000 7fa3a9323440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:34:57.819 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:34:57.818+0000 7fa3a9323440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:34:58.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-18T15:34:58.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:34:58.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T15:34:58.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:34:58.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:34:58.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:58.204 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:34:58.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:34:58.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:58.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:34:58.336 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:34:58.335+0000 7fa3a9323440 -1 Falling back to public interface 2024-09-18T15:34:58.612 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:34:59.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:34:59.614 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:34:59.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:34:59.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:34:59.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:34:59.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:34:59.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:00.829 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:35:00.828+0000 7fa3a9323440 -1 osd.3 0 log_to_monitors true 2024-09-18T15:35:00.988 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:00.988 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:00.988 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:35:00.989 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:35:00.989 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:00.989 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:35:01.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:02.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:02.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:02.531 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:35:02.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:35:02.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:02.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:35:03.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:04.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:04.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:04.011 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:35:04.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:35:04.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:04.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:35:04.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:05.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:05.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:05.450 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:35:05.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:35:05.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:05.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:35:06.157 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:07.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:07.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:07.159 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:35:07.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:35:07.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:07.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:35:07.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:08.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:08.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:08.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:35:08.615 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:35:08.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:08.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:35:09.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:10.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:10.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:10.044 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:35:10.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:35:10.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:10.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:35:10.432 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:11.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:11.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:11.433 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:35:11.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:35:11.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:11.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:35:11.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:12.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:12.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:12.957 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:35:12.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:35:12.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:12.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:35:13.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:14.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:14.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:14.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:35:14.564 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:35:14.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:14.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:35:15.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:16.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:16.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:16.283 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:35:16.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:35:16.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:16.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:35:17.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:18.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:18.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:18.027 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:35:18.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:35:18.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:18.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:35:18.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:19.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:19.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:19.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:35:19.744 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:35:19.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:19.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:35:20.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:21.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:21.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:21.218 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:35:21.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:35:21.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:21.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:35:21.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:22.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:22.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:22.954 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:35:22.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:35:22.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:22.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:35:23.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:24.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:24.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:24.764 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T15:35:24.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:35:24.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:24.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:35:25.157 INFO:tasks.workunit.client.0.smithi047.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/376136,v1:127.0.0.1:6827/376136] [v2:127.0.0.1:6828/376136,v1:127.0.0.1:6829/376136] exists,up bb4bcea0-42b5-4891-86fa-2c0043b59a59 2024-09-18T15:35:25.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:35:25.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:35:25.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:35:25.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:194: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 4 2024-09-18T15:35:25.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:35:25.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:35:25.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-18T15:35:25.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:35:25.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2024-09-18T15:35:25.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:35:25.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:35:25.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:35:25.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:35:25.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2024-09-18T15:35:25.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2024-09-18T15:35:25.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:35:25.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:35:25.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:35:25.163 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:35:25.163 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:35:25.164 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:35:25.164 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:35:25.164 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:35:25.164 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:35:25.164 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:35:25.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:35:25.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:35:25.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:35:25.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:35:25.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:35:25.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:35:25.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:35:25.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:35:25.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:35:25.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:35:25.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:35:25.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:35:25.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2024-09-18T15:35:25.165 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:35:25.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7bd934e9-15c7-4c2b-afef-68a457660f58 2024-09-18T15:35:25.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 7bd934e9-15c7-4c2b-afef-68a457660f58' 2024-09-18T15:35:25.166 INFO:tasks.workunit.client.0.smithi047.stdout:add osd4 7bd934e9-15c7-4c2b-afef-68a457660f58 2024-09-18T15:35:25.166 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:35:25.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA98+pmTFR5ChAA5MoHXS263JGukb0iYH6btg== 2024-09-18T15:35:25.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA98+pmTFR5ChAA5MoHXS263JGukb0iYH6btg=="}' 2024-09-18T15:35:25.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7bd934e9-15c7-4c2b-afef-68a457660f58 -i td/osd-recovery-stats/4/new.json 2024-09-18T15:35:25.718 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:35:25.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2024-09-18T15:35:25.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA98+pmTFR5ChAA5MoHXS263JGukb0iYH6btg== --osd-uuid 7bd934e9-15c7-4c2b-afef-68a457660f58 2024-09-18T15:35:25.755 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:35:25.754+0000 7f0074c2c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:35:25.756 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:35:25.756+0000 7f0074c2c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:35:25.758 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:35:25.757+0000 7f0074c2c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:35:25.758 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:35:25.757+0000 7f0074c2c440 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-18T15:35:25.758 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:35:25.757+0000 7f0074c2c440 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-18T15:35:25.758 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:35:25.757+0000 7f0074c2c440 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-18T15:35:25.835 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:35:25.834+0000 7f0074c2c440 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2024-09-18T15:35:31.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2024-09-18T15:35:31.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:35:31.694 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd4 key to auth repository 2024-09-18T15:35:31.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-18T15:35:31.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:35:32.205 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.4 2024-09-18T15:35:32.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-18T15:35:32.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:35:32.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:35:32.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:35:32.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:35:32.228 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:35:32.227+0000 7f91fe362440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:35:32.230 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:35:32.229+0000 7f91fe362440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:35:32.231 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:35:32.230+0000 7f91fe362440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:35:32.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-18T15:35:32.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:35:32.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-18T15:35:32.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:35:32.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:35:32.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:32.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:35:32.570 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:35:32.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:32.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:35:32.753 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:35:32.752+0000 7f91fe362440 -1 Falling back to public interface 2024-09-18T15:35:32.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:33.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:33.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:33.978 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:35:33.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:35:33.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:33.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:35:34.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:35.285 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:35:35.284+0000 7f91fe362440 -1 osd.4 0 log_to_monitors true 2024-09-18T15:35:35.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:35.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:35.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:35:35.397 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:35:35.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:35.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:35:35.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:36.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:36.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:36.998 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:35:36.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:35:36.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:36.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:35:37.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:38.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:38.527 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:38.528 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:35:38.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:35:38.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:38.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:35:38.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:39.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:39.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:39.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:35:39.784 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:35:39.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:39.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:35:40.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:41.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:41.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:41.187 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:35:41.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:35:41.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:41.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:35:41.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:42.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:42.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:42.563 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:35:42.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:35:42.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:42.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:35:43.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:44.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:44.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:44.089 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:35:44.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:35:44.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:44.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:35:44.492 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:45.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:45.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:45.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:35:45.494 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:35:45.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:45.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:35:46.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:47.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:47.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:47.143 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:35:47.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:35:47.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:47.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:35:47.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:48.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:48.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:48.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:35:48.658 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:35:48.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:48.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:35:49.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:50.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:50.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:50.385 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:35:50.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:35:50.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:50.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:35:51.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:52.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:52.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:52.113 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:35:52.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:35:52.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:52.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:35:53.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:54.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:54.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:54.161 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:35:54.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:35:54.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:54.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:35:54.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:55.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:55.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:55.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:35:55.881 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:35:55.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:55.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:35:56.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:35:57.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:35:57.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:35:57.609 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:35:57.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:35:57.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:35:57.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T15:35:58.013 INFO:tasks.workunit.client.0.smithi047.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/377890,v1:127.0.0.1:6835/377890] [v2:127.0.0.1:6836/377890,v1:127.0.0.1:6837/377890] exists,up 7bd934e9-15c7-4c2b-afef-68a457660f58 2024-09-18T15:35:58.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:35:58.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:35:58.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:35:58.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:195: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 5 2024-09-18T15:35:58.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:35:58.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:35:58.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-18T15:35:58.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:35:58.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2024-09-18T15:35:58.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:35:58.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:35:58.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:35:58.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:35:58.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2024-09-18T15:35:58.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2024-09-18T15:35:58.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:35:58.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:35:58.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:35:58.018 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:35:58.018 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:35:58.018 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:35:58.018 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:35:58.018 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:35:58.018 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:35:58.018 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:35:58.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:35:58.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:35:58.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:35:58.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:35:58.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:35:58.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:35:58.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:35:58.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:35:58.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:35:58.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:35:58.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:35:58.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:35:58.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2024-09-18T15:35:58.020 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:35:58.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6da79515-41b8-49fc-a69b-d1690f8ac9b1 2024-09-18T15:35:58.020 INFO:tasks.workunit.client.0.smithi047.stdout:add osd5 6da79515-41b8-49fc-a69b-d1690f8ac9b1 2024-09-18T15:35:58.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 6da79515-41b8-49fc-a69b-d1690f8ac9b1' 2024-09-18T15:35:58.020 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:35:58.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBe8+pmKBnVARAAhyfQxI+SQ9GYRAO/T5vueg== 2024-09-18T15:35:58.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBe8+pmKBnVARAAhyfQxI+SQ9GYRAO/T5vueg=="}' 2024-09-18T15:35:58.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6da79515-41b8-49fc-a69b-d1690f8ac9b1 -i td/osd-recovery-stats/5/new.json 2024-09-18T15:35:58.484 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:35:58.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2024-09-18T15:35:58.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBe8+pmKBnVARAAhyfQxI+SQ9GYRAO/T5vueg== --osd-uuid 6da79515-41b8-49fc-a69b-d1690f8ac9b1 2024-09-18T15:35:58.520 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:35:58.519+0000 7f27c1812440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:35:58.538 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:35:58.537+0000 7f27c1812440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:35:58.539 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:35:58.538+0000 7f27c1812440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:35:58.539 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:35:58.538+0000 7f27c1812440 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-18T15:35:58.539 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:35:58.538+0000 7f27c1812440 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-18T15:35:58.539 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:35:58.538+0000 7f27c1812440 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-18T15:35:58.578 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:35:58.577+0000 7f27c1812440 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2024-09-18T15:36:04.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2024-09-18T15:36:04.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:36:04.689 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd5 key to auth repository 2024-09-18T15:36:04.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-18T15:36:04.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:36:05.202 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.5 2024-09-18T15:36:05.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-18T15:36:05.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:36:05.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:36:05.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:36:05.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:36:05.225 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:36:05.224+0000 7f64e3e09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:36:05.247 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:36:05.246+0000 7f64e3e09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:36:05.248 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:36:05.247+0000 7f64e3e09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:36:05.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-18T15:36:05.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:36:05.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-18T15:36:05.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:36:05.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:36:05.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:36:05.635 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:36:05.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:36:05.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:36:05.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:36:05.781 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:36:05.780+0000 7f64e3e09440 -1 Falling back to public interface 2024-09-18T15:36:06.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:36:07.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:36:07.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:36:07.005 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:36:07.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:36:07.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:36:07.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:36:07.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:36:08.362 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:36:08.361+0000 7f64e3e09440 -1 osd.5 0 log_to_monitors true 2024-09-18T15:36:08.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:36:08.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:36:08.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:36:08.423 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:36:08.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:36:08.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:36:09.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:36:10.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:36:10.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:36:10.042 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:36:10.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:36:10.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:36:10.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:36:10.420 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:36:11.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:36:11.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:36:11.422 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:36:11.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:36:11.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:36:11.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:36:11.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:36:12.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:36:12.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:36:12.810 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:36:12.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:36:12.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:36:12.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:36:13.299 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:36:14.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:36:14.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:36:14.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:36:14.301 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:36:14.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:36:14.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:36:14.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:36:15.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:36:15.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:36:15.710 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:36:15.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:36:15.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:36:15.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:36:16.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:36:17.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:36:17.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:36:17.226 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:36:17.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:36:17.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:36:17.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:36:17.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:36:18.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:36:18.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:36:18.629 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:36:18.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:36:18.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:36:18.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:36:19.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:36:20.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:36:20.129 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:36:20.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:36:20.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:36:20.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:36:20.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:36:20.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:36:21.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:36:21.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:36:21.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:36:21.628 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:36:21.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:36:21.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:36:22.390 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:36:23.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:36:23.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:36:23.391 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:36:23.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:36:23.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:36:23.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:36:23.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:36:24.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:36:24.778 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:36:24.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:36:24.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:36:24.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:36:24.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:36:25.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:36:26.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:36:26.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:36:26.486 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:36:26.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:36:26.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:36:26.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:36:27.299 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:36:28.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:36:28.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:36:28.301 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:36:28.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:36:28.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:36:28.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:36:29.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:36:30.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:36:30.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:36:30.027 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:36:30.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:36:30.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:36:30.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:36:30.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:36:31.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:36:31.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:36:31.953 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T15:36:31.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:36:31.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:36:31.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T15:36:32.479 INFO:tasks.workunit.client.0.smithi047.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/379620,v1:127.0.0.1:6843/379620] [v2:127.0.0.1:6844/379620,v1:127.0.0.1:6845/379620] exists,up 6da79515-41b8-49fc-a69b-d1690f8ac9b1 2024-09-18T15:36:32.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:36:32.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:36:32.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:36:32.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:197: TEST_recovery_sizeup: create_pool test 1 1 2024-09-18T15:36:32.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-18T15:36:33.097 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test' already exists 2024-09-18T15:36:33.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:36:34.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:198: TEST_recovery_sizeup: ceph osd pool set test size 2 2024-09-18T15:36:35.217 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 1 size to 2 2024-09-18T15:36:35.231 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:200: TEST_recovery_sizeup: wait_for_clean 2024-09-18T15:36:35.231 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:36:35.231 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:36:35.231 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:36:35.232 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:36:35.233 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:36:35.233 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:36:35.233 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:36:35.233 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:36:35.233 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:36:35.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:36:35.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:36:35.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:36:35.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:36:35.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:36:35.330 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:36:35.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:36:35.839 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:36:35.839 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T15:36:35.839 INFO:tasks.workunit.client.0.smithi047.stderr:3 2024-09-18T15:36:35.839 INFO:tasks.workunit.client.0.smithi047.stderr:4 2024-09-18T15:36:35.840 INFO:tasks.workunit.client.0.smithi047.stderr:5' 2024-09-18T15:36:35.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:36:35.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:36:35.840 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:36:36.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836521 2024-09-18T15:36:36.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836521 2024-09-18T15:36:36.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521' 2024-09-18T15:36:36.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:36:36.023 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:36:36.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672996 2024-09-18T15:36:36.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672996 2024-09-18T15:36:36.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521 1-42949672996' 2024-09-18T15:36:36.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:36:36.218 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:36:36.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509467 2024-09-18T15:36:36.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509467 2024-09-18T15:36:36.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521 1-42949672996 2-64424509467' 2024-09-18T15:36:36.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:36:36.326 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:36:36.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345942 2024-09-18T15:36:36.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345942 2024-09-18T15:36:36.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521 1-42949672996 2-64424509467 3-85899345942' 2024-09-18T15:36:36.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:36:36.434 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:36:36.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182413 2024-09-18T15:36:36.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182413 2024-09-18T15:36:36.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521 1-42949672996 2-64424509467 3-85899345942 4-107374182413' 2024-09-18T15:36:36.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:36:36.542 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:36:36.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018891 2024-09-18T15:36:36.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018891 2024-09-18T15:36:36.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521 1-42949672996 2-64424509467 3-85899345942 4-107374182413 5-128849018891' 2024-09-18T15:36:36.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:36:36.652 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836521 2024-09-18T15:36:36.652 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:36:36.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:36:36.653 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836521 2024-09-18T15:36:36.653 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:36:36.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836521 2024-09-18T15:36:36.654 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836521 2024-09-18T15:36:36.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836521' 2024-09-18T15:36:36.655 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:36:37.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836519 -lt 21474836521 2024-09-18T15:36:37.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:36:38.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:36:38.067 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:36:38.476 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836522 -lt 21474836521 2024-09-18T15:36:38.476 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:36:38.476 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672996 2024-09-18T15:36:38.476 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:36:38.477 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:36:38.478 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672996 2024-09-18T15:36:38.478 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:36:38.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672996 2024-09-18T15:36:38.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672996' 2024-09-18T15:36:38.479 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672996 2024-09-18T15:36:38.479 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:36:38.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672997 -lt 42949672996 2024-09-18T15:36:38.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:36:38.846 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509467 2024-09-18T15:36:38.846 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:36:38.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:36:38.847 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509467 2024-09-18T15:36:38.847 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:36:38.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509467 2024-09-18T15:36:38.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509467' 2024-09-18T15:36:38.849 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509467 2024-09-18T15:36:38.849 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:36:39.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509467 -lt 64424509467 2024-09-18T15:36:39.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:36:39.215 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345942 2024-09-18T15:36:39.215 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:36:39.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:36:39.217 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345942 2024-09-18T15:36:39.217 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:36:39.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345942 2024-09-18T15:36:39.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345942' 2024-09-18T15:36:39.218 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 85899345942 2024-09-18T15:36:39.218 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:36:39.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345942 -lt 85899345942 2024-09-18T15:36:39.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:36:39.589 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182413 2024-09-18T15:36:39.589 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:36:39.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:36:39.590 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182413 2024-09-18T15:36:39.590 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:36:39.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182413 2024-09-18T15:36:39.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182413' 2024-09-18T15:36:39.592 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.4 seq 107374182413 2024-09-18T15:36:39.592 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:36:39.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182414 -lt 107374182413 2024-09-18T15:36:40.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:36:40.000 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018891 2024-09-18T15:36:40.000 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:36:40.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:36:40.001 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018891 2024-09-18T15:36:40.002 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:36:40.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018891 2024-09-18T15:36:40.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018891' 2024-09-18T15:36:40.003 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.5 seq 128849018891 2024-09-18T15:36:40.003 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:36:40.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018892 -lt 128849018891 2024-09-18T15:36:40.389 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:36:40.389 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:36:40.389 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:36:40.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:36:40.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:36:40.881 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:36:40.881 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:36:40.881 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:36:40.881 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:36:40.881 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:36:40.881 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:36:41.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:36:41.277 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:36:41.278 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:36:41.278 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:36:41.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:36:41.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:36:41.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:36:41.849 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: seq 1 200 2024-09-18T15:36:41.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:41.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj1 /dev/null 2024-09-18T15:36:41.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:41.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj2 /dev/null 2024-09-18T15:36:42.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:42.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj3 /dev/null 2024-09-18T15:36:42.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:42.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj4 /dev/null 2024-09-18T15:36:42.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:42.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj5 /dev/null 2024-09-18T15:36:42.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:42.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj6 /dev/null 2024-09-18T15:36:42.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:42.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj7 /dev/null 2024-09-18T15:36:42.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:42.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj8 /dev/null 2024-09-18T15:36:42.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:42.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj9 /dev/null 2024-09-18T15:36:42.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:42.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj10 /dev/null 2024-09-18T15:36:42.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:42.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj11 /dev/null 2024-09-18T15:36:42.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:42.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj12 /dev/null 2024-09-18T15:36:43.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:43.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj13 /dev/null 2024-09-18T15:36:43.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:43.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj14 /dev/null 2024-09-18T15:36:43.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:43.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj15 /dev/null 2024-09-18T15:36:43.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:43.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj16 /dev/null 2024-09-18T15:36:43.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:43.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj17 /dev/null 2024-09-18T15:36:43.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:43.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj18 /dev/null 2024-09-18T15:36:43.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:43.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj19 /dev/null 2024-09-18T15:36:43.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:43.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj20 /dev/null 2024-09-18T15:36:43.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:43.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj21 /dev/null 2024-09-18T15:36:44.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:44.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj22 /dev/null 2024-09-18T15:36:44.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:44.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj23 /dev/null 2024-09-18T15:36:44.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:44.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj24 /dev/null 2024-09-18T15:36:44.355 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:44.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj25 /dev/null 2024-09-18T15:36:44.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:44.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj26 /dev/null 2024-09-18T15:36:44.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:44.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj27 /dev/null 2024-09-18T15:36:44.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:44.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj28 /dev/null 2024-09-18T15:36:44.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:44.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj29 /dev/null 2024-09-18T15:36:44.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:44.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj30 /dev/null 2024-09-18T15:36:44.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:45.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj31 /dev/null 2024-09-18T15:36:45.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:45.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj32 /dev/null 2024-09-18T15:36:45.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:45.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj33 /dev/null 2024-09-18T15:36:45.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:45.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj34 /dev/null 2024-09-18T15:36:45.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:45.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj35 /dev/null 2024-09-18T15:36:45.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:45.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj36 /dev/null 2024-09-18T15:36:45.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:45.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj37 /dev/null 2024-09-18T15:36:45.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:45.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj38 /dev/null 2024-09-18T15:36:45.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:45.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj39 /dev/null 2024-09-18T15:36:45.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:45.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj40 /dev/null 2024-09-18T15:36:46.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:46.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj41 /dev/null 2024-09-18T15:36:46.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:46.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj42 /dev/null 2024-09-18T15:36:46.231 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:46.231 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj43 /dev/null 2024-09-18T15:36:46.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:46.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj44 /dev/null 2024-09-18T15:36:46.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:46.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj45 /dev/null 2024-09-18T15:36:46.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:46.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj46 /dev/null 2024-09-18T15:36:46.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:46.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj47 /dev/null 2024-09-18T15:36:46.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:46.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj48 /dev/null 2024-09-18T15:36:46.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:46.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj49 /dev/null 2024-09-18T15:36:46.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:46.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj50 /dev/null 2024-09-18T15:36:47.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:47.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj51 /dev/null 2024-09-18T15:36:47.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:47.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj52 /dev/null 2024-09-18T15:36:47.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:47.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj53 /dev/null 2024-09-18T15:36:47.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:47.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj54 /dev/null 2024-09-18T15:36:47.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:47.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj55 /dev/null 2024-09-18T15:36:47.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:47.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj56 /dev/null 2024-09-18T15:36:47.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:47.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj57 /dev/null 2024-09-18T15:36:47.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:47.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj58 /dev/null 2024-09-18T15:36:47.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:47.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj59 /dev/null 2024-09-18T15:36:47.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:47.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj60 /dev/null 2024-09-18T15:36:48.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:48.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj61 /dev/null 2024-09-18T15:36:48.157 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:48.157 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj62 /dev/null 2024-09-18T15:36:48.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:48.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj63 /dev/null 2024-09-18T15:36:48.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:48.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj64 /dev/null 2024-09-18T15:36:48.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:48.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj65 /dev/null 2024-09-18T15:36:48.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:48.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj66 /dev/null 2024-09-18T15:36:48.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:48.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj67 /dev/null 2024-09-18T15:36:48.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:48.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj68 /dev/null 2024-09-18T15:36:48.826 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:48.826 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj69 /dev/null 2024-09-18T15:36:48.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:48.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj70 /dev/null 2024-09-18T15:36:49.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:49.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj71 /dev/null 2024-09-18T15:36:49.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:49.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj72 /dev/null 2024-09-18T15:36:49.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:49.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj73 /dev/null 2024-09-18T15:36:49.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:49.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj74 /dev/null 2024-09-18T15:36:49.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:49.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj75 /dev/null 2024-09-18T15:36:49.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:49.514 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj76 /dev/null 2024-09-18T15:36:49.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:49.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj77 /dev/null 2024-09-18T15:36:49.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:49.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj78 /dev/null 2024-09-18T15:36:49.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:49.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj79 /dev/null 2024-09-18T15:36:49.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:49.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj80 /dev/null 2024-09-18T15:36:50.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:50.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj81 /dev/null 2024-09-18T15:36:50.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:50.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj82 /dev/null 2024-09-18T15:36:50.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:50.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj83 /dev/null 2024-09-18T15:36:50.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:50.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj84 /dev/null 2024-09-18T15:36:50.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:50.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj85 /dev/null 2024-09-18T15:36:50.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:50.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj86 /dev/null 2024-09-18T15:36:50.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:50.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj87 /dev/null 2024-09-18T15:36:50.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:50.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj88 /dev/null 2024-09-18T15:36:50.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:50.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj89 /dev/null 2024-09-18T15:36:50.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:50.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj90 /dev/null 2024-09-18T15:36:51.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:51.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj91 /dev/null 2024-09-18T15:36:51.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:51.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj92 /dev/null 2024-09-18T15:36:51.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:51.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj93 /dev/null 2024-09-18T15:36:51.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:51.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj94 /dev/null 2024-09-18T15:36:51.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:51.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj95 /dev/null 2024-09-18T15:36:51.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:51.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj96 /dev/null 2024-09-18T15:36:51.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:51.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj97 /dev/null 2024-09-18T15:36:51.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:51.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj98 /dev/null 2024-09-18T15:36:51.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:51.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj99 /dev/null 2024-09-18T15:36:51.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:51.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj100 /dev/null 2024-09-18T15:36:52.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:52.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj101 /dev/null 2024-09-18T15:36:52.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:52.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj102 /dev/null 2024-09-18T15:36:52.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:52.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj103 /dev/null 2024-09-18T15:36:52.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:52.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj104 /dev/null 2024-09-18T15:36:52.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:52.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj105 /dev/null 2024-09-18T15:36:52.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:52.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj106 /dev/null 2024-09-18T15:36:52.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:52.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj107 /dev/null 2024-09-18T15:36:52.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:52.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj108 /dev/null 2024-09-18T15:36:52.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:52.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj109 /dev/null 2024-09-18T15:36:52.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:52.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj110 /dev/null 2024-09-18T15:36:53.022 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:53.022 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj111 /dev/null 2024-09-18T15:36:53.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:53.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj112 /dev/null 2024-09-18T15:36:53.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:53.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj113 /dev/null 2024-09-18T15:36:53.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:53.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj114 /dev/null 2024-09-18T15:36:53.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:53.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj115 /dev/null 2024-09-18T15:36:53.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:53.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj116 /dev/null 2024-09-18T15:36:53.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:53.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj117 /dev/null 2024-09-18T15:36:53.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:53.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj118 /dev/null 2024-09-18T15:36:53.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:53.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj119 /dev/null 2024-09-18T15:36:53.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:53.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj120 /dev/null 2024-09-18T15:36:54.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:54.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj121 /dev/null 2024-09-18T15:36:54.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:54.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj122 /dev/null 2024-09-18T15:36:54.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:54.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj123 /dev/null 2024-09-18T15:36:54.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:54.355 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj124 /dev/null 2024-09-18T15:36:54.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:54.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj125 /dev/null 2024-09-18T15:36:54.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:54.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj126 /dev/null 2024-09-18T15:36:54.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:54.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj127 /dev/null 2024-09-18T15:36:54.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:54.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj128 /dev/null 2024-09-18T15:36:54.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:54.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj129 /dev/null 2024-09-18T15:36:54.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:54.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj130 /dev/null 2024-09-18T15:36:55.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:55.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj131 /dev/null 2024-09-18T15:36:55.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:55.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj132 /dev/null 2024-09-18T15:36:55.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:55.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj133 /dev/null 2024-09-18T15:36:55.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:55.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj134 /dev/null 2024-09-18T15:36:55.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:55.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj135 /dev/null 2024-09-18T15:36:55.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:55.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj136 /dev/null 2024-09-18T15:36:55.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:55.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj137 /dev/null 2024-09-18T15:36:55.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:55.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj138 /dev/null 2024-09-18T15:36:55.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:55.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj139 /dev/null 2024-09-18T15:36:56.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:56.021 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj140 /dev/null 2024-09-18T15:36:56.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:56.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj141 /dev/null 2024-09-18T15:36:56.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:56.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj142 /dev/null 2024-09-18T15:36:56.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:56.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj143 /dev/null 2024-09-18T15:36:56.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:56.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj144 /dev/null 2024-09-18T15:36:56.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:56.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj145 /dev/null 2024-09-18T15:36:56.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:56.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj146 /dev/null 2024-09-18T15:36:56.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:56.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj147 /dev/null 2024-09-18T15:36:56.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:56.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj148 /dev/null 2024-09-18T15:36:56.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:56.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj149 /dev/null 2024-09-18T15:36:57.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:57.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj150 /dev/null 2024-09-18T15:36:57.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:57.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj151 /dev/null 2024-09-18T15:36:57.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:57.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj152 /dev/null 2024-09-18T15:36:57.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:57.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj153 /dev/null 2024-09-18T15:36:57.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:57.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj154 /dev/null 2024-09-18T15:36:57.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:57.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj155 /dev/null 2024-09-18T15:36:57.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:57.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj156 /dev/null 2024-09-18T15:36:57.746 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:57.746 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj157 /dev/null 2024-09-18T15:36:57.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:57.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj158 /dev/null 2024-09-18T15:36:57.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:57.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj159 /dev/null 2024-09-18T15:36:58.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:58.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj160 /dev/null 2024-09-18T15:36:58.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:58.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj161 /dev/null 2024-09-18T15:36:58.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:58.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj162 /dev/null 2024-09-18T15:36:58.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:58.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj163 /dev/null 2024-09-18T15:36:58.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:58.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj164 /dev/null 2024-09-18T15:36:58.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:58.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj165 /dev/null 2024-09-18T15:36:58.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:58.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj166 /dev/null 2024-09-18T15:36:58.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:58.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj167 /dev/null 2024-09-18T15:36:58.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:58.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj168 /dev/null 2024-09-18T15:36:58.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:58.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj169 /dev/null 2024-09-18T15:36:59.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:59.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj170 /dev/null 2024-09-18T15:36:59.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:59.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj171 /dev/null 2024-09-18T15:36:59.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:59.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj172 /dev/null 2024-09-18T15:36:59.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:59.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj173 /dev/null 2024-09-18T15:36:59.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:59.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj174 /dev/null 2024-09-18T15:36:59.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:59.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj175 /dev/null 2024-09-18T15:36:59.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:59.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj176 /dev/null 2024-09-18T15:36:59.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:59.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj177 /dev/null 2024-09-18T15:36:59.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:59.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj178 /dev/null 2024-09-18T15:36:59.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:36:59.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj179 /dev/null 2024-09-18T15:37:00.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:37:00.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj180 /dev/null 2024-09-18T15:37:00.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:37:00.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj181 /dev/null 2024-09-18T15:37:00.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:37:00.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj182 /dev/null 2024-09-18T15:37:00.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:37:00.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj183 /dev/null 2024-09-18T15:37:00.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:37:00.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj184 /dev/null 2024-09-18T15:37:00.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:37:00.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj185 /dev/null 2024-09-18T15:37:00.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:37:00.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj186 /dev/null 2024-09-18T15:37:00.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:37:00.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj187 /dev/null 2024-09-18T15:37:00.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:37:00.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj188 /dev/null 2024-09-18T15:37:01.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:37:01.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj189 /dev/null 2024-09-18T15:37:01.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:37:01.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj190 /dev/null 2024-09-18T15:37:01.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:37:01.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj191 /dev/null 2024-09-18T15:37:01.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:37:01.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj192 /dev/null 2024-09-18T15:37:01.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:37:01.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj193 /dev/null 2024-09-18T15:37:01.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:37:01.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj194 /dev/null 2024-09-18T15:37:01.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:37:01.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj195 /dev/null 2024-09-18T15:37:01.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:37:01.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj196 /dev/null 2024-09-18T15:37:01.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:37:01.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj197 /dev/null 2024-09-18T15:37:02.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:37:02.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj198 /dev/null 2024-09-18T15:37:02.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:37:02.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj199 /dev/null 2024-09-18T15:37:02.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-18T15:37:02.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj200 /dev/null 2024-09-18T15:37:02.328 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:207: TEST_recovery_sizeup: get_primary test obj1 2024-09-18T15:37:02.328 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-18T15:37:02.328 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T15:37:02.328 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-18T15:37:02.328 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:37:02.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:207: TEST_recovery_sizeup: local primary=1 2024-09-18T15:37:02.701 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:208: TEST_recovery_sizeup: get_pg test obj1 2024-09-18T15:37:02.702 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-18T15:37:02.702 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-09-18T15:37:02.702 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-09-18T15:37:02.702 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:37:03.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:208: TEST_recovery_sizeup: local PG=1.0 2024-09-18T15:37:03.116 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:210: TEST_recovery_sizeup: get_not_primary test obj1 2024-09-18T15:37:03.116 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-09-18T15:37:03.116 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-09-18T15:37:03.116 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-09-18T15:37:03.116 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-18T15:37:03.116 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T15:37:03.116 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-18T15:37:03.116 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:37:03.533 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-18T15:37:03.533 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-09-18T15:37:03.533 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-18T15:37:03.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:210: TEST_recovery_sizeup: local otherosd=0 2024-09-18T15:37:03.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:212: TEST_recovery_sizeup: ceph osd set norecover 2024-09-18T15:37:04.592 INFO:tasks.workunit.client.0.smithi047.stderr:norecover is set 2024-09-18T15:37:04.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:213: TEST_recovery_sizeup: ceph osd out osd.1 osd.0 2024-09-18T15:37:05.253 INFO:tasks.workunit.client.0.smithi047.stderr:osd.1 is already out. osd.0 is already out. 2024-09-18T15:37:05.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:214: TEST_recovery_sizeup: ceph osd pool set test size 4 2024-09-18T15:37:06.855 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 1 size to 4 2024-09-18T15:37:06.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:215: TEST_recovery_sizeup: ceph osd unset norecover 2024-09-18T15:37:08.138 INFO:tasks.workunit.client.0.smithi047.stderr:norecover is unset 2024-09-18T15:37:08.151 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:217: TEST_recovery_sizeup: get_primary test obj1 2024-09-18T15:37:08.151 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-18T15:37:08.151 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T15:37:08.151 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-18T15:37:08.151 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:37:08.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:217: TEST_recovery_sizeup: primary=2 2024-09-18T15:37:08.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:219: TEST_recovery_sizeup: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-18T15:37:08.876 INFO:tasks.workunit.client.0.smithi047.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-18T15:37:08.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:220: TEST_recovery_sizeup: sleep 2 2024-09-18T15:37:10.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:222: TEST_recovery_sizeup: wait_for_clean 2024-09-18T15:37:10.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:37:10.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:37:10.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:37:10.886 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:37:10.887 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:37:10.887 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:37:10.887 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:37:10.887 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:37:10.887 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:37:10.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:37:10.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:37:10.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:37:10.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:37:10.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:37:10.990 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:37:11.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:37:11.386 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:37:11.386 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T15:37:11.386 INFO:tasks.workunit.client.0.smithi047.stderr:3 2024-09-18T15:37:11.386 INFO:tasks.workunit.client.0.smithi047.stderr:4 2024-09-18T15:37:11.386 INFO:tasks.workunit.client.0.smithi047.stderr:5' 2024-09-18T15:37:11.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:37:11.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:37:11.386 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:37:11.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836530 2024-09-18T15:37:11.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836530 2024-09-18T15:37:11.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836530' 2024-09-18T15:37:11.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:37:11.529 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:37:11.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673006 2024-09-18T15:37:11.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673006 2024-09-18T15:37:11.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836530 1-42949673006' 2024-09-18T15:37:11.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:37:11.646 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:37:11.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509476 2024-09-18T15:37:11.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509476 2024-09-18T15:37:11.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836530 1-42949673006 2-64424509476' 2024-09-18T15:37:11.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:37:11.766 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:37:11.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345951 2024-09-18T15:37:11.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345951 2024-09-18T15:37:11.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836530 1-42949673006 2-64424509476 3-85899345951' 2024-09-18T15:37:11.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:37:11.932 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T15:37:12.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182422 2024-09-18T15:37:12.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182422 2024-09-18T15:37:12.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836530 1-42949673006 2-64424509476 3-85899345951 4-107374182422' 2024-09-18T15:37:12.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:37:12.041 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T15:37:12.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018901 2024-09-18T15:37:12.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018901 2024-09-18T15:37:12.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836530 1-42949673006 2-64424509476 3-85899345951 4-107374182422 5-128849018901' 2024-09-18T15:37:12.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:37:12.150 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836530 2024-09-18T15:37:12.150 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:37:12.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:37:12.151 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836530 2024-09-18T15:37:12.151 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:37:12.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836530 2024-09-18T15:37:12.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836530' 2024-09-18T15:37:12.153 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836530 2024-09-18T15:37:12.153 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:37:12.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836528 -lt 21474836530 2024-09-18T15:37:12.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:37:13.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:37:13.545 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:37:13.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836531 -lt 21474836530 2024-09-18T15:37:13.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:37:13.925 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673006 2024-09-18T15:37:13.925 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:37:13.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:37:13.926 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673006 2024-09-18T15:37:13.926 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:37:13.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673006 2024-09-18T15:37:13.927 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949673006 2024-09-18T15:37:13.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673006' 2024-09-18T15:37:13.928 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:37:14.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673006 -lt 42949673006 2024-09-18T15:37:14.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:37:14.450 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509476 2024-09-18T15:37:14.450 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:37:14.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:37:14.451 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509476 2024-09-18T15:37:14.451 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:37:14.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509476 2024-09-18T15:37:14.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509476' 2024-09-18T15:37:14.453 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509476 2024-09-18T15:37:14.453 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:37:14.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509476 -lt 64424509476 2024-09-18T15:37:14.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:37:14.986 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345951 2024-09-18T15:37:14.986 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:37:14.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:37:14.988 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345951 2024-09-18T15:37:14.988 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:37:14.989 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345951 2024-09-18T15:37:14.989 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 85899345951 2024-09-18T15:37:14.989 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345951' 2024-09-18T15:37:14.989 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:37:15.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345951 -lt 85899345951 2024-09-18T15:37:15.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:37:15.456 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182422 2024-09-18T15:37:15.457 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:37:15.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T15:37:15.458 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182422 2024-09-18T15:37:15.458 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:37:15.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182422 2024-09-18T15:37:15.459 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.4 seq 107374182422 2024-09-18T15:37:15.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182422' 2024-09-18T15:37:15.460 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T15:37:16.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182423 -lt 107374182422 2024-09-18T15:37:16.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:37:16.044 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018901 2024-09-18T15:37:16.044 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:37:16.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T15:37:16.045 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018901 2024-09-18T15:37:16.045 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:37:16.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018901 2024-09-18T15:37:16.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018901' 2024-09-18T15:37:16.047 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.5 seq 128849018901 2024-09-18T15:37:16.047 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T15:37:16.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018901 -lt 128849018901 2024-09-18T15:37:16.567 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:37:16.567 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:37:16.567 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:37:17.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:37:17.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:37:17.179 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:37:17.179 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:37:17.179 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:37:17.179 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:37:17.180 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:37:17.180 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:37:17.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:37:17.538 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:37:17.538 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:37:17.538 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:37:18.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:37:18.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-18T15:37:18.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T15:37:18.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-18T15:37:18.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:37:18.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:37:18.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:37:18.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:37:18.271 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:37:18.271 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:37:18.271 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:37:18.271 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:37:18.271 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:37:18.271 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:37:18.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:37:18.696 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:37:18.697 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:37:18.697 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:37:19.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:37:19.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:37:19.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:37:19.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:37:19.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:37:19.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:37:19.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:37:19.195 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:37:19.195 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:37:19.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-09-18T15:37:19.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-09-18T15:37:19.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:37:19.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:37:19.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:37:19.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:37:19.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:37:19.872 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:37:19.872 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:37:19.872 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:37:19.872 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:37:19.872 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:37:19.872 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:37:20.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:37:20.227 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:37:20.228 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:37:20.228 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:37:20.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:37:20.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:37:20.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:37:20.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:37:20.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:37:20.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:37:20.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:37:20.787 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:37:20.787 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:37:21.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-09-18T15:37:21.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-09-18T15:37:21.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:37:21.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:37:21.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:37:21.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:37:21.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:37:21.483 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:37:21.483 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:37:21.483 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:37:21.483 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:37:21.483 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:37:21.483 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:37:21.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:37:21.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:37:21.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:37:21.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:37:22.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:37:22.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:37:22.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:37:22.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:37:22.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:37:22.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:37:22.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:37:22.421 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:37:22.421 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:37:22.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-09-18T15:37:22.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-09-18T15:37:22.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:37:22.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:37:22.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:37:23.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:37:23.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:37:23.028 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:37:23.028 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:37:23.028 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:37:23.028 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:37:23.028 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:37:23.028 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:37:23.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:37:23.386 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:37:23.386 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:37:23.386 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:37:23.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:37:23.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:37:23.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:37:23.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:37:23.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:37:23.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:37:23.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:37:23.995 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:37:23.995 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:37:24.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-18T15:37:24.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-18T15:37:24.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:37:24.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:37:24.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:37:24.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:37:24.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:37:24.693 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:37:24.693 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:37:24.694 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:37:24.694 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:37:24.694 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:37:24.694 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:37:25.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:37:25.142 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:37:25.142 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:37:25.142 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:37:25.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:37:25.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:37:25.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:37:25.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:37:25.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:37:25.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:37:25.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:37:25.809 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:37:25.809 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:37:26.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-18T15:37:26.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-18T15:37:26.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:37:26.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:37:26.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:37:26.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:37:26.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:37:26.519 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:37:26.519 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:37:26.519 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:37:26.519 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:37:26.519 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:37:26.519 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:37:26.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:37:26.879 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:37:26.879 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:37:26.880 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:37:27.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:37:27.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:37:27.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:37:27.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:37:27.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:37:27.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:37:27.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:37:27.562 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:37:27.562 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:37:28.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-18T15:37:28.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-18T15:37:28.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:37:28.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:37:28.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:37:28.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:37:28.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:37:28.246 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:37:28.246 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:37:28.246 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:37:28.246 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:37:28.246 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:37:28.246 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:37:28.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:37:28.681 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:37:28.682 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:37:28.682 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:37:29.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:37:29.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:37:29.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:37:29.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:37:29.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:37:29.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:37:29.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:37:29.191 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:37:29.191 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:37:29.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-18T15:37:29.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-18T15:37:29.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:37:29.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:37:29.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:37:29.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:37:29.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:37:29.812 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:37:29.812 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:37:29.813 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:37:29.813 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:37:29.813 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:37:29.813 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:37:30.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:37:30.214 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:37:30.214 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:37:30.214 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:37:30.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:37:30.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:37:30.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:37:30.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:37:30.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:37:30.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:37:30.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:37:30.831 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:37:30.831 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:37:31.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-18T15:37:31.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-18T15:37:31.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:37:31.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:37:31.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:37:31.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:37:31.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:37:31.722 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:37:31.722 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:37:31.723 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:37:31.723 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:37:31.723 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:37:31.723 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:37:32.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:37:32.221 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:37:32.221 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:37:32.222 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:37:32.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:37:32.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:37:32.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:37:32.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:37:32.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:37:32.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:37:32.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:37:32.771 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:37:32.771 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:37:33.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-18T15:37:33.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-18T15:37:33.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:37:33.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:37:33.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:37:33.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:37:33.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:37:33.452 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:37:33.452 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:37:33.452 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:37:33.453 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:37:33.453 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:37:33.453 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:37:33.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:37:33.863 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:37:33.864 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:37:33.864 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:37:34.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:37:34.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:37:34.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:37:34.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:37:34.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:37:34.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:37:34.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:37:34.458 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:37:34.458 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:37:35.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-18T15:37:35.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-18T15:37:35.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:37:35.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:37:35.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:37:35.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:37:35.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:37:35.230 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:37:35.230 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:37:35.230 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:37:35.230 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:37:35.230 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:37:35.230 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:37:35.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:37:35.652 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:37:35.653 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:37:35.653 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:37:36.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:37:36.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:37:36.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:37:36.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:37:36.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:37:36.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:37:36.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:37:36.237 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:37:36.237 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:37:36.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-18T15:37:36.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-18T15:37:36.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:37:36.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:37:36.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:37:36.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:37:36.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:37:36.915 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:37:36.915 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:37:36.915 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:37:36.915 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:37:36.915 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:37:36.915 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:37:37.477 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:37:37.477 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:37:37.478 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:37:37.478 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:37:38.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:37:38.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:37:38.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:37:38.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:37:38.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:37:38.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:37:38.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:37:38.038 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:37:38.039 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:37:38.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-18T15:37:38.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-18T15:37:38.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:37:38.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:37:38.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:37:38.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:37:38.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:37:38.751 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:37:38.751 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:37:38.752 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:37:38.752 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:37:38.752 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:37:38.752 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:37:39.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:37:39.201 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:37:39.201 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:37:39.201 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:37:39.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:37:39.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:37:39.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:37:39.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:37:39.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:37:39.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:37:39.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:37:39.759 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:37:39.759 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:37:40.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-18T15:37:40.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-18T15:37:40.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:37:40.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:37:40.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:37:40.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:37:40.514 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:37:40.514 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:37:40.514 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:37:40.514 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:37:40.514 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:37:40.514 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:37:40.514 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:37:40.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:37:40.953 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:37:40.953 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:37:40.953 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:37:41.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:37:41.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:37:41.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:37:41.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:37:41.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:37:41.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:37:41.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:37:41.506 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:37:41.506 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:37:42.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-18T15:37:42.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-18T15:37:42.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:37:42.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:37:42.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:37:42.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:37:42.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:37:42.225 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:37:42.225 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:37:42.225 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:37:42.225 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:37:42.225 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:37:42.225 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:37:42.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:37:42.584 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:37:42.584 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:37:42.584 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:37:43.096 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:37:43.096 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:37:43.096 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:37:43.096 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:37:43.096 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:37:43.096 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:37:43.096 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:37:43.096 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:37:43.097 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:37:43.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-18T15:37:43.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-18T15:37:43.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:37:43.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:37:43.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:37:43.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:37:43.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:37:43.818 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:37:43.818 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:37:43.818 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:37:43.818 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:37:43.818 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:37:43.818 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:37:44.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:37:44.274 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:37:44.274 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:37:44.274 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:37:44.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:37:44.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:37:44.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:37:44.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:37:44.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:37:44.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:37:44.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:37:44.888 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:37:44.888 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:37:45.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-18T15:37:45.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-18T15:37:45.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:37:45.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:37:45.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:37:45.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:37:45.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:37:45.506 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:37:45.506 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:37:45.506 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:37:45.506 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:37:45.506 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:37:45.506 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:37:45.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:37:45.887 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:37:45.887 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:37:45.887 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:37:46.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:37:46.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:37:46.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:37:46.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:37:46.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:37:46.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:37:46.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:37:46.526 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:37:46.526 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:37:47.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-18T15:37:47.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-18T15:37:47.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:37:47.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:37:47.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:37:47.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:37:47.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:37:47.217 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:37:47.217 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:37:47.217 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:37:47.217 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:37:47.217 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:37:47.217 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:37:47.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:37:47.572 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:37:47.572 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:37:47.572 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:37:48.135 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:37:48.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:37:48.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:37:48.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:37:48.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:37:48.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:37:48.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:37:48.136 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:37:48.136 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:37:48.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-18T15:37:48.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-18T15:37:48.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:37:48.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:37:48.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:37:48.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:37:48.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:37:48.739 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:37:48.739 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:37:48.739 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:37:48.739 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:37:48.739 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:37:48.739 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:37:49.096 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:37:49.097 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:37:49.097 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:37:49.097 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:37:49.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:37:49.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:37:49.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:37:49.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:37:49.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:37:49.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:37:49.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:37:49.632 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:37:49.632 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:37:50.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-09-18T15:37:50.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-09-18T15:37:50.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:37:50.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:37:50.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:37:50.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:37:50.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:37:50.277 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:37:50.277 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:37:50.277 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:37:50.277 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:37:50.277 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:37:50.277 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:37:50.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T15:37:50.631 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:37:50.631 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:37:50.631 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:37:51.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T15:37:51.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T15:37:51.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:37:51.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:37:51.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:37:51.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:37:51.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:37:51.179 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:37:51.180 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:37:51.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-09-18T15:37:51.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-09-18T15:37:51.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T15:37:51.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:37:51.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:37:51.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:37:51.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:37:51.780 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:37:51.780 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:37:51.780 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:37:51.780 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:37:51.780 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:37:51.781 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:37:52.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:37:52.137 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:37:52.137 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:37:52.137 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:37:52.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:37:52.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:37:52.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:37:52.628 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:224: TEST_recovery_sizeup: expr 200 '*' 2 2024-09-18T15:37:52.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:224: TEST_recovery_sizeup: local degraded=400 2024-09-18T15:37:52.629 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:225: TEST_recovery_sizeup: expr 200 '*' 2 2024-09-18T15:37:52.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:225: TEST_recovery_sizeup: local misplaced=400 2024-09-18T15:37:52.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:226: TEST_recovery_sizeup: local log=td/osd-recovery-stats/osd.2.log 2024-09-18T15:37:52.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:227: TEST_recovery_sizeup: check td/osd-recovery-stats 1.0 2 replicated 400 0 400 0 200 0 2024-09-18T15:37:52.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-09-18T15:37:52.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-09-18T15:37:52.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=2 2024-09-18T15:37:52.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2024-09-18T15:37:52.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=400 2024-09-18T15:37:52.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-09-18T15:37:52.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=400 2024-09-18T15:37:52.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-09-18T15:37:52.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start=200 2024-09-18T15:37:52.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end=0 2024-09-18T15:37:52.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.2.log 2024-09-18T15:37:52.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-09-18T15:37:52.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2024-09-18T15:37:52.633 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-09-18T15:37:52.633 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.2.log 2024-09-18T15:37:52.633 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-09-18T15:37:52.633 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-09-18T15:37:52.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[2,4,3,5]/[2,3]' 2024-09-18T15:37:52.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-09-18T15:37:52.659 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-18T15:37:52.659 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.2.log 2024-09-18T15:37:52.659 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [2,4,3,5]/[2,3] ' 2024-09-18T15:37:52.659 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-09-18T15:37:52.659 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-09-18T15:37:52.660 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-18T15:37:52.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=400 2024-09-18T15:37:52.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 400 400 2024-09-18T15:37:52.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=400 2024-09-18T15:37:52.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-18T15:37:52.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=400 2024-09-18T15:37:52.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-18T15:37:52.663 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-18T15:37:52.663 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.2.log 2024-09-18T15:37:52.664 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-09-18T15:37:52.664 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-18T15:37:52.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-09-18T15:37:52.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-09-18T15:37:52.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-18T15:37:52.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-18T15:37:52.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-18T15:37:52.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-18T15:37:52.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-09-18T15:37:52.674 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-18T15:37:52.674 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.2.log 2024-09-18T15:37:52.674 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [2,4,3,5]/[2,3] ' 2024-09-18T15:37:52.674 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-09-18T15:37:52.675 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-09-18T15:37:52.675 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-18T15:37:52.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=400 2024-09-18T15:37:52.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 400 400 2024-09-18T15:37:52.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=400 2024-09-18T15:37:52.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-18T15:37:52.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=400 2024-09-18T15:37:52.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-18T15:37:52.678 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-18T15:37:52.678 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.2.log 2024-09-18T15:37:52.678 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-09-18T15:37:52.678 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-18T15:37:52.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-09-18T15:37:52.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-09-18T15:37:52.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-18T15:37:52.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-18T15:37:52.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-18T15:37:52.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-18T15:37:52.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n 200 ']' 2024-09-18T15:37:52.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:102: check: which='shard 2' 2024-09-18T15:37:52.689 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-18T15:37:52.689 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 2 ' td/osd-recovery-stats/osd.2.log 2024-09-18T15:37:52.689 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -F ' [2,4,3,5]/[2,3] ' 2024-09-18T15:37:52.689 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -v est 2024-09-18T15:37:52.689 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: head -1 2024-09-18T15:37:52.690 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-18T15:37:52.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: FIRST=200 2024-09-18T15:37:52.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:104: check: below_margin 200 200 2024-09-18T15:37:52.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-09-18T15:37:52.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-18T15:37:52.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-09-18T15:37:52.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-18T15:37:52.693 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-18T15:37:52.693 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 2 ' td/osd-recovery-stats/osd.2.log 2024-09-18T15:37:52.693 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: tail -1 2024-09-18T15:37:52.693 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-18T15:37:52.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: LAST=0 2024-09-18T15:37:52.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:106: check: above_margin 0 0 2024-09-18T15:37:52.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-18T15:37:52.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-18T15:37:52.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-18T15:37:52.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-18T15:37:52.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:229: TEST_recovery_sizeup: delete_pool test 2024-09-18T15:37:52.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-18T15:37:52.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-18T15:37:53.339 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test' does not exist 2024-09-18T15:37:53.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:230: TEST_recovery_sizeup: kill_daemons td/osd-recovery-stats 2024-09-18T15:37:53.350 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:37:53.350 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:37:53.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:37:53.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:37:53.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:37:58.666 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:37:58.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-09-18T15:37:58.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-18T15:37:58.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:37:58.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-18T15:37:58.667 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:37:58.667 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:37:58.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:37:58.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:37:58.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:37:58.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:37:58.676 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:37:58.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:37:58.677 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:37:58.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:37:58.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:37:58.679 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:37:58.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:37:58.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:37:58.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:37:58.680 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:37:58.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:37:58.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:37:58.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-18T15:37:58.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:37:58.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:37:58.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:37:58.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.288083 2024-09-18T15:37:58.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:37:58.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:37:58.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-09-18T15:37:58.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-09-18T15:37:58.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-09-18T15:37:58.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-09-18T15:37:58.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-18T15:37:58.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:37:58.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-18T15:37:58.727 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:37:58.727 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:37:58.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:37:58.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:37:58.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:37:58.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:37:58.729 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:37:58.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:37:58.730 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:37:58.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:37:58.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:37:58.731 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:37:58.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:37:58.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:37:58.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:37:58.733 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:37:58.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:37:58.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:37:58.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-18T15:37:58.736 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:37:58.736 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:37:58.736 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:37:58.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.288083 2024-09-18T15:37:58.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:37:58.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:37:58.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-09-18T15:37:58.739 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T15:37:58.739 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:37:58.739 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:37:58.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.288083 2024-09-18T15:37:58.741 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T15:37:58.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T15:37:58.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T15:37:58.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-09-18T15:37:58.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_undersized td/osd-recovery-stats 2024-09-18T15:37:58.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:306: TEST_recovery_undersized: local dir=td/osd-recovery-stats 2024-09-18T15:37:58.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:308: TEST_recovery_undersized: local osds=3 2024-09-18T15:37:58.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:309: TEST_recovery_undersized: run_mon td/osd-recovery-stats a 2024-09-18T15:37:58.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-09-18T15:37:58.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T15:37:58.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T15:37:58.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T15:37:58.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-09-18T15:37:58.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-09-18T15:37:59.106 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T15:37:59.107 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:37:59.107 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:37:59.107 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:37:59.107 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:37:59.107 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:37:59.107 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:37:59.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T15:37:59.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T15:37:59.350 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T15:37:59.350 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:37:59.350 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:37:59.350 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T15:37:59.351 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T15:37:59.351 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:37:59.352 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:37:59.352 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:37:59.352 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:37:59.352 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:37:59.352 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:37:59.352 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.288083/ceph-mon.a.asok 2024-09-18T15:37:59.352 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:37:59.352 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.288083/ceph-mon.a.asok config get fsid 2024-09-18T15:37:59.439 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T15:37:59.439 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:37:59.440 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:37:59.440 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T15:37:59.440 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T15:37:59.440 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:37:59.440 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:37:59.440 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:37:59.441 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:37:59.441 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:37:59.441 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:37:59.441 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.288083/ceph-mon.a.asok 2024-09-18T15:37:59.441 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:37:59.441 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.288083/ceph-mon.a.asok config get mon_host 2024-09-18T15:37:59.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:310: TEST_recovery_undersized: run_mgr td/osd-recovery-stats x 2024-09-18T15:37:59.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-09-18T15:37:59.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T15:37:59.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T15:37:59.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T15:37:59.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-09-18T15:37:59.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T15:38:00.363 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T15:38:00.363 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:38:00.363 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:38:00.363 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:38:00.363 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:38:00.363 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:38:00.363 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:38:00.364 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:38:00.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:38:00.389 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: expr 3 - 1 2024-09-18T15:38:00.390 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: seq 0 2 2024-09-18T15:38:00.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: for i in $(seq 0 $(expr $osds - 1)) 2024-09-18T15:38:00.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:313: TEST_recovery_undersized: run_osd td/osd-recovery-stats 0 2024-09-18T15:38:00.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:38:00.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:38:00.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T15:38:00.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:38:00.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-09-18T15:38:00.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:38:00.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:38:00.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:38:00.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:38:00.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-09-18T15:38:00.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-09-18T15:38:00.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:38:00.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:38:00.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:38:00.396 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:38:00.396 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:38:00.396 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:38:00.396 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:38:00.396 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:38:00.396 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:38:00.396 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:38:00.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:38:00.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:38:00.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:38:00.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:38:00.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:38:00.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:38:00.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:38:00.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:38:00.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:38:00.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:38:00.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:38:00.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:38:00.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-09-18T15:38:00.398 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:38:00.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c5e5e350-c820-4642-9fe9-d1a53aa8de7d 2024-09-18T15:38:00.398 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 c5e5e350-c820-4642-9fe9-d1a53aa8de7d 2024-09-18T15:38:00.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c5e5e350-c820-4642-9fe9-d1a53aa8de7d' 2024-09-18T15:38:00.398 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:38:00.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDY8+pm9T9cGBAAX1JzOZS+awZVPhiR6zlPYQ== 2024-09-18T15:38:00.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDY8+pm9T9cGBAAX1JzOZS+awZVPhiR6zlPYQ=="}' 2024-09-18T15:38:00.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c5e5e350-c820-4642-9fe9-d1a53aa8de7d -i td/osd-recovery-stats/0/new.json 2024-09-18T15:38:00.671 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:38:00.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-09-18T15:38:00.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDY8+pm9T9cGBAAX1JzOZS+awZVPhiR6zlPYQ== --osd-uuid c5e5e350-c820-4642-9fe9-d1a53aa8de7d 2024-09-18T15:38:00.706 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:38:00.705+0000 7fd47788b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:38:00.716 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:38:00.715+0000 7fd47788b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:38:00.717 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:38:00.716+0000 7fd47788b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:38:00.717 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:38:00.716+0000 7fd47788b440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-18T15:38:00.717 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:38:00.716+0000 7fd47788b440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-18T15:38:00.717 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:38:00.716+0000 7fd47788b440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-18T15:38:00.756 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:38:00.755+0000 7fd47788b440 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-09-18T15:38:06.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-09-18T15:38:06.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:38:06.066 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T15:38:06.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T15:38:06.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:38:06.576 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T15:38:06.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T15:38:06.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:38:06.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:38:06.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:38:06.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:38:06.599 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:38:06.598+0000 7fd235b4c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:38:06.601 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:38:06.600+0000 7fd235b4c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:38:06.602 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:38:06.601+0000 7fd235b4c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:38:06.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T15:38:06.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:38:06.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:38:06.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:38:06.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:38:06.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:06.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:38:06.940 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:38:06.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:06.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:38:07.160 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:38:07.159+0000 7fd235b4c440 -1 Falling back to public interface 2024-09-18T15:38:07.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:08.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:08.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:08.307 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:38:08.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:38:08.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:08.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:38:08.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:09.633 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:38:09.632+0000 7fd235b4c440 -1 osd.0 0 log_to_monitors true 2024-09-18T15:38:09.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:09.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:09.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:38:09.712 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:38:09.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:09.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:38:10.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:11.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:11.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:11.193 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:38:11.194 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:38:11.194 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:11.194 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:38:11.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:12.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:12.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:12.597 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:38:12.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:38:12.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:12.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:38:12.992 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:13.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:14.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:14.001 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:38:14.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:38:14.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:14.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:38:14.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:15.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:15.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:15.467 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:38:15.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:38:15.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:15.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:38:15.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:16.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:16.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:16.856 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:38:16.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:38:16.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:16.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:38:17.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:18.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:18.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:18.352 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:38:18.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:38:18.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:18.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:38:18.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:19.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:19.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:19.765 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:38:19.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:38:19.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:19.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:38:20.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:21.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:21.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:21.220 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:38:21.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:38:21.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:21.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:38:22.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:23.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:23.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:23.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:38:23.130 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:38:23.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:23.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:38:24.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:25.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:25.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:25.050 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:38:25.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:38:25.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:25.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:38:25.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:26.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:26.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:26.911 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:38:26.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:38:26.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:26.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:38:28.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:29.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:29.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:29.048 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:38:29.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:38:29.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:29.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:38:29.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:30.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:30.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:30.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:38:30.909 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:38:30.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:30.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:38:31.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:32.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:32.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:32.490 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:38:32.491 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:38:32.491 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:32.491 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:38:32.867 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/388622,v1:127.0.0.1:6803/388622] [v2:127.0.0.1:6804/388622,v1:127.0.0.1:6805/388622] exists,up c5e5e350-c820-4642-9fe9-d1a53aa8de7d 2024-09-18T15:38:32.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:38:32.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:38:32.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:38:32.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: for i in $(seq 0 $(expr $osds - 1)) 2024-09-18T15:38:32.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:313: TEST_recovery_undersized: run_osd td/osd-recovery-stats 1 2024-09-18T15:38:32.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:38:32.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:38:32.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T15:38:32.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:38:32.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-09-18T15:38:32.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:38:32.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:38:32.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:38:32.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:38:32.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-09-18T15:38:32.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-09-18T15:38:32.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:38:32.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:38:32.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:38:32.872 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:38:32.872 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:38:32.872 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:38:32.872 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:38:32.872 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:38:32.873 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:38:32.873 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:38:32.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:38:32.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:38:32.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:38:32.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:38:32.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:38:32.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:38:32.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:38:32.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:38:32.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:38:32.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:38:32.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:38:32.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:38:32.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-09-18T15:38:32.874 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:38:32.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=783dafed-f4c5-4e3f-bae3-0f9ce36f8448 2024-09-18T15:38:32.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 783dafed-f4c5-4e3f-bae3-0f9ce36f8448' 2024-09-18T15:38:32.874 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 783dafed-f4c5-4e3f-bae3-0f9ce36f8448 2024-09-18T15:38:32.874 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:38:32.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD48+pmCpy7NBAAQB8XK73kxsB5lPWt2tfFbg== 2024-09-18T15:38:32.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD48+pmCpy7NBAAQB8XK73kxsB5lPWt2tfFbg=="}' 2024-09-18T15:38:32.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 783dafed-f4c5-4e3f-bae3-0f9ce36f8448 -i td/osd-recovery-stats/1/new.json 2024-09-18T15:38:33.339 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:38:33.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-09-18T15:38:33.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD48+pmCpy7NBAAQB8XK73kxsB5lPWt2tfFbg== --osd-uuid 783dafed-f4c5-4e3f-bae3-0f9ce36f8448 2024-09-18T15:38:33.376 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:38:33.375+0000 7fb4ece16440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:38:33.378 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:38:33.377+0000 7fb4ece16440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:38:33.379 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:38:33.378+0000 7fb4ece16440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:38:33.379 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:38:33.378+0000 7fb4ece16440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-18T15:38:33.379 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:38:33.378+0000 7fb4ece16440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-18T15:38:33.379 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:38:33.378+0000 7fb4ece16440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-18T15:38:33.438 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:38:33.437+0000 7fb4ece16440 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-09-18T15:38:39.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-09-18T15:38:39.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:38:39.060 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T15:38:39.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T15:38:39.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:38:39.570 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T15:38:39.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T15:38:39.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:38:39.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:38:39.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:38:39.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:38:39.593 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:38:39.592+0000 7fa82231c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:38:39.619 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:38:39.618+0000 7fa82231c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:38:39.621 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:38:39.620+0000 7fa82231c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:38:39.988 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T15:38:39.988 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:38:39.988 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:38:39.988 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:38:39.989 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:38:39.989 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:39.989 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:38:39.989 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:38:39.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:39.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:38:40.150 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:38:40.149+0000 7fa82231c440 -1 Falling back to public interface 2024-09-18T15:38:40.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:41.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:41.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:41.351 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:38:41.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:38:41.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:41.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:38:41.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:42.709 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:38:42.708+0000 7fa82231c440 -1 osd.1 0 log_to_monitors true 2024-09-18T15:38:42.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:42.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:42.731 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:38:42.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:38:42.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:42.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:38:43.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:44.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:44.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:44.278 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:38:44.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:38:44.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:44.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:38:44.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:45.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:45.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:45.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:38:45.787 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:38:45.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:45.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:38:46.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:47.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:47.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:47.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:38:47.411 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:38:47.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:47.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:38:47.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:48.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:48.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:48.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:38:48.820 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:38:48.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:48.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:38:49.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:50.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:50.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:50.378 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:38:50.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:38:50.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:50.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:38:50.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:51.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:51.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:51.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:38:51.875 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:38:51.875 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:51.875 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:38:52.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:53.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:53.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:53.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:38:53.365 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:38:53.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:53.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:38:53.740 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:54.741 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:54.741 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:54.742 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:38:54.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:38:54.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:54.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:38:55.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:56.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:56.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:56.591 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:38:56.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:38:56.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:56.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:38:57.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:38:58.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:38:58.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:38:58.713 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:38:58.714 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:38:58.714 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:38:58.714 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:38:59.477 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:00.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:00.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:00.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:39:00.479 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:39:00.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:00.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:39:01.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:02.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:02.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:02.192 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:39:02.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:39:02.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:02.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:39:03.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:04.172 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:04.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:04.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:39:04.173 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:39:04.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:04.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:39:04.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:05.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:05.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:05.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:39:05.765 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:39:05.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:05.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:39:06.208 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/390351,v1:127.0.0.1:6811/390351] [v2:127.0.0.1:6812/390351,v1:127.0.0.1:6813/390351] exists,up 783dafed-f4c5-4e3f-bae3-0f9ce36f8448 2024-09-18T15:39:06.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:39:06.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:39:06.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:39:06.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: for i in $(seq 0 $(expr $osds - 1)) 2024-09-18T15:39:06.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:313: TEST_recovery_undersized: run_osd td/osd-recovery-stats 2 2024-09-18T15:39:06.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-18T15:39:06.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:39:06.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T15:39:06.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:39:06.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-09-18T15:39:06.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T15:39:06.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:39:06.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:39:06.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:39:06.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-09-18T15:39:06.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-09-18T15:39:06.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:39:06.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:39:06.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-18T15:39:06.215 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:39:06.215 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:39:06.215 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:39:06.215 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:39:06.215 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:39:06.215 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:39:06.215 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:39:06.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' 2024-09-18T15:39:06.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:39:06.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:39:06.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:39:06.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-18T15:39:06.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-18T15:39:06.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:39:06.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:39:06.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:39:06.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:39:06.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:39:06.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:39:06.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-09-18T15:39:06.216 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:39:06.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=75dda621-d257-483c-9261-ac8116e5b0d2 2024-09-18T15:39:06.217 INFO:tasks.workunit.client.0.smithi047.stdout:add osd2 75dda621-d257-483c-9261-ac8116e5b0d2 2024-09-18T15:39:06.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 75dda621-d257-483c-9261-ac8116e5b0d2' 2024-09-18T15:39:06.218 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:39:06.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAa9OpmQiuHDRAAnnzfVOsZzNdmvsJqBGdIKA== 2024-09-18T15:39:06.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAa9OpmQiuHDRAAnnzfVOsZzNdmvsJqBGdIKA=="}' 2024-09-18T15:39:06.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 75dda621-d257-483c-9261-ac8116e5b0d2 -i td/osd-recovery-stats/2/new.json 2024-09-18T15:39:06.777 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:39:06.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-09-18T15:39:06.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAa9OpmQiuHDRAAnnzfVOsZzNdmvsJqBGdIKA== --osd-uuid 75dda621-d257-483c-9261-ac8116e5b0d2 2024-09-18T15:39:06.813 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:39:06.812+0000 7f639964f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:39:06.818 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:39:06.817+0000 7f639964f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:39:06.819 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:39:06.818+0000 7f639964f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:39:06.819 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:39:06.818+0000 7f639964f440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-18T15:39:06.819 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:39:06.818+0000 7f639964f440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-18T15:39:06.819 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:39:06.818+0000 7f639964f440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-18T15:39:06.884 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:39:06.884+0000 7f639964f440 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-09-18T15:39:12.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-09-18T15:39:12.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:39:12.410 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd2 key to auth repository 2024-09-18T15:39:12.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T15:39:12.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:39:12.924 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T15:39:12.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T15:39:12.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=8ea6e60c-4545-40ce-bd42-d48f1a889885 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.288083/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:39:12.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:39:12.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:39:12.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:39:12.948 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:39:12.948+0000 7f1469b24440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:39:12.973 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:39:12.974+0000 7f1469b24440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:39:12.974 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:39:12.975+0000 7f1469b24440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:39:13.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T15:39:13.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:39:13.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:39:13.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:39:13.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:39:13.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:13.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:39:13.363 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:39:13.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:13.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:39:13.506 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:39:13.505+0000 7f1469b24440 -1 Falling back to public interface 2024-09-18T15:39:13.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:14.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:14.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:14.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:39:14.747 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:39:14.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:14.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:39:15.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:15.898 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:39:15.897+0000 7f1469b24440 -1 osd.2 0 log_to_monitors true 2024-09-18T15:39:16.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:16.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:16.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:39:16.169 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:39:16.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:16.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:39:16.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:17.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:17.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:17.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:39:17.708 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:39:17.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:17.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:39:18.096 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:19.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:19.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:19.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:39:19.098 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:39:19.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:19.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:39:19.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:20.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:20.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:20.473 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:39:20.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:39:20.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:20.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:39:21.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:22.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:22.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:22.003 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:39:22.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:39:22.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:22.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:39:22.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:23.469 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:23.469 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:23.469 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:39:23.469 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:39:23.470 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:23.470 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:39:23.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:24.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:24.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:24.889 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:39:24.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:39:24.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:24.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:39:25.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:26.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:26.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:26.269 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:39:26.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:39:26.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:26.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:39:26.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:27.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:27.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:27.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:39:27.679 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:39:27.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:27.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:39:28.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:29.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:29.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:29.340 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:39:29.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:39:29.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:29.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:39:30.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:31.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:31.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:31.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:39:31.013 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:39:31.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:31.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:39:31.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:32.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:32.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:32.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:39:32.616 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:39:32.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:32.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:39:33.390 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:34.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:34.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:34.392 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:39:34.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:39:34.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:34.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:39:35.212 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:36.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:36.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:36.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:39:36.214 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:39:36.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:36.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:39:36.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:39:37.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:39:37.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:39:37.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:39:37.966 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:39:37.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:39:37.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:39:38.355 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/392081,v1:127.0.0.1:6819/392081] [v2:127.0.0.1:6820/392081,v1:127.0.0.1:6821/392081] exists,up 75dda621-d257-483c-9261-ac8116e5b0d2 2024-09-18T15:39:38.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:39:38.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:39:38.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:39:38.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:316: TEST_recovery_undersized: create_pool test 1 1 2024-09-18T15:39:38.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-18T15:39:39.101 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test' already exists 2024-09-18T15:39:39.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:39:40.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:317: TEST_recovery_undersized: ceph osd pool set test size 1 --yes-i-really-mean-it 2024-09-18T15:39:41.120 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 1 size to 1 2024-09-18T15:39:41.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:319: TEST_recovery_undersized: wait_for_clean 2024-09-18T15:39:41.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:39:41.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:39:41.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:39:41.134 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:39:41.134 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:39:41.134 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:39:41.134 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:39:41.134 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:39:41.135 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:39:41.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:39:41.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:39:41.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:39:41.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:39:41.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:39:41.229 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:39:41.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:39:41.593 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:39:41.593 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T15:39:41.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:39:41.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:39:41.593 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:39:41.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836503 2024-09-18T15:39:41.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836503 2024-09-18T15:39:41.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836503' 2024-09-18T15:39:41.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:39:41.701 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:39:41.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672975 2024-09-18T15:39:41.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672975 2024-09-18T15:39:41.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836503 1-42949672975' 2024-09-18T15:39:41.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:39:41.809 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:39:41.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509447 2024-09-18T15:39:41.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509447 2024-09-18T15:39:41.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836503 1-42949672975 2-64424509447' 2024-09-18T15:39:41.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:39:41.917 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836503 2024-09-18T15:39:41.917 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:39:41.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:39:41.918 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836503 2024-09-18T15:39:41.918 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:39:41.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836503 2024-09-18T15:39:41.919 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836503 2024-09-18T15:39:41.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-09-18T15:39:41.920 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:39:42.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836504 -lt 21474836503 2024-09-18T15:39:42.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:39:42.285 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672975 2024-09-18T15:39:42.285 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:39:42.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:39:42.286 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672975 2024-09-18T15:39:42.286 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:39:42.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672975 2024-09-18T15:39:42.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-09-18T15:39:42.288 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672975 2024-09-18T15:39:42.288 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:39:42.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672975 -lt 42949672975 2024-09-18T15:39:42.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:39:42.655 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509447 2024-09-18T15:39:42.655 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:39:42.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:39:42.656 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509447 2024-09-18T15:39:42.656 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:39:42.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509447 2024-09-18T15:39:42.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-09-18T15:39:42.657 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509447 2024-09-18T15:39:42.658 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:39:43.022 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509447 -lt 64424509447 2024-09-18T15:39:43.022 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:39:43.023 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:39:43.023 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:39:43.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:39:43.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:39:43.523 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:39:43.523 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:39:43.523 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:39:43.523 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:39:43.523 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:39:43.523 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:39:43.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:39:43.920 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:39:43.920 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:39:43.920 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:39:44.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:39:44.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:39:44.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:39:44.415 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: seq 1 200 2024-09-18T15:39:44.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:44.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj1 /dev/null 2024-09-18T15:39:44.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:44.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj2 /dev/null 2024-09-18T15:39:44.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:44.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj3 /dev/null 2024-09-18T15:39:44.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:44.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj4 /dev/null 2024-09-18T15:39:44.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:44.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj5 /dev/null 2024-09-18T15:39:44.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:44.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj6 /dev/null 2024-09-18T15:39:44.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:44.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj7 /dev/null 2024-09-18T15:39:44.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:44.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj8 /dev/null 2024-09-18T15:39:44.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:44.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj9 /dev/null 2024-09-18T15:39:45.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:45.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj10 /dev/null 2024-09-18T15:39:45.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:45.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj11 /dev/null 2024-09-18T15:39:45.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:45.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj12 /dev/null 2024-09-18T15:39:45.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:45.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj13 /dev/null 2024-09-18T15:39:45.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:45.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj14 /dev/null 2024-09-18T15:39:45.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:45.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj15 /dev/null 2024-09-18T15:39:45.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:45.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj16 /dev/null 2024-09-18T15:39:45.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:45.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj17 /dev/null 2024-09-18T15:39:45.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:45.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj18 /dev/null 2024-09-18T15:39:45.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:45.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj19 /dev/null 2024-09-18T15:39:45.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:45.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj20 /dev/null 2024-09-18T15:39:45.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:45.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj21 /dev/null 2024-09-18T15:39:45.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:45.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj22 /dev/null 2024-09-18T15:39:45.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:45.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj23 /dev/null 2024-09-18T15:39:45.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:45.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj24 /dev/null 2024-09-18T15:39:45.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:45.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj25 /dev/null 2024-09-18T15:39:46.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:46.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj26 /dev/null 2024-09-18T15:39:46.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:46.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj27 /dev/null 2024-09-18T15:39:46.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:46.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj28 /dev/null 2024-09-18T15:39:46.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:46.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj29 /dev/null 2024-09-18T15:39:46.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:46.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj30 /dev/null 2024-09-18T15:39:46.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:46.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj31 /dev/null 2024-09-18T15:39:46.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:46.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj32 /dev/null 2024-09-18T15:39:46.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:46.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj33 /dev/null 2024-09-18T15:39:46.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:46.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj34 /dev/null 2024-09-18T15:39:46.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:46.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj35 /dev/null 2024-09-18T15:39:46.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:46.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj36 /dev/null 2024-09-18T15:39:46.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:46.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj37 /dev/null 2024-09-18T15:39:46.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:46.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj38 /dev/null 2024-09-18T15:39:46.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:46.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj39 /dev/null 2024-09-18T15:39:46.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:46.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj40 /dev/null 2024-09-18T15:39:47.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:47.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj41 /dev/null 2024-09-18T15:39:47.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:47.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj42 /dev/null 2024-09-18T15:39:47.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:47.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj43 /dev/null 2024-09-18T15:39:47.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:47.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj44 /dev/null 2024-09-18T15:39:47.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:47.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj45 /dev/null 2024-09-18T15:39:47.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:47.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj46 /dev/null 2024-09-18T15:39:47.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:47.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj47 /dev/null 2024-09-18T15:39:47.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:47.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj48 /dev/null 2024-09-18T15:39:47.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:47.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj49 /dev/null 2024-09-18T15:39:47.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:47.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj50 /dev/null 2024-09-18T15:39:47.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:47.660 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj51 /dev/null 2024-09-18T15:39:47.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:47.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj52 /dev/null 2024-09-18T15:39:47.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:47.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj53 /dev/null 2024-09-18T15:39:47.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:47.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj54 /dev/null 2024-09-18T15:39:47.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:47.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj55 /dev/null 2024-09-18T15:39:47.985 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:47.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj56 /dev/null 2024-09-18T15:39:48.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:48.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj57 /dev/null 2024-09-18T15:39:48.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:48.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj58 /dev/null 2024-09-18T15:39:48.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:48.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj59 /dev/null 2024-09-18T15:39:48.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:48.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj60 /dev/null 2024-09-18T15:39:48.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:48.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj61 /dev/null 2024-09-18T15:39:48.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:48.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj62 /dev/null 2024-09-18T15:39:48.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:48.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj63 /dev/null 2024-09-18T15:39:48.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:48.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj64 /dev/null 2024-09-18T15:39:48.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:48.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj65 /dev/null 2024-09-18T15:39:48.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:48.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj66 /dev/null 2024-09-18T15:39:48.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:48.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj67 /dev/null 2024-09-18T15:39:48.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:48.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj68 /dev/null 2024-09-18T15:39:48.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:48.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj69 /dev/null 2024-09-18T15:39:48.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:48.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj70 /dev/null 2024-09-18T15:39:48.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:48.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj71 /dev/null 2024-09-18T15:39:49.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:49.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj72 /dev/null 2024-09-18T15:39:49.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:49.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj73 /dev/null 2024-09-18T15:39:49.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:49.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj74 /dev/null 2024-09-18T15:39:49.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:49.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj75 /dev/null 2024-09-18T15:39:49.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:49.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj76 /dev/null 2024-09-18T15:39:49.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:49.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj77 /dev/null 2024-09-18T15:39:49.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:49.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj78 /dev/null 2024-09-18T15:39:49.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:49.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj79 /dev/null 2024-09-18T15:39:49.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:49.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj80 /dev/null 2024-09-18T15:39:49.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:49.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj81 /dev/null 2024-09-18T15:39:49.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:49.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj82 /dev/null 2024-09-18T15:39:49.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:49.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj83 /dev/null 2024-09-18T15:39:49.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:49.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj84 /dev/null 2024-09-18T15:39:49.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:49.826 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj85 /dev/null 2024-09-18T15:39:49.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:49.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj86 /dev/null 2024-09-18T15:39:49.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:49.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj87 /dev/null 2024-09-18T15:39:50.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:50.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj88 /dev/null 2024-09-18T15:39:50.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:50.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj89 /dev/null 2024-09-18T15:39:50.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:50.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj90 /dev/null 2024-09-18T15:39:50.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:50.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj91 /dev/null 2024-09-18T15:39:50.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:50.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj92 /dev/null 2024-09-18T15:39:50.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:50.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj93 /dev/null 2024-09-18T15:39:50.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:50.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj94 /dev/null 2024-09-18T15:39:50.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:50.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj95 /dev/null 2024-09-18T15:39:50.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:50.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj96 /dev/null 2024-09-18T15:39:50.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:50.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj97 /dev/null 2024-09-18T15:39:50.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:50.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj98 /dev/null 2024-09-18T15:39:50.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:50.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj99 /dev/null 2024-09-18T15:39:50.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:50.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj100 /dev/null 2024-09-18T15:39:50.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:50.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj101 /dev/null 2024-09-18T15:39:51.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:51.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj102 /dev/null 2024-09-18T15:39:51.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:51.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj103 /dev/null 2024-09-18T15:39:51.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:51.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj104 /dev/null 2024-09-18T15:39:51.197 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:51.197 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj105 /dev/null 2024-09-18T15:39:51.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:51.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj106 /dev/null 2024-09-18T15:39:51.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:51.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj107 /dev/null 2024-09-18T15:39:51.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:51.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj108 /dev/null 2024-09-18T15:39:51.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:51.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj109 /dev/null 2024-09-18T15:39:51.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:51.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj110 /dev/null 2024-09-18T15:39:51.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:51.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj111 /dev/null 2024-09-18T15:39:51.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:51.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj112 /dev/null 2024-09-18T15:39:51.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:51.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj113 /dev/null 2024-09-18T15:39:51.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:51.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj114 /dev/null 2024-09-18T15:39:51.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:51.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj115 /dev/null 2024-09-18T15:39:51.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:51.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj116 /dev/null 2024-09-18T15:39:51.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:51.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj117 /dev/null 2024-09-18T15:39:52.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:52.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj118 /dev/null 2024-09-18T15:39:52.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:52.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj119 /dev/null 2024-09-18T15:39:52.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:52.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj120 /dev/null 2024-09-18T15:39:52.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:52.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj121 /dev/null 2024-09-18T15:39:52.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:52.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj122 /dev/null 2024-09-18T15:39:52.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:52.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj123 /dev/null 2024-09-18T15:39:52.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:52.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj124 /dev/null 2024-09-18T15:39:52.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:52.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj125 /dev/null 2024-09-18T15:39:52.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:52.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj126 /dev/null 2024-09-18T15:39:52.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:52.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj127 /dev/null 2024-09-18T15:39:52.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:52.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj128 /dev/null 2024-09-18T15:39:52.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:52.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj129 /dev/null 2024-09-18T15:39:52.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:52.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj130 /dev/null 2024-09-18T15:39:52.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:52.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj131 /dev/null 2024-09-18T15:39:52.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:52.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj132 /dev/null 2024-09-18T15:39:53.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:53.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj133 /dev/null 2024-09-18T15:39:53.079 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:53.079 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj134 /dev/null 2024-09-18T15:39:53.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:53.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj135 /dev/null 2024-09-18T15:39:53.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:53.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj136 /dev/null 2024-09-18T15:39:53.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:53.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj137 /dev/null 2024-09-18T15:39:53.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:53.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj138 /dev/null 2024-09-18T15:39:53.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:53.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj139 /dev/null 2024-09-18T15:39:53.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:53.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj140 /dev/null 2024-09-18T15:39:53.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:53.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj141 /dev/null 2024-09-18T15:39:53.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:53.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj142 /dev/null 2024-09-18T15:39:53.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:53.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj143 /dev/null 2024-09-18T15:39:53.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:53.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj144 /dev/null 2024-09-18T15:39:53.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:53.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj145 /dev/null 2024-09-18T15:39:53.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:53.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj146 /dev/null 2024-09-18T15:39:53.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:53.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj147 /dev/null 2024-09-18T15:39:53.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:53.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj148 /dev/null 2024-09-18T15:39:53.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:53.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj149 /dev/null 2024-09-18T15:39:53.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:53.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj150 /dev/null 2024-09-18T15:39:54.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:54.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj151 /dev/null 2024-09-18T15:39:54.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:54.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj152 /dev/null 2024-09-18T15:39:54.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:54.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj153 /dev/null 2024-09-18T15:39:54.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:54.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj154 /dev/null 2024-09-18T15:39:54.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:54.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj155 /dev/null 2024-09-18T15:39:54.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:54.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj156 /dev/null 2024-09-18T15:39:54.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:54.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj157 /dev/null 2024-09-18T15:39:54.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:54.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj158 /dev/null 2024-09-18T15:39:54.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:54.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj159 /dev/null 2024-09-18T15:39:54.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:54.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj160 /dev/null 2024-09-18T15:39:54.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:54.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj161 /dev/null 2024-09-18T15:39:54.860 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:54.861 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj162 /dev/null 2024-09-18T15:39:54.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:54.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj163 /dev/null 2024-09-18T15:39:54.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:54.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj164 /dev/null 2024-09-18T15:39:55.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:55.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj165 /dev/null 2024-09-18T15:39:55.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:55.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj166 /dev/null 2024-09-18T15:39:55.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:55.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj167 /dev/null 2024-09-18T15:39:55.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:55.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj168 /dev/null 2024-09-18T15:39:55.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:55.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj169 /dev/null 2024-09-18T15:39:55.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:55.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj170 /dev/null 2024-09-18T15:39:55.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:55.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj171 /dev/null 2024-09-18T15:39:55.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:55.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj172 /dev/null 2024-09-18T15:39:55.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:55.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj173 /dev/null 2024-09-18T15:39:55.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:55.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj174 /dev/null 2024-09-18T15:39:55.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:55.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj175 /dev/null 2024-09-18T15:39:55.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:55.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj176 /dev/null 2024-09-18T15:39:55.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:55.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj177 /dev/null 2024-09-18T15:39:55.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:55.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj178 /dev/null 2024-09-18T15:39:55.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:55.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj179 /dev/null 2024-09-18T15:39:55.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:55.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj180 /dev/null 2024-09-18T15:39:56.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:56.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj181 /dev/null 2024-09-18T15:39:56.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:56.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj182 /dev/null 2024-09-18T15:39:56.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:56.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj183 /dev/null 2024-09-18T15:39:56.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:56.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj184 /dev/null 2024-09-18T15:39:56.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:56.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj185 /dev/null 2024-09-18T15:39:56.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:56.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj186 /dev/null 2024-09-18T15:39:56.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:56.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj187 /dev/null 2024-09-18T15:39:56.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:56.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj188 /dev/null 2024-09-18T15:39:56.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:56.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj189 /dev/null 2024-09-18T15:39:56.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:56.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj190 /dev/null 2024-09-18T15:39:56.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:56.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj191 /dev/null 2024-09-18T15:39:56.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:56.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj192 /dev/null 2024-09-18T15:39:56.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:56.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj193 /dev/null 2024-09-18T15:39:56.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:56.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj194 /dev/null 2024-09-18T15:39:56.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:56.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj195 /dev/null 2024-09-18T15:39:56.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:56.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj196 /dev/null 2024-09-18T15:39:57.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:57.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj197 /dev/null 2024-09-18T15:39:57.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:57.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj198 /dev/null 2024-09-18T15:39:57.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:57.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj199 /dev/null 2024-09-18T15:39:57.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-18T15:39:57.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj200 /dev/null 2024-09-18T15:39:57.278 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:326: TEST_recovery_undersized: get_primary test obj1 2024-09-18T15:39:57.278 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-18T15:39:57.278 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T15:39:57.278 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-18T15:39:57.278 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:39:57.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:326: TEST_recovery_undersized: local primary=1 2024-09-18T15:39:57.655 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:327: TEST_recovery_undersized: get_pg test obj1 2024-09-18T15:39:57.655 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-18T15:39:57.655 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-09-18T15:39:57.655 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-09-18T15:39:57.656 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:39:58.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:327: TEST_recovery_undersized: local PG=1.0 2024-09-18T15:39:58.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:329: TEST_recovery_undersized: ceph osd set norecover 2024-09-18T15:39:58.773 INFO:tasks.workunit.client.0.smithi047.stderr:norecover is set 2024-09-18T15:39:58.786 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:331: TEST_recovery_undersized: expr 3 - 1 2024-09-18T15:39:58.787 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:331: TEST_recovery_undersized: seq 0 2 2024-09-18T15:39:58.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:331: TEST_recovery_undersized: for i in $(seq 0 $(expr $osds - 1)) 2024-09-18T15:39:58.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:333: TEST_recovery_undersized: '[' 0 = 1 ']' 2024-09-18T15:39:58.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:337: TEST_recovery_undersized: ceph osd out osd.0 2024-09-18T15:39:59.335 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already out. 2024-09-18T15:39:59.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:338: TEST_recovery_undersized: break 2024-09-18T15:39:59.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:340: TEST_recovery_undersized: ceph osd pool set test size 4 2024-09-18T15:39:59.961 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 1 size to 4 2024-09-18T15:39:59.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:341: TEST_recovery_undersized: ceph osd unset norecover 2024-09-18T15:40:00.941 INFO:tasks.workunit.client.0.smithi047.stderr:norecover is unset 2024-09-18T15:40:00.953 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:342: TEST_recovery_undersized: get_primary test obj1 2024-09-18T15:40:00.954 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-18T15:40:00.954 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T15:40:00.954 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-18T15:40:00.954 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:40:01.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:342: TEST_recovery_undersized: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-18T15:40:01.497 INFO:tasks.workunit.client.0.smithi047.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-18T15:40:01.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:344: TEST_recovery_undersized: sleep 10 2024-09-18T15:40:11.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:345: TEST_recovery_undersized: flush_pg_stats 2024-09-18T15:40:11.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:40:11.508 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:40:11.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:40:11.923 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:40:11.923 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T15:40:11.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:40:11.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:40:11.923 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:40:12.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836512 2024-09-18T15:40:12.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836512 2024-09-18T15:40:12.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836512' 2024-09-18T15:40:12.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:40:12.031 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:40:12.212 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672983 2024-09-18T15:40:12.212 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672983 2024-09-18T15:40:12.212 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836512 1-42949672983' 2024-09-18T15:40:12.212 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:40:12.212 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:40:12.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509455 2024-09-18T15:40:12.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509455 2024-09-18T15:40:12.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836512 1-42949672983 2-64424509455' 2024-09-18T15:40:12.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:40:12.321 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836512 2024-09-18T15:40:12.321 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:40:12.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:40:12.322 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836512 2024-09-18T15:40:12.323 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:40:12.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836512 2024-09-18T15:40:12.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-09-18T15:40:12.324 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836512 2024-09-18T15:40:12.324 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:40:12.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836512 -lt 21474836512 2024-09-18T15:40:12.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:40:12.692 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672983 2024-09-18T15:40:12.692 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:40:12.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:40:12.693 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672983 2024-09-18T15:40:12.693 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:40:12.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672983 2024-09-18T15:40:12.695 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672983 2024-09-18T15:40:12.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-09-18T15:40:12.695 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:40:13.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672981 -lt 42949672983 2024-09-18T15:40:13.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:40:14.079 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:40:14.079 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:40:14.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672983 -lt 42949672983 2024-09-18T15:40:14.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:40:14.479 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509455 2024-09-18T15:40:14.479 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:40:14.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:40:14.480 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509455 2024-09-18T15:40:14.481 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:40:14.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509455 2024-09-18T15:40:14.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-09-18T15:40:14.482 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509455 2024-09-18T15:40:14.482 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:40:14.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509456 -lt 64424509455 2024-09-18T15:40:14.887 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:350: TEST_recovery_undersized: seq 1 300 2024-09-18T15:40:14.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:350: TEST_recovery_undersized: for i in $(seq 1 300) 2024-09-18T15:40:14.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: ceph pg dump pgs 2024-09-18T15:40:14.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep '^1.0' 2024-09-18T15:40:14.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep -qv recovering 2024-09-18T15:40:15.237 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T15:40:15.247 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:354: TEST_recovery_undersized: break 2024-09-18T15:40:15.248 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:365: TEST_recovery_undersized: get_primary test obj1 2024-09-18T15:40:15.248 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-18T15:40:15.248 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-18T15:40:15.248 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-18T15:40:15.248 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:40:15.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:365: TEST_recovery_undersized: primary=1 2024-09-18T15:40:15.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:366: TEST_recovery_undersized: local log=td/osd-recovery-stats/osd.1.log 2024-09-18T15:40:15.617 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:368: TEST_recovery_undersized: expr 200 '*' 3 2024-09-18T15:40:15.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:368: TEST_recovery_undersized: local first_degraded=600 2024-09-18T15:40:15.618 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:369: TEST_recovery_undersized: expr 200 '*' 2 2024-09-18T15:40:15.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:369: TEST_recovery_undersized: local last_degraded=400 2024-09-18T15:40:15.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:370: TEST_recovery_undersized: check td/osd-recovery-stats 1.0 1 replicated 600 400 0 0 2024-09-18T15:40:15.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-09-18T15:40:15.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-09-18T15:40:15.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=1 2024-09-18T15:40:15.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2024-09-18T15:40:15.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=600 2024-09-18T15:40:15.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=400 2024-09-18T15:40:15.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=0 2024-09-18T15:40:15.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-09-18T15:40:15.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2024-09-18T15:40:15.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2024-09-18T15:40:15.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.1.log 2024-09-18T15:40:15.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-09-18T15:40:15.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2024-09-18T15:40:15.621 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-09-18T15:40:15.621 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.1.log 2024-09-18T15:40:15.621 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-09-18T15:40:15.622 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-09-18T15:40:15.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[1,2]' 2024-09-18T15:40:15.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-09-18T15:40:15.632 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-18T15:40:15.632 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-09-18T15:40:15.632 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [1,2] ' 2024-09-18T15:40:15.632 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-09-18T15:40:15.632 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-09-18T15:40:15.632 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-18T15:40:15.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=600 2024-09-18T15:40:15.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 600 600 2024-09-18T15:40:15.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=600 2024-09-18T15:40:15.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-18T15:40:15.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=600 2024-09-18T15:40:15.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-18T15:40:15.637 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-18T15:40:15.637 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-09-18T15:40:15.637 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-09-18T15:40:15.637 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-18T15:40:15.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=400 2024-09-18T15:40:15.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 400 400 2024-09-18T15:40:15.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=400 2024-09-18T15:40:15.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-18T15:40:15.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=400 2024-09-18T15:40:15.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-18T15:40:15.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-09-18T15:40:15.645 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-18T15:40:15.645 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-09-18T15:40:15.645 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [1,2] ' 2024-09-18T15:40:15.645 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-09-18T15:40:15.645 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-09-18T15:40:15.645 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-18T15:40:15.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=0 2024-09-18T15:40:15.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 0 0 2024-09-18T15:40:15.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=0 2024-09-18T15:40:15.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-18T15:40:15.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=0 2024-09-18T15:40:15.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-18T15:40:15.650 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-18T15:40:15.650 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-09-18T15:40:15.650 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-09-18T15:40:15.650 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-18T15:40:15.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-09-18T15:40:15.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-09-18T15:40:15.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-18T15:40:15.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-18T15:40:15.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-18T15:40:15.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-18T15:40:15.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2024-09-18T15:40:15.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:372: TEST_recovery_undersized: delete_pool test 2024-09-18T15:40:15.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-18T15:40:15.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-18T15:40:16.262 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test' does not exist 2024-09-18T15:40:16.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:373: TEST_recovery_undersized: kill_daemons td/osd-recovery-stats 2024-09-18T15:40:16.273 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:40:16.273 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:40:16.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:40:16.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:40:16.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:40:17.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:40:17.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-09-18T15:40:17.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-18T15:40:17.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:40:17.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-18T15:40:17.584 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:40:17.584 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:40:17.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:40:17.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:40:17.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:40:17.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:40:17.591 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:40:17.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:40:17.592 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:40:17.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:40:17.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:40:17.594 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:40:17.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:40:17.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:40:17.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:40:17.596 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:40:17.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:40:17.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:40:17.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-18T15:40:17.621 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:40:17.621 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:40:17.621 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:40:17.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.288083 2024-09-18T15:40:17.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:40:17.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:40:17.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T15:40:17.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/osd-recovery-stats 0 2024-09-18T15:40:17.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-18T15:40:17.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T15:40:17.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-18T15:40:17.623 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:40:17.623 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:40:17.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:40:17.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:40:17.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:40:17.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:40:17.625 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:40:17.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:40:17.626 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:40:17.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:40:17.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:40:17.628 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:40:17.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:40:17.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:40:17.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:40:17.629 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:40:17.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:40:17.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T15:40:17.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-18T15:40:17.633 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:40:17.633 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:40:17.633 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.288083 2024-09-18T15:40:17.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.288083 2024-09-18T15:40:17.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:40:17.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:40:17.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T15:40:17.635 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T15:40:17.635 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T15:40:17.703 INFO:tasks.workunit:Running workunit osd/osd-rep-recov-eio.sh... 2024-09-18T15:40:17.703 DEBUG:teuthology.orchestra.run.smithi047:workunit test osd/osd-rep-recov-eio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh 2024-09-18T15:40:17.768 INFO:tasks.workunit.client.0.smithi047.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T15:40:17.774 INFO:tasks.workunit.client.0.smithi047.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T15:40:17.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T15:40:17.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T15:40:17.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T15:40:17.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T15:40:17.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T15:40:17.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T15:40:17.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T15:40:17.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T15:40:17.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/osd-rep-recov-eio.sh 2024-09-18T15:40:17.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:25: run: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:40:17.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:26: run: shift 2024-09-18T15:40:17.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:28: run: export CEPH_MON=127.0.0.1:7140 2024-09-18T15:40:17.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:28: run: CEPH_MON=127.0.0.1:7140 2024-09-18T15:40:17.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:29: run: export CEPH_ARGS 2024-09-18T15:40:17.775 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:30: run: uuidgen 2024-09-18T15:40:17.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:30: run: CEPH_ARGS+='--fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none ' 2024-09-18T15:40:17.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:31: run: CEPH_ARGS+='--mon-host=127.0.0.1:7140 ' 2024-09-18T15:40:17.776 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:34: run: set 2024-09-18T15:40:17.776 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:34: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-18T15:40:17.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:34: run: local 'funcs=TEST_rados_get_with_eio 2024-09-18T15:40:17.778 INFO:tasks.workunit.client.0.smithi047.stderr:TEST_rados_repair_warning 2024-09-18T15:40:17.778 INFO:tasks.workunit.client.0.smithi047.stderr:TEST_rep_backfill_unfound 2024-09-18T15:40:17.778 INFO:tasks.workunit.client.0.smithi047.stderr:TEST_rep_read_unfound 2024-09-18T15:40:17.778 INFO:tasks.workunit.client.0.smithi047.stderr:TEST_rep_recovery_unfound' 2024-09-18T15:40:17.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:35: run: for func in $funcs 2024-09-18T15:40:17.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:36: run: setup td/osd-rep-recov-eio.sh 2024-09-18T15:40:17.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:40:17.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-rep-recov-eio.sh 2024-09-18T15:40:17.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:40:17.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:40:17.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-09-18T15:40:17.779 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:40:17.779 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:40:17.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:40:17.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:40:17.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:40:17.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:40:17.779 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:40:17.780 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:40:17.780 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:40:17.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:40:17.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:40:17.782 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:40:17.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:40:17.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:40:17.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:40:17.783 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:40:17.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:40:17.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:40:17.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-09-18T15:40:17.786 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:40:17.786 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:40:17.786 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:40:17.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.397939 2024-09-18T15:40:17.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:40:17.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:40:17.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-rep-recov-eio.sh 2024-09-18T15:40:17.788 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T15:40:17.788 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:40:17.789 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:40:17.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.397939 2024-09-18T15:40:17.790 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T15:40:17.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T15:40:17.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T15:40:17.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T15:40:17.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-rep-recov-eio.sh 1' TERM HUP INT 2024-09-18T15:40:17.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:38: run: run_mon td/osd-rep-recov-eio.sh a --mon_osd_warn_num_repaired=10 2024-09-18T15:40:17.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:40:17.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T15:40:17.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T15:40:17.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T15:40:17.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-rep-recov-eio.sh/a 2024-09-18T15:40:17.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-rep-recov-eio.sh/a --run-dir=td/osd-rep-recov-eio.sh --mon_osd_warn_num_repaired=10 2024-09-18T15:40:18.132 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T15:40:18.132 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:40:18.132 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:40:18.132 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:40:18.132 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:40:18.132 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:40:18.132 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:40:18.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-rep-recov-eio.sh/a '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --mon-cluster-log-file=td/osd-rep-recov-eio.sh/log --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_osd_warn_num_repaired=10 2024-09-18T15:40:18.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T15:40:18.392 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T15:40:18.392 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:40:18.392 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:40:18.392 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T15:40:18.392 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T15:40:18.392 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:40:18.392 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:40:18.392 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:40:18.393 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:40:18.393 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:40:18.393 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:40:18.393 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-mon.a.asok 2024-09-18T15:40:18.393 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:40:18.394 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.397939/ceph-mon.a.asok config get fsid 2024-09-18T15:40:18.476 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T15:40:18.476 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:40:18.476 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:40:18.476 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T15:40:18.476 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T15:40:18.476 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:40:18.476 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:40:18.476 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:40:18.476 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:40:18.476 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:40:18.477 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:40:18.477 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-mon.a.asok 2024-09-18T15:40:18.477 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:40:18.477 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.397939/ceph-mon.a.asok config get mon_host 2024-09-18T15:40:18.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:39: run: run_mgr td/osd-rep-recov-eio.sh x 2024-09-18T15:40:18.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:40:18.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T15:40:18.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T15:40:18.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T15:40:18.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-rep-recov-eio.sh/x 2024-09-18T15:40:18.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T15:40:19.404 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T15:40:19.404 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:40:19.404 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:40:19.404 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:40:19.404 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:40:19.404 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:40:19.404 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:40:19.405 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:40:19.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-rep-recov-eio.sh/x '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:40:19.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:40: run: ceph osd pool create foo 8 2024-09-18T15:40:19.727 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'foo' already exists 2024-09-18T15:40:19.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:42: run: TEST_rados_get_with_eio td/osd-rep-recov-eio.sh 2024-09-18T15:40:19.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:168: TEST_rados_get_with_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:40:19.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:170: TEST_rados_get_with_eio: setup_osds 4 2024-09-18T15:40:19.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:48: setup_osds: local count=4 2024-09-18T15:40:19.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:49: setup_osds: shift 2024-09-18T15:40:19.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:50: setup_osds: local type= 2024-09-18T15:40:19.738 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: expr 4 - 1 2024-09-18T15:40:19.738 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: seq 0 3 2024-09-18T15:40:19.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T15:40:19.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 0 2024-09-18T15:40:19.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:40:19.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:40:19.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T15:40:19.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:40:19.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-09-18T15:40:19.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-09-18T15:40:19.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:40:19.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:40:19.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:40:19.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-09-18T15:40:19.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-09-18T15:40:19.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:40:19.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:40:19.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-18T15:40:19.744 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:40:19.744 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:40:19.744 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:40:19.744 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:40:19.744 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:40:19.744 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:40:19.744 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:40:19.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:40:19.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:40:19.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:40:19.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:40:19.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-18T15:40:19.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-18T15:40:19.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:40:19.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:40:19.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:40:19.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:40:19.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:40:19.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:40:19.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-09-18T15:40:19.745 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:40:19.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b8c05245-ffaa-452f-8a94-090c5b6bf6be 2024-09-18T15:40:19.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b8c05245-ffaa-452f-8a94-090c5b6bf6be' 2024-09-18T15:40:19.746 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 b8c05245-ffaa-452f-8a94-090c5b6bf6be 2024-09-18T15:40:19.746 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:40:19.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBj9OpmEvoELRAAfLbIqw7hWhvll6yIBzePrg== 2024-09-18T15:40:19.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBj9OpmEvoELRAAfLbIqw7hWhvll6yIBzePrg=="}' 2024-09-18T15:40:19.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b8c05245-ffaa-452f-8a94-090c5b6bf6be -i td/osd-rep-recov-eio.sh/0/new.json 2024-09-18T15:40:20.023 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:40:20.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/0/new.json 2024-09-18T15:40:20.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBj9OpmEvoELRAAfLbIqw7hWhvll6yIBzePrg== --osd-uuid b8c05245-ffaa-452f-8a94-090c5b6bf6be 2024-09-18T15:40:20.056 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:40:20.055+0000 7f939628d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:40:20.076 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:40:20.075+0000 7f939628d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:40:20.077 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:40:20.077+0000 7f939628d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:40:20.078 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:40:20.077+0000 7f939628d440 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-18T15:40:20.078 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:40:20.077+0000 7f939628d440 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-18T15:40:20.078 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:40:20.077+0000 7f939628d440 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-18T15:40:20.108 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:40:20.107+0000 7f939628d440 -1 bluestore(td/osd-rep-recov-eio.sh/0) _read_fsid unparsable uuid 2024-09-18T15:40:25.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/0/keyring 2024-09-18T15:40:25.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:40:25.372 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T15:40:25.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T15:40:25.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:40:25.883 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T15:40:25.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T15:40:25.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:40:25.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:40:25.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:40:25.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:40:25.904 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:40:25.903+0000 7f3f3591d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:40:25.948 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:40:25.947+0000 7f3f3591d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:40:25.949 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:40:25.949+0000 7f3f3591d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:40:26.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T15:40:26.247 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:40:26.247 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:40:26.247 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:40:26.247 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:40:26.247 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:40:26.247 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:40:26.248 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:40:26.248 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:40:26.248 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:40:26.477 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:40:26.477+0000 7f3f3591d440 -1 Falling back to public interface 2024-09-18T15:40:26.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:40:27.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:40:27.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:40:27.609 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:40:27.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:40:27.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:40:27.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:40:28.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:40:28.919 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:40:28.918+0000 7f3f3591d440 -1 osd.0 0 log_to_monitors true 2024-09-18T15:40:29.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:40:29.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:40:29.015 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:40:29.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:40:29.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:40:29.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:40:29.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:40:30.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:40:30.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:40:30.489 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:40:30.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:40:30.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:40:30.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:40:30.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:40:31.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:40:31.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:40:31.894 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:40:31.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:40:31.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:40:31.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:40:32.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:40:33.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:40:33.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:40:33.290 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:40:33.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:40:33.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:40:33.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:40:33.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:40:34.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:40:34.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:40:34.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:40:34.702 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:40:34.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:40:34.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:40:35.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:40:36.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:40:36.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:40:36.111 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:40:36.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:40:36.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:40:36.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:40:36.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:40:37.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:40:37.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:40:37.591 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:40:37.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:40:37.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:40:37.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:40:37.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:40:38.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:40:38.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:40:38.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:40:38.981 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:40:38.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:40:38.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:40:39.355 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:40:40.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:40:40.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:40:40.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:40:40.357 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:40:40.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:40:40.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:40:41.264 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:40:42.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:40:42.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:40:42.266 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:40:42.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:40:42.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:40:42.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:40:42.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:40:43.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:40:43.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:40:43.775 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:40:43.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:40:43.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:40:43.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:40:44.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:40:45.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:40:45.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:40:45.530 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:40:45.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:40:45.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:40:45.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:40:46.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:40:47.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:40:47.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:40:47.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:40:47.358 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:40:47.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:40:47.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:40:47.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:40:48.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:40:48.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:40:48.875 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:40:48.875 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:40:48.875 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:40:48.875 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:40:49.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:40:50.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:40:50.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:40:50.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:40:50.557 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:40:50.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:40:50.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:40:51.019 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/399058,v1:127.0.0.1:6803/399058] [v2:127.0.0.1:6804/399058,v1:127.0.0.1:6805/399058] exists,up b8c05245-ffaa-452f-8a94-090c5b6bf6be 2024-09-18T15:40:51.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:40:51.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:40:51.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:40:51.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T15:40:51.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 1 2024-09-18T15:40:51.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:40:51.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:40:51.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T15:40:51.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:40:51.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2024-09-18T15:40:51.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-09-18T15:40:51.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:40:51.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:40:51.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:40:51.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2024-09-18T15:40:51.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2024-09-18T15:40:51.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:40:51.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:40:51.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-18T15:40:51.024 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:40:51.024 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:40:51.024 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:40:51.025 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:40:51.025 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:40:51.025 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:40:51.025 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:40:51.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:40:51.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:40:51.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:40:51.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:40:51.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-18T15:40:51.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-18T15:40:51.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:40:51.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:40:51.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:40:51.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:40:51.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:40:51.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:40:51.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2024-09-18T15:40:51.026 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:40:51.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bd0ccc6d-3562-4f16-8161-163dafa47ebf 2024-09-18T15:40:51.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 bd0ccc6d-3562-4f16-8161-163dafa47ebf' 2024-09-18T15:40:51.026 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 bd0ccc6d-3562-4f16-8161-163dafa47ebf 2024-09-18T15:40:51.026 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:40:51.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCD9OpmsvUeAhAAFvlRlOkTvXwLOvKHr4/ukw== 2024-09-18T15:40:51.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCD9OpmsvUeAhAAFvlRlOkTvXwLOvKHr4/ukw=="}' 2024-09-18T15:40:51.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bd0ccc6d-3562-4f16-8161-163dafa47ebf -i td/osd-rep-recov-eio.sh/1/new.json 2024-09-18T15:40:51.670 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:40:51.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/1/new.json 2024-09-18T15:40:51.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCD9OpmsvUeAhAAFvlRlOkTvXwLOvKHr4/ukw== --osd-uuid bd0ccc6d-3562-4f16-8161-163dafa47ebf 2024-09-18T15:40:51.704 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:40:51.703+0000 7fd711580440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:40:51.706 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:40:51.705+0000 7fd711580440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:40:51.707 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:40:51.706+0000 7fd711580440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:40:51.708 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:40:51.707+0000 7fd711580440 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-18T15:40:51.708 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:40:51.707+0000 7fd711580440 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-18T15:40:51.708 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:40:51.707+0000 7fd711580440 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-18T15:40:51.746 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:40:51.745+0000 7fd711580440 -1 bluestore(td/osd-rep-recov-eio.sh/1) _read_fsid unparsable uuid 2024-09-18T15:40:57.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/1/keyring 2024-09-18T15:40:57.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:40:57.541 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T15:40:57.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T15:40:57.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:40:58.050 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T15:40:58.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T15:40:58.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:40:58.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:40:58.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:40:58.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:40:58.071 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:40:58.070+0000 7f2c7f212440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:40:58.073 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:40:58.072+0000 7f2c7f212440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:40:58.074 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:40:58.073+0000 7f2c7f212440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:40:58.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T15:40:58.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:40:58.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:40:58.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:40:58.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:40:58.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:40:58.445 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:40:58.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:40:58.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:40:58.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:40:58.588 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:40:58.587+0000 7f2c7f212440 -1 Falling back to public interface 2024-09-18T15:40:58.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:40:59.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:40:59.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:40:59.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:40:59.972 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:40:59.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:40:59.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:41:00.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:01.077 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:41:01.076+0000 7f2c7f212440 -1 osd.1 0 log_to_monitors true 2024-09-18T15:41:01.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:01.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:01.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:41:01.256 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:41:01.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:01.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:41:01.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:02.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:02.696 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:41:02.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:02.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:41:02.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:02.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:41:03.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:04.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:04.101 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:41:04.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:04.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:41:04.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:04.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:41:04.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:05.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:05.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:05.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:41:05.498 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:41:05.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:05.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:41:05.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:06.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:06.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:06.890 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:41:06.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:41:06.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:06.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:41:07.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:08.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:08.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:08.327 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:41:08.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:41:08.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:08.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:41:08.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:09.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:09.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:09.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:41:09.735 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:41:09.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:09.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:41:10.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:11.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:11.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:11.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:41:11.130 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:41:11.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:11.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:41:11.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:12.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:12.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:12.545 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:41:12.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:41:12.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:12.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:41:13.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:14.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:14.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:14.458 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:41:14.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:41:14.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:14.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:41:15.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:16.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:16.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:16.291 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:41:16.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:41:16.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:16.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:41:16.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:17.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:17.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:17.972 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:41:17.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:41:17.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:17.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:41:18.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:19.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:19.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:19.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:41:19.759 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:41:19.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:19.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:41:20.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:21.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:21.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:21.821 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:41:21.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:41:21.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:21.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:41:22.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:23.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:23.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:23.574 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:41:23.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:41:23.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:23.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:41:24.062 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/400786,v1:127.0.0.1:6811/400786] [v2:127.0.0.1:6812/400786,v1:127.0.0.1:6813/400786] exists,up bd0ccc6d-3562-4f16-8161-163dafa47ebf 2024-09-18T15:41:24.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:41:24.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:41:24.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:41:24.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T15:41:24.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 2 2024-09-18T15:41:24.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:41:24.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:41:24.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T15:41:24.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:41:24.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/2 2024-09-18T15:41:24.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-09-18T15:41:24.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:41:24.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:41:24.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:41:24.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2024-09-18T15:41:24.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2024-09-18T15:41:24.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:41:24.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:41:24.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-18T15:41:24.067 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:41:24.067 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:41:24.067 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:41:24.068 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:41:24.068 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:41:24.068 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:41:24.068 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:41:24.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:41:24.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:41:24.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:41:24.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:41:24.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-18T15:41:24.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-18T15:41:24.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:41:24.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:41:24.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:41:24.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:41:24.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:41:24.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:41:24.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/2 2024-09-18T15:41:24.069 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:41:24.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f3a2ea0b-7193-4b81-b586-b965dec55c24 2024-09-18T15:41:24.070 INFO:tasks.workunit.client.0.smithi047.stdout:add osd2 f3a2ea0b-7193-4b81-b586-b965dec55c24 2024-09-18T15:41:24.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f3a2ea0b-7193-4b81-b586-b965dec55c24' 2024-09-18T15:41:24.070 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:41:24.080 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCk9OpmrqPCBBAAUCqBkm3KqLO6qJIT/H4Vvw== 2024-09-18T15:41:24.080 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCk9OpmrqPCBBAAUCqBkm3KqLO6qJIT/H4Vvw=="}' 2024-09-18T15:41:24.080 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f3a2ea0b-7193-4b81-b586-b965dec55c24 -i td/osd-rep-recov-eio.sh/2/new.json 2024-09-18T15:41:24.648 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:41:24.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/2/new.json 2024-09-18T15:41:24.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCk9OpmrqPCBBAAUCqBkm3KqLO6qJIT/H4Vvw== --osd-uuid f3a2ea0b-7193-4b81-b586-b965dec55c24 2024-09-18T15:41:24.682 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:41:24.681+0000 7fb3631fe440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:41:24.684 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:41:24.683+0000 7fb3631fe440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:41:24.685 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:41:24.684+0000 7fb3631fe440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:41:24.685 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:41:24.684+0000 7fb3631fe440 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-18T15:41:24.685 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:41:24.684+0000 7fb3631fe440 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-18T15:41:24.685 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:41:24.684+0000 7fb3631fe440 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-18T15:41:24.730 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:41:24.730+0000 7fb3631fe440 -1 bluestore(td/osd-rep-recov-eio.sh/2) _read_fsid unparsable uuid 2024-09-18T15:41:30.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/2/keyring 2024-09-18T15:41:30.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:41:30.754 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd2 key to auth repository 2024-09-18T15:41:30.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T15:41:30.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:41:31.266 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T15:41:31.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T15:41:31.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:41:31.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:41:31.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:41:31.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:41:31.287 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:41:31.287+0000 7fa907b1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:41:31.313 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:41:31.312+0000 7fa907b1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:41:31.314 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:41:31.313+0000 7fa907b1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:41:31.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T15:41:31.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:41:31.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:41:31.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:41:31.705 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:41:31.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:41:31.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:31.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:41:31.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:31.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:41:31.853 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:41:31.852+0000 7fa907b1f440 -1 Falling back to public interface 2024-09-18T15:41:32.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:33.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:33.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:33.121 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:41:33.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:41:33.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:33.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:41:33.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:34.352 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:41:34.351+0000 7fa907b1f440 -1 osd.2 0 log_to_monitors true 2024-09-18T15:41:34.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:34.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:34.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:41:34.498 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:41:34.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:34.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:41:35.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:36.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:36.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:36.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:41:36.093 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:41:36.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:36.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:41:36.492 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:37.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:37.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:37.495 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:41:37.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:41:37.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:37.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:41:37.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:38.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:38.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:38.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:41:38.876 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:41:38.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:38.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:41:39.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:40.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:40.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:40.275 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:41:40.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:41:40.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:40.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:41:40.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:41.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:41.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:41.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:41:41.680 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:41:41.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:41.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:41:42.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:43.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:43.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:43.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:41:43.146 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:41:43.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:43.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:41:43.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:44.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:44.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:44.593 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:41:44.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:41:44.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:44.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:41:44.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:45.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:45.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:45.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:41:45.982 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:41:45.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:45.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:41:46.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:47.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:47.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:47.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:41:47.905 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:41:47.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:47.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:41:49.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:50.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:50.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:50.102 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:41:50.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:41:50.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:50.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:41:50.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:51.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:51.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:51.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:41:51.838 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:41:51.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:51.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:41:52.285 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:53.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:53.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:53.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:41:53.287 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:41:53.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:53.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:41:54.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:55.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:55.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:55.436 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:41:55.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:41:55.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:55.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:41:56.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:41:57.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:41:57.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:41:57.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:41:57.313 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:41:57.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:41:57.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:41:57.772 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/402516,v1:127.0.0.1:6819/402516] [v2:127.0.0.1:6820/402516,v1:127.0.0.1:6821/402516] exists,up f3a2ea0b-7193-4b81-b586-b965dec55c24 2024-09-18T15:41:57.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:41:57.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:41:57.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:41:57.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T15:41:57.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 3 2024-09-18T15:41:57.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:41:57.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:41:57.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-18T15:41:57.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:41:57.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/3 2024-09-18T15:41:57.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-09-18T15:41:57.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:41:57.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:41:57.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:41:57.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/3' 2024-09-18T15:41:57.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/3/journal' 2024-09-18T15:41:57.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:41:57.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:41:57.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-18T15:41:57.777 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:41:57.777 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:41:57.777 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:41:57.777 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:41:57.777 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:41:57.778 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:41:57.778 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:41:57.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:41:57.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:41:57.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:41:57.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:41:57.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-18T15:41:57.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-18T15:41:57.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:41:57.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:41:57.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:41:57.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:41:57.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:41:57.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:41:57.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/3 2024-09-18T15:41:57.779 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:41:57.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=70534b9c-354d-4434-9368-9c1a1a5e9c06 2024-09-18T15:41:57.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 70534b9c-354d-4434-9368-9c1a1a5e9c06' 2024-09-18T15:41:57.780 INFO:tasks.workunit.client.0.smithi047.stdout:add osd3 70534b9c-354d-4434-9368-9c1a1a5e9c06 2024-09-18T15:41:57.780 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:41:57.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDF9OpmfAgSLxAAtS51VX58eEnsF3RXk37w8A== 2024-09-18T15:41:57.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDF9OpmfAgSLxAAtS51VX58eEnsF3RXk37w8A=="}' 2024-09-18T15:41:57.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 70534b9c-354d-4434-9368-9c1a1a5e9c06 -i td/osd-rep-recov-eio.sh/3/new.json 2024-09-18T15:41:58.623 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:41:58.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/3/new.json 2024-09-18T15:41:58.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/3 --osd-journal=td/osd-rep-recov-eio.sh/3/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDF9OpmfAgSLxAAtS51VX58eEnsF3RXk37w8A== --osd-uuid 70534b9c-354d-4434-9368-9c1a1a5e9c06 2024-09-18T15:41:58.657 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:41:58.657+0000 7f025d131440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:41:58.672 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:41:58.671+0000 7f025d131440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:41:58.673 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:41:58.672+0000 7f025d131440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:41:58.674 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:41:58.672+0000 7f025d131440 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-09-18T15:41:58.674 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:41:58.672+0000 7f025d131440 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-09-18T15:41:58.674 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:41:58.672+0000 7f025d131440 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-09-18T15:41:58.772 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:41:58.771+0000 7f025d131440 -1 bluestore(td/osd-rep-recov-eio.sh/3) _read_fsid unparsable uuid 2024-09-18T15:42:04.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/3/keyring 2024-09-18T15:42:04.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:42:04.571 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd3 key to auth repository 2024-09-18T15:42:04.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-18T15:42:04.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:42:05.084 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.3 2024-09-18T15:42:05.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-18T15:42:05.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/3 --osd-journal=td/osd-rep-recov-eio.sh/3/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:42:05.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:42:05.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:42:05.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:42:05.105 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:42:05.104+0000 7fe5ddb62440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:42:05.129 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:42:05.129+0000 7fe5ddb62440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:42:05.130 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:42:05.130+0000 7fe5ddb62440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:42:05.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-18T15:42:05.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:42:05.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T15:42:05.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:42:05.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:42:05.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:05.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:42:05.511 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:42:05.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:05.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:42:05.661 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:42:05.660+0000 7fe5ddb62440 -1 Falling back to public interface 2024-09-18T15:42:05.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:42:06.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:42:06.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:06.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:42:06.872 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:42:06.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:06.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:42:07.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:42:08.220 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:42:08.219+0000 7fe5ddb62440 -1 osd.3 0 log_to_monitors true 2024-09-18T15:42:08.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:42:08.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:08.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:42:08.246 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:42:08.247 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:08.247 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:42:08.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:42:09.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:42:09.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:09.974 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:42:09.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:42:09.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:09.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:42:10.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:42:11.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:42:11.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:11.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:42:11.631 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:42:11.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:11.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:42:12.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:42:13.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:42:13.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:13.191 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:42:13.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:42:13.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:13.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:42:13.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:42:14.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:42:14.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:14.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:42:14.585 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:42:14.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:14.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:42:15.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:42:16.052 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:42:16.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:16.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:42:16.053 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:42:16.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:16.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:42:16.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:42:17.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:42:17.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:17.523 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:42:17.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:42:17.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:17.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:42:17.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:42:18.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:42:18.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:18.902 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:42:18.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:42:18.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:18.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:42:19.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:42:20.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:42:20.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:20.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:42:20.359 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:42:20.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:20.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:42:20.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:42:21.740 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:42:21.740 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:21.740 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:42:21.741 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:42:21.741 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:21.741 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:42:22.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:42:23.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:42:23.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:23.531 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:42:23.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:42:23.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:23.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:42:24.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:42:25.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:42:25.209 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:42:25.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:25.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:42:25.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:25.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:42:25.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:42:26.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:42:26.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:26.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:42:26.896 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:42:26.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:26.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:42:27.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:42:28.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:42:28.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:28.624 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:42:28.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:42:28.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:28.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:42:29.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:42:30.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:42:30.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:30.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:42:30.524 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:42:30.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:30.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:42:31.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:42:32.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:42:32.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:42:32.297 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T15:42:32.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:42:32.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:42:32.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:42:32.849 INFO:tasks.workunit.client.0.smithi047.stdout:osd.3 up in weight 1 up_from 32 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/404245,v1:127.0.0.1:6827/404245] [v2:127.0.0.1:6828/404245,v1:127.0.0.1:6829/404245] exists,up 70534b9c-354d-4434-9368-9c1a1a5e9c06 2024-09-18T15:42:32.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:42:32.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:42:32.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:42:32.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:55: setup_osds: wait_for_clean 2024-09-18T15:42:32.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:42:32.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:42:32.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:42:32.851 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:42:32.851 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:42:32.851 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:42:32.851 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:42:32.851 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:42:32.851 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:42:32.938 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:42:32.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:42:32.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:42:32.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:42:32.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:42:32.939 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:42:33.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:42:33.589 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:42:33.589 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T15:42:33.589 INFO:tasks.workunit.client.0.smithi047.stderr:3' 2024-09-18T15:42:33.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:42:33.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:42:33.589 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:42:33.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803802 2024-09-18T15:42:33.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803802 2024-09-18T15:42:33.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803802' 2024-09-18T15:42:33.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:42:33.771 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:42:33.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=51539607572 2024-09-18T15:42:33.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 51539607572 2024-09-18T15:42:33.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803802 1-51539607572' 2024-09-18T15:42:33.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:42:33.985 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:42:34.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378637 2024-09-18T15:42:34.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378637 2024-09-18T15:42:34.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803802 1-51539607572 2-81604378637' 2024-09-18T15:42:34.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:42:34.190 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:42:34.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=137438953481 2024-09-18T15:42:34.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 137438953481 2024-09-18T15:42:34.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803802 1-51539607572 2-81604378637 3-137438953481' 2024-09-18T15:42:34.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:42:34.378 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803802 2024-09-18T15:42:34.378 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:42:34.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:42:34.379 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803802 2024-09-18T15:42:34.379 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:42:34.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803802 2024-09-18T15:42:34.380 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 25769803802 2024-09-18T15:42:34.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803802' 2024-09-18T15:42:34.381 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:42:34.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803800 -lt 25769803802 2024-09-18T15:42:34.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:42:35.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:42:35.814 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:42:36.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803802 -lt 25769803802 2024-09-18T15:42:36.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:42:36.235 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-51539607572 2024-09-18T15:42:36.235 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:42:36.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:42:36.237 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-51539607572 2024-09-18T15:42:36.237 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:42:36.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=51539607572 2024-09-18T15:42:36.238 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 51539607572 2024-09-18T15:42:36.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 51539607572' 2024-09-18T15:42:36.238 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:42:36.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 51539607572 -lt 51539607572 2024-09-18T15:42:36.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:42:36.637 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378637 2024-09-18T15:42:36.637 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:42:36.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:42:36.639 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378637 2024-09-18T15:42:36.639 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:42:36.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378637 2024-09-18T15:42:36.640 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 81604378637 2024-09-18T15:42:36.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378637' 2024-09-18T15:42:36.640 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:42:37.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378637 -lt 81604378637 2024-09-18T15:42:37.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:42:37.004 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-137438953481 2024-09-18T15:42:37.004 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:42:37.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:42:37.006 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-137438953481 2024-09-18T15:42:37.006 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:42:37.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=137438953481 2024-09-18T15:42:37.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 137438953481' 2024-09-18T15:42:37.007 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 137438953481 2024-09-18T15:42:37.007 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:42:37.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 137438953481 -lt 137438953481 2024-09-18T15:42:37.421 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:42:37.421 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:42:37.421 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:42:37.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-18T15:42:37.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:42:37.860 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:42:37.860 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:42:37.860 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:42:37.860 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:42:37.860 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:42:37.860 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:42:38.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:42:38.222 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:42:38.223 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:42:38.223 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:42:38.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 8 2024-09-18T15:42:38.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-09-18T15:42:38.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T15:42:38.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-09-18T15:42:38.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:42:38.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:42:38.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:42:38.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:42:38.816 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:42:38.816 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:42:38.817 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:42:38.817 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:42:38.817 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:42:38.817 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:42:39.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-18T15:42:39.181 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:42:39.181 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:42:39.181 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:42:39.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-18T15:42:39.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:42:39.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:42:39.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:172: TEST_rados_get_with_eio: local poolname=pool-rep 2024-09-18T15:42:39.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:173: TEST_rados_get_with_eio: create_pool pool-rep 1 1 2024-09-18T15:42:39.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-rep 1 1 2024-09-18T15:42:40.183 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'pool-rep' already exists 2024-09-18T15:42:40.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:42:41.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:174: TEST_rados_get_with_eio: wait_for_clean 2024-09-18T15:42:41.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:42:41.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:42:41.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:42:41.196 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:42:41.196 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:42:41.196 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:42:41.197 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:42:41.197 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:42:41.197 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:42:41.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:42:41.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:42:41.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:42:41.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:42:41.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:42:41.297 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:42:41.666 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:42:41.666 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:42:41.666 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T15:42:41.666 INFO:tasks.workunit.client.0.smithi047.stderr:3' 2024-09-18T15:42:41.666 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:42:41.666 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:42:41.667 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:42:41.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803806 2024-09-18T15:42:41.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803806 2024-09-18T15:42:41.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803806' 2024-09-18T15:42:41.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:42:41.772 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:42:41.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=51539607575 2024-09-18T15:42:41.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 51539607575 2024-09-18T15:42:41.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803806 1-51539607575' 2024-09-18T15:42:41.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:42:41.880 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:42:41.985 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378640 2024-09-18T15:42:41.985 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378640 2024-09-18T15:42:41.985 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803806 1-51539607575 2-81604378640' 2024-09-18T15:42:41.985 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:42:41.986 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:42:42.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=137438953484 2024-09-18T15:42:42.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 137438953484 2024-09-18T15:42:42.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803806 1-51539607575 2-81604378640 3-137438953484' 2024-09-18T15:42:42.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:42:42.092 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803806 2024-09-18T15:42:42.092 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:42:42.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:42:42.093 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803806 2024-09-18T15:42:42.094 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:42:42.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803806 2024-09-18T15:42:42.095 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803806' 2024-09-18T15:42:42.095 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 25769803806 2024-09-18T15:42:42.095 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:42:42.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803803 -lt 25769803806 2024-09-18T15:42:42.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:42:43.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:42:43.504 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:42:43.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803806 -lt 25769803806 2024-09-18T15:42:43.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:42:43.886 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-51539607575 2024-09-18T15:42:43.886 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:42:43.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:42:43.888 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-51539607575 2024-09-18T15:42:43.888 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:42:43.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=51539607575 2024-09-18T15:42:43.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 51539607575' 2024-09-18T15:42:43.889 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 51539607575 2024-09-18T15:42:43.889 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:42:44.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 51539607575 -lt 51539607575 2024-09-18T15:42:44.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:42:44.254 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378640 2024-09-18T15:42:44.254 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:42:44.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:42:44.255 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378640 2024-09-18T15:42:44.255 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:42:44.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378640 2024-09-18T15:42:44.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378640' 2024-09-18T15:42:44.257 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 81604378640 2024-09-18T15:42:44.257 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:42:44.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378641 -lt 81604378640 2024-09-18T15:42:44.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:42:44.617 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-137438953484 2024-09-18T15:42:44.617 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:42:44.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:42:44.619 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-137438953484 2024-09-18T15:42:44.619 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:42:44.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=137438953484 2024-09-18T15:42:44.620 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 137438953484 2024-09-18T15:42:44.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 137438953484' 2024-09-18T15:42:44.621 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:42:45.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 137438953485 -lt 137438953484 2024-09-18T15:42:45.009 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:42:45.009 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:42:45.010 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:42:45.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-18T15:42:45.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:42:45.506 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:42:45.506 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:42:45.506 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:42:45.506 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:42:45.506 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:42:45.506 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:42:45.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-09-18T15:42:45.878 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:42:45.878 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:42:45.878 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:42:46.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-09-18T15:42:46.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:42:46.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:42:46.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:175: TEST_rados_get_with_eio: rados_get_data eio td/osd-rep-recov-eio.sh 2024-09-18T15:42:46.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:111: rados_get_data: local inject=eio 2024-09-18T15:42:46.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:112: rados_get_data: shift 2024-09-18T15:42:46.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:113: rados_get_data: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:42:46.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:115: rados_get_data: local poolname=pool-rep 2024-09-18T15:42:46.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:116: rados_get_data: local objname=obj-eio-397939 2024-09-18T15:42:46.380 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:117: rados_get_data: get_pg pool-rep obj-eio-397939 2024-09-18T15:42:46.380 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=pool-rep 2024-09-18T15:42:46.380 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj-eio-397939 2024-09-18T15:42:46.380 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map pool-rep obj-eio-397939 2024-09-18T15:42:46.380 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:42:46.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:117: rados_get_data: local pgid=2.0 2024-09-18T15:42:46.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:119: rados_get_data: rados_put td/osd-rep-recov-eio.sh pool-rep obj-eio-397939 2024-09-18T15:42:46.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:42:46.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-18T15:42:46.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-eio-397939 2024-09-18T15:42:46.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:42:46.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-18T15:42:46.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:42:46.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-18T15:42:46.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:42:46.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-18T15:42:46.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:42:46.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-18T15:42:46.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-eio-397939 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:42:46.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:120: rados_get_data: inject_eio rep data pool-rep obj-eio-397939 td/osd-rep-recov-eio.sh 0 2024-09-18T15:42:46.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:42:46.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:42:46.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:42:46.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:42:46.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:42:46.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:42:46.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-eio-397939 2024-09-18T15:42:46.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:42:46.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:42:46.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:42:46.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T15:42:46.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:42:46.964 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-eio-397939 2024-09-18T15:42:46.964 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:42:46.964 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-397939 2024-09-18T15:42:46.964 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-397939 2024-09-18T15:42:46.965 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:42:47.332 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:42:47.332 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:42:47.332 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:42:47.332 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:42:47.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:42:47.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:42:47.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-18T15:42:47.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:42:47.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:42:47.333 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-18T15:42:47.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:42:47.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-18T15:42:47.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:42:47.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:42:47.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:42:47.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:42:47.335 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:42:47.335 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:42:47.335 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:42:47.335 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:42:47.335 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:42:47.335 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:42:47.335 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:42:47.335 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:42:47.335 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:42:47.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:42:47.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:42:47.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:42:47.418 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-18T15:42:47.418 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:42:47.418 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:42:47.418 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:42:47.419 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:42:47.419 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:42:47.419 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:42:47.419 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:42:47.419 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.3.asok injectdataerr pool-rep obj-eio-397939 2024-09-18T15:42:47.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:121: rados_get_data: rados_get td/osd-rep-recov-eio.sh pool-rep obj-eio-397939 2024-09-18T15:42:47.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:42:47.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-18T15:42:47.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-eio-397939 2024-09-18T15:42:47.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-18T15:42:47.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-18T15:42:47.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-18T15:42:47.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-eio-397939 td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:42:47.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:42:47.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:42:47.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:123: rados_get_data: wait_for_clean 2024-09-18T15:42:47.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:42:47.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:42:47.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:42:47.532 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:42:47.533 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:42:47.533 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:42:47.533 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:42:47.533 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:42:47.533 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:42:47.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:42:47.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:42:47.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:42:47.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:42:47.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:42:47.621 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:42:48.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:42:48.077 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:42:48.077 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T15:42:48.077 INFO:tasks.workunit.client.0.smithi047.stderr:3' 2024-09-18T15:42:48.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:42:48.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:42:48.078 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:42:48.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803809 2024-09-18T15:42:48.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803809 2024-09-18T15:42:48.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803809' 2024-09-18T15:42:48.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:42:48.183 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:42:48.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=51539607578 2024-09-18T15:42:48.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 51539607578 2024-09-18T15:42:48.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803809 1-51539607578' 2024-09-18T15:42:48.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:42:48.290 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:42:48.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378644 2024-09-18T15:42:48.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378644 2024-09-18T15:42:48.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803809 1-51539607578 2-81604378644' 2024-09-18T15:42:48.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:42:48.410 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:42:48.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=137438953488 2024-09-18T15:42:48.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 137438953488 2024-09-18T15:42:48.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803809 1-51539607578 2-81604378644 3-137438953488' 2024-09-18T15:42:48.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:42:48.516 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803809 2024-09-18T15:42:48.516 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:42:48.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:42:48.517 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803809 2024-09-18T15:42:48.517 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:42:48.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803809 2024-09-18T15:42:48.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803809' 2024-09-18T15:42:48.519 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 25769803809 2024-09-18T15:42:48.519 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:42:48.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803807 -lt 25769803809 2024-09-18T15:42:48.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:42:49.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:42:49.930 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:42:50.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803809 -lt 25769803809 2024-09-18T15:42:50.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:42:50.356 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-51539607578 2024-09-18T15:42:50.356 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:42:50.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:42:50.358 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-51539607578 2024-09-18T15:42:50.358 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:42:50.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=51539607578 2024-09-18T15:42:50.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 51539607578' 2024-09-18T15:42:50.359 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 51539607578 2024-09-18T15:42:50.360 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:42:50.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 51539607579 -lt 51539607578 2024-09-18T15:42:50.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:42:50.766 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378644 2024-09-18T15:42:50.766 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:42:50.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:42:50.768 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378644 2024-09-18T15:42:50.768 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:42:50.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378644 2024-09-18T15:42:50.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378644' 2024-09-18T15:42:50.770 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 81604378644 2024-09-18T15:42:50.770 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:42:51.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378644 -lt 81604378644 2024-09-18T15:42:51.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:42:51.147 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-137438953488 2024-09-18T15:42:51.148 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:42:51.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:42:51.149 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-137438953488 2024-09-18T15:42:51.149 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:42:51.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=137438953488 2024-09-18T15:42:51.150 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 137438953488 2024-09-18T15:42:51.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 137438953488' 2024-09-18T15:42:51.151 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:42:51.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 137438953488 -lt 137438953488 2024-09-18T15:42:51.543 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:42:51.543 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:42:51.543 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:42:52.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-18T15:42:52.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:42:52.035 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:42:52.036 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:42:52.036 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:42:52.036 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:42:52.036 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:42:52.036 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:42:52.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-09-18T15:42:52.432 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:42:52.432 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:42:52.432 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:42:52.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-09-18T15:42:52.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:42:52.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:42:52.952 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:124: rados_get_data: ceph pg 2.0 query 2024-09-18T15:42:52.952 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:124: rados_get_data: jq .info.stats.stat_sum.num_objects_repaired 2024-09-18T15:42:53.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:124: rados_get_data: COUNT=1 2024-09-18T15:42:53.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:125: rados_get_data: test 1 = 1 2024-09-18T15:42:53.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:126: rados_get_data: flush_pg_stats 2024-09-18T15:42:53.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:42:53.063 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:42:53.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:42:53.425 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:42:53.426 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T15:42:53.426 INFO:tasks.workunit.client.0.smithi047.stderr:3' 2024-09-18T15:42:53.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:42:53.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:42:53.426 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:42:53.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803812 2024-09-18T15:42:53.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803812 2024-09-18T15:42:53.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803812' 2024-09-18T15:42:53.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:42:53.532 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:42:53.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=51539607582 2024-09-18T15:42:53.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 51539607582 2024-09-18T15:42:53.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803812 1-51539607582' 2024-09-18T15:42:53.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:42:53.655 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:42:53.760 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378647 2024-09-18T15:42:53.760 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378647 2024-09-18T15:42:53.760 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803812 1-51539607582 2-81604378647' 2024-09-18T15:42:53.760 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:42:53.760 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:42:53.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=137438953491 2024-09-18T15:42:53.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 137438953491 2024-09-18T15:42:53.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803812 1-51539607582 2-81604378647 3-137438953491' 2024-09-18T15:42:53.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:42:53.867 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803812 2024-09-18T15:42:53.867 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:42:53.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:42:53.868 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803812 2024-09-18T15:42:53.868 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:42:53.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803812 2024-09-18T15:42:53.869 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 25769803812 2024-09-18T15:42:53.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803812' 2024-09-18T15:42:53.870 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:42:54.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803810 -lt 25769803812 2024-09-18T15:42:54.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:42:55.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:42:55.233 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:42:55.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803812 -lt 25769803812 2024-09-18T15:42:55.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:42:55.612 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-51539607582 2024-09-18T15:42:55.612 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:42:55.612 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:42:55.613 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-51539607582 2024-09-18T15:42:55.613 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:42:55.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=51539607582 2024-09-18T15:42:55.614 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 51539607582 2024-09-18T15:42:55.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 51539607582' 2024-09-18T15:42:55.615 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:42:55.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 51539607582 -lt 51539607582 2024-09-18T15:42:55.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:42:56.000 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378647 2024-09-18T15:42:56.000 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:42:56.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:42:56.001 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378647 2024-09-18T15:42:56.002 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:42:56.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378647 2024-09-18T15:42:56.003 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 81604378647 2024-09-18T15:42:56.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378647' 2024-09-18T15:42:56.003 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:42:56.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378647 -lt 81604378647 2024-09-18T15:42:56.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:42:56.401 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-137438953491 2024-09-18T15:42:56.401 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:42:56.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:42:56.403 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-137438953491 2024-09-18T15:42:56.403 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:42:56.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=137438953491 2024-09-18T15:42:56.404 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 137438953491 2024-09-18T15:42:56.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 137438953491' 2024-09-18T15:42:56.404 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:42:56.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 137438953491 -lt 137438953491 2024-09-18T15:42:56.766 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:127: rados_get_data: ceph pg dump --format=json-pretty 2024-09-18T15:42:56.766 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:127: rados_get_data: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-09-18T15:42:57.116 INFO:tasks.workunit.client.0.smithi047.stderr:dumped all 2024-09-18T15:42:57.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:127: rados_get_data: COUNT=1 2024-09-18T15:42:57.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:128: rados_get_data: test 1 = 1 2024-09-18T15:42:57.126 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:130: rados_get_data: get_osds pool-rep obj-eio-397939 2024-09-18T15:42:57.126 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:42:57.126 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-397939 2024-09-18T15:42:57.126 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-397939 2024-09-18T15:42:57.126 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:42:57.539 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:42:57.539 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:42:57.539 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:42:57.539 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:42:57.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:130: rados_get_data: object_osds=('3' '1' '0') 2024-09-18T15:42:57.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:130: rados_get_data: local object_osds 2024-09-18T15:42:57.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:131: rados_get_data: local primary=3 2024-09-18T15:42:57.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:132: rados_get_data: local bad_peer=1 2024-09-18T15:42:57.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:133: rados_get_data: inject_eio rep data pool-rep obj-eio-397939 td/osd-rep-recov-eio.sh 0 2024-09-18T15:42:57.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:42:57.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:42:57.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:42:57.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:42:57.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:42:57.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:42:57.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-eio-397939 2024-09-18T15:42:57.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:42:57.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:42:57.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:42:57.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T15:42:57.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:42:57.541 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-eio-397939 2024-09-18T15:42:57.541 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:42:57.541 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-397939 2024-09-18T15:42:57.541 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-397939 2024-09-18T15:42:57.541 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:42:57.909 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:42:57.909 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:42:57.910 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:42:57.910 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:42:57.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:42:57.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:42:57.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-18T15:42:57.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:42:57.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:42:57.910 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-18T15:42:57.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:42:57.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-18T15:42:57.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:42:57.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:42:57.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:42:57.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:42:57.912 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:42:57.912 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:42:57.912 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:42:57.912 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:42:57.912 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:42:57.913 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:42:57.913 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:42:57.913 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:42:57.913 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:42:57.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:42:57.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:42:57.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:42:57.996 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-18T15:42:57.996 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:42:57.996 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:42:57.996 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:42:57.996 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:42:57.996 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:42:57.996 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:42:57.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:42:57.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.3.asok injectdataerr pool-rep obj-eio-397939 2024-09-18T15:42:58.080 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:134: rados_get_data: inject_eio rep data pool-rep obj-eio-397939 td/osd-rep-recov-eio.sh 1 2024-09-18T15:42:58.080 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:42:58.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:42:58.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:42:58.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:42:58.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:42:58.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:42:58.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-eio-397939 2024-09-18T15:42:58.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:42:58.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:42:58.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:42:58.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-18T15:42:58.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:42:58.081 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-eio-397939 2024-09-18T15:42:58.082 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:42:58.082 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-397939 2024-09-18T15:42:58.082 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-397939 2024-09-18T15:42:58.082 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:42:58.491 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:42:58.491 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:42:58.491 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:42:58.491 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:42:58.491 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:42:58.491 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:42:58.491 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-18T15:42:58.491 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:42:58.492 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:42:58.492 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-18T15:42:58.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:42:58.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-18T15:42:58.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:42:58.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-18T15:42:58.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:42:58.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:42:58.494 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:42:58.494 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-18T15:42:58.494 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:42:58.494 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:42:58.494 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:42:58.494 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:42:58.494 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:42:58.494 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.1.asok 2024-09-18T15:42:58.494 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:42:58.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:42:58.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:42:58.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:42:58.578 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-18T15:42:58.579 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:42:58.579 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:42:58.579 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:42:58.579 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:42:58.579 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:42:58.579 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.1.asok 2024-09-18T15:42:58.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:42:58.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.1.asok injectdataerr pool-rep obj-eio-397939 2024-09-18T15:42:58.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:136: rados_get_data: set_config osd 3 osd_debug_feed_pullee 1 2024-09-18T15:42:58.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:42:58.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:42:58.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-18T15:42:58.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-18T15:42:58.665 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:42:58.665 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:42:58.665 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:42:58.665 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:42:58.665 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:42:58.665 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:42:58.665 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:42:58.665 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:42:58.665 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-18T15:42:58.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:42:58.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:137: rados_get_data: rados_get td/osd-rep-recov-eio.sh pool-rep obj-eio-397939 2024-09-18T15:42:58.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:42:58.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-18T15:42:58.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-eio-397939 2024-09-18T15:42:58.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-18T15:42:58.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-18T15:42:58.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-18T15:42:58.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-eio-397939 td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:42:58.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:42:58.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:42:58.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:140: rados_get_data: wait_for_clean 2024-09-18T15:42:58.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:42:58.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:42:58.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:42:58.776 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:42:58.776 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:42:58.776 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:42:58.776 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:42:58.776 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:42:58.776 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:42:58.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:42:58.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:42:58.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:42:58.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:42:58.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:42:58.864 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:42:59.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:42:59.232 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:42:59.232 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T15:42:59.232 INFO:tasks.workunit.client.0.smithi047.stderr:3' 2024-09-18T15:42:59.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:42:59.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:42:59.232 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:42:59.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803815 2024-09-18T15:42:59.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803815 2024-09-18T15:42:59.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803815' 2024-09-18T15:42:59.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:42:59.338 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:42:59.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=51539607585 2024-09-18T15:42:59.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 51539607585 2024-09-18T15:42:59.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803815 1-51539607585' 2024-09-18T15:42:59.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:42:59.467 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:42:59.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378650 2024-09-18T15:42:59.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378650 2024-09-18T15:42:59.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803815 1-51539607585 2-81604378650' 2024-09-18T15:42:59.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:42:59.615 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:42:59.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=137438953494 2024-09-18T15:42:59.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 137438953494 2024-09-18T15:42:59.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803815 1-51539607585 2-81604378650 3-137438953494' 2024-09-18T15:42:59.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:42:59.722 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803815 2024-09-18T15:42:59.722 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:42:59.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:42:59.723 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803815 2024-09-18T15:42:59.723 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:42:59.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803815 2024-09-18T15:42:59.724 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 25769803815 2024-09-18T15:42:59.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803815' 2024-09-18T15:42:59.725 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:43:00.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803813 -lt 25769803815 2024-09-18T15:43:00.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:43:01.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:43:01.091 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:43:01.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803815 -lt 25769803815 2024-09-18T15:43:01.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:43:01.485 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-51539607585 2024-09-18T15:43:01.485 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:43:01.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:43:01.486 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-51539607585 2024-09-18T15:43:01.486 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:43:01.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=51539607585 2024-09-18T15:43:01.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 51539607585' 2024-09-18T15:43:01.488 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 51539607585 2024-09-18T15:43:01.488 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:43:01.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 51539607585 -lt 51539607585 2024-09-18T15:43:01.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:43:01.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378650 2024-09-18T15:43:01.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:43:01.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:43:01.850 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378650 2024-09-18T15:43:01.850 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:43:01.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378650 2024-09-18T15:43:01.851 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 81604378650 2024-09-18T15:43:01.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378650' 2024-09-18T15:43:01.851 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:43:02.212 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378650 -lt 81604378650 2024-09-18T15:43:02.212 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:43:02.213 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-137438953494 2024-09-18T15:43:02.213 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:43:02.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:43:02.214 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-137438953494 2024-09-18T15:43:02.214 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:43:02.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=137438953494 2024-09-18T15:43:02.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 137438953494' 2024-09-18T15:43:02.216 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 137438953494 2024-09-18T15:43:02.216 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:43:02.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 137438953494 -lt 137438953494 2024-09-18T15:43:02.626 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:43:02.626 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:43:02.626 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:43:03.119 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-18T15:43:03.119 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:43:03.120 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:43:03.120 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:43:03.120 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:43:03.120 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:43:03.120 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:43:03.120 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:43:03.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-09-18T15:43:03.499 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:43:03.499 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:43:03.499 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:43:04.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-09-18T15:43:04.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:43:04.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:43:04.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:142: rados_get_data: inject_eio rep data pool-rep obj-eio-397939 td/osd-rep-recov-eio.sh 0 2024-09-18T15:43:04.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:43:04.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:43:04.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:43:04.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:43:04.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:43:04.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:43:04.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-eio-397939 2024-09-18T15:43:04.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:43:04.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:43:04.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:43:04.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T15:43:04.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:43:04.031 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-eio-397939 2024-09-18T15:43:04.032 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:43:04.032 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-397939 2024-09-18T15:43:04.032 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-397939 2024-09-18T15:43:04.032 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:43:04.432 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:43:04.433 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:43:04.433 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:43:04.433 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:43:04.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:43:04.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:43:04.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-18T15:43:04.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:43:04.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:43:04.434 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-18T15:43:04.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:43:04.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-18T15:43:04.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:43:04.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:43:04.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:43:04.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:43:04.436 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:43:04.436 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:43:04.436 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:43:04.436 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:43:04.436 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:43:04.436 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:43:04.436 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:43:04.436 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:43:04.437 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:43:04.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:43:04.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:43:04.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:43:04.520 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-18T15:43:04.520 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:43:04.520 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:43:04.521 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:43:04.521 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:43:04.521 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:43:04.521 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:43:04.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:43:04.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.3.asok injectdataerr pool-rep obj-eio-397939 2024-09-18T15:43:04.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:143: rados_get_data: inject_eio rep data pool-rep obj-eio-397939 td/osd-rep-recov-eio.sh 2 2024-09-18T15:43:04.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:43:04.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:43:04.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:43:04.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:43:04.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:43:04.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:43:04.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-eio-397939 2024-09-18T15:43:04.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:43:04.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:43:04.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:43:04.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=2 2024-09-18T15:43:04.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:43:04.606 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-eio-397939 2024-09-18T15:43:04.606 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:43:04.606 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-397939 2024-09-18T15:43:04.606 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-397939 2024-09-18T15:43:04.606 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:43:04.967 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:43:04.968 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:43:04.968 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:43:04.968 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:43:04.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:43:04.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:43:04.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=0 2024-09-18T15:43:04.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:43:04.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:43:04.969 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/0/type 2024-09-18T15:43:04.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:43:04.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-09-18T15:43:04.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:43:04.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-09-18T15:43:04.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:43:04.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:43:04.971 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:43:04.971 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-09-18T15:43:04.971 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T15:43:04.971 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T15:43:04.971 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:43:04.971 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:43:04.971 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:43:04.971 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.0.asok 2024-09-18T15:43:04.971 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:43:05.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:43:05.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:43:05.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:43:05.055 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.0 2024-09-18T15:43:05.055 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T15:43:05.055 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T15:43:05.055 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:43:05.055 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:43:05.055 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:43:05.055 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.0.asok 2024-09-18T15:43:05.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:43:05.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.0.asok injectdataerr pool-rep obj-eio-397939 2024-09-18T15:43:05.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:144: rados_get_data: rados_get td/osd-rep-recov-eio.sh pool-rep obj-eio-397939 2024-09-18T15:43:05.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:43:05.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-18T15:43:05.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-eio-397939 2024-09-18T15:43:05.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-18T15:43:05.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-18T15:43:05.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-18T15:43:05.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-eio-397939 td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:43:05.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:43:05.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:43:05.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:146: rados_get_data: wait_for_clean 2024-09-18T15:43:05.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:43:05.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:43:05.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:43:05.168 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:43:05.168 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:43:05.168 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:43:05.168 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:43:05.169 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:43:05.169 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:43:05.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:43:05.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:43:05.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:43:05.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:43:05.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:43:05.257 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:43:05.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:43:05.708 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:43:05.708 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T15:43:05.708 INFO:tasks.workunit.client.0.smithi047.stderr:3' 2024-09-18T15:43:05.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:43:05.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:43:05.709 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:43:05.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803818 2024-09-18T15:43:05.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803818 2024-09-18T15:43:05.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803818' 2024-09-18T15:43:05.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:43:05.814 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:43:05.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=51539607588 2024-09-18T15:43:05.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 51539607588 2024-09-18T15:43:05.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803818 1-51539607588' 2024-09-18T15:43:05.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:43:05.920 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:43:06.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378653 2024-09-18T15:43:06.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378653 2024-09-18T15:43:06.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803818 1-51539607588 2-81604378653' 2024-09-18T15:43:06.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:43:06.025 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:43:06.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=137438953497 2024-09-18T15:43:06.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 137438953497 2024-09-18T15:43:06.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803818 1-51539607588 2-81604378653 3-137438953497' 2024-09-18T15:43:06.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:43:06.132 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803818 2024-09-18T15:43:06.132 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:43:06.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:43:06.133 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803818 2024-09-18T15:43:06.133 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:43:06.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803818 2024-09-18T15:43:06.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803818' 2024-09-18T15:43:06.134 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 25769803818 2024-09-18T15:43:06.135 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:43:06.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803816 -lt 25769803818 2024-09-18T15:43:06.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:43:07.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:43:07.536 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:43:07.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803819 -lt 25769803818 2024-09-18T15:43:07.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:43:07.913 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-51539607588 2024-09-18T15:43:07.913 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:43:07.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:43:07.915 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-51539607588 2024-09-18T15:43:07.915 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:43:07.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=51539607588 2024-09-18T15:43:07.916 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 51539607588 2024-09-18T15:43:07.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 51539607588' 2024-09-18T15:43:07.916 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:43:08.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 51539607588 -lt 51539607588 2024-09-18T15:43:08.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:43:08.281 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378653 2024-09-18T15:43:08.281 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:43:08.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:43:08.283 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378653 2024-09-18T15:43:08.283 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:43:08.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378653 2024-09-18T15:43:08.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378653' 2024-09-18T15:43:08.284 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 81604378653 2024-09-18T15:43:08.284 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:43:08.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378653 -lt 81604378653 2024-09-18T15:43:08.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:43:08.688 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-137438953497 2024-09-18T15:43:08.689 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:43:08.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:43:08.690 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-137438953497 2024-09-18T15:43:08.690 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:43:08.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=137438953497 2024-09-18T15:43:08.691 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 137438953497 2024-09-18T15:43:08.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 137438953497' 2024-09-18T15:43:08.692 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:43:09.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 137438953497 -lt 137438953497 2024-09-18T15:43:09.106 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:43:09.106 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:43:09.106 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:43:09.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-18T15:43:09.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:43:09.598 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:43:09.598 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:43:09.598 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:43:09.598 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:43:09.598 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:43:09.598 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:43:09.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-09-18T15:43:09.962 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:43:09.962 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:43:09.962 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:43:10.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-09-18T15:43:10.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:43:10.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:43:10.453 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:147: rados_get_data: ceph pg 2.0 query 2024-09-18T15:43:10.453 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:147: rados_get_data: jq .info.stats.stat_sum.num_objects_repaired 2024-09-18T15:43:10.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:147: rados_get_data: COUNT=3 2024-09-18T15:43:10.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:148: rados_get_data: test 3 = 3 2024-09-18T15:43:10.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:149: rados_get_data: flush_pg_stats 2024-09-18T15:43:10.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:43:10.596 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:43:11.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:43:11.001 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:43:11.001 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T15:43:11.001 INFO:tasks.workunit.client.0.smithi047.stderr:3' 2024-09-18T15:43:11.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:43:11.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:43:11.002 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:43:11.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803821 2024-09-18T15:43:11.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803821 2024-09-18T15:43:11.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803821' 2024-09-18T15:43:11.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:43:11.107 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:43:11.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=51539607591 2024-09-18T15:43:11.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 51539607591 2024-09-18T15:43:11.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803821 1-51539607591' 2024-09-18T15:43:11.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:43:11.214 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:43:11.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378656 2024-09-18T15:43:11.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378656 2024-09-18T15:43:11.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803821 1-51539607591 2-81604378656' 2024-09-18T15:43:11.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:43:11.320 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:43:11.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=137438953500 2024-09-18T15:43:11.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 137438953500 2024-09-18T15:43:11.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803821 1-51539607591 2-81604378656 3-137438953500' 2024-09-18T15:43:11.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:43:11.427 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803821 2024-09-18T15:43:11.428 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:43:11.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:43:11.429 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803821 2024-09-18T15:43:11.429 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:43:11.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803821 2024-09-18T15:43:11.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803821' 2024-09-18T15:43:11.430 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 25769803821 2024-09-18T15:43:11.430 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:43:11.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803821 -lt 25769803821 2024-09-18T15:43:11.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:43:11.816 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-51539607591 2024-09-18T15:43:11.816 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:43:11.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:43:11.818 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-51539607591 2024-09-18T15:43:11.818 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:43:11.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=51539607591 2024-09-18T15:43:11.819 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 51539607591 2024-09-18T15:43:11.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 51539607591' 2024-09-18T15:43:11.819 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:43:12.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 51539607589 -lt 51539607591 2024-09-18T15:43:12.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:43:13.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:43:13.185 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:43:13.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 51539607591 -lt 51539607591 2024-09-18T15:43:13.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:43:13.557 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378656 2024-09-18T15:43:13.558 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:43:13.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:43:13.559 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378656 2024-09-18T15:43:13.559 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:43:13.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378656 2024-09-18T15:43:13.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378656' 2024-09-18T15:43:13.560 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 81604378656 2024-09-18T15:43:13.560 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:43:13.956 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378657 -lt 81604378656 2024-09-18T15:43:13.956 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:43:13.957 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-137438953500 2024-09-18T15:43:13.957 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:43:13.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:43:13.958 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-137438953500 2024-09-18T15:43:13.959 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:43:13.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=137438953500 2024-09-18T15:43:13.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 137438953500' 2024-09-18T15:43:13.960 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 137438953500 2024-09-18T15:43:13.960 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:43:14.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 137438953501 -lt 137438953500 2024-09-18T15:43:14.349 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:150: rados_get_data: ceph pg dump --format=json-pretty 2024-09-18T15:43:14.349 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:150: rados_get_data: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-09-18T15:43:14.698 INFO:tasks.workunit.client.0.smithi047.stderr:dumped all 2024-09-18T15:43:14.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:150: rados_get_data: COUNT=4 2024-09-18T15:43:14.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:151: rados_get_data: test 4 = 4 2024-09-18T15:43:14.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:153: rados_get_data: inject_eio rep data pool-rep obj-eio-397939 td/osd-rep-recov-eio.sh 0 2024-09-18T15:43:14.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:43:14.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:43:14.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:43:14.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:43:14.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:43:14.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:43:14.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-eio-397939 2024-09-18T15:43:14.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:43:14.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:43:14.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:43:14.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T15:43:14.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:43:14.712 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-eio-397939 2024-09-18T15:43:14.712 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:43:14.712 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-397939 2024-09-18T15:43:14.712 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-397939 2024-09-18T15:43:14.712 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:43:15.117 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:43:15.118 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:43:15.118 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:43:15.118 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:43:15.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:43:15.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:43:15.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-18T15:43:15.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:43:15.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:43:15.120 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-18T15:43:15.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:43:15.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-18T15:43:15.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:43:15.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:43:15.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:43:15.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:43:15.121 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:43:15.121 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:43:15.121 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:43:15.121 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:43:15.121 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:43:15.121 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:43:15.121 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:43:15.121 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:43:15.122 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:43:15.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:43:15.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:43:15.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:43:15.204 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-18T15:43:15.204 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:43:15.204 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:43:15.205 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:43:15.205 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:43:15.205 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:43:15.205 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:43:15.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:43:15.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.3.asok injectdataerr pool-rep obj-eio-397939 2024-09-18T15:43:15.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:154: rados_get_data: inject_eio rep data pool-rep obj-eio-397939 td/osd-rep-recov-eio.sh 1 2024-09-18T15:43:15.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:43:15.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:43:15.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:43:15.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:43:15.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:43:15.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:43:15.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-eio-397939 2024-09-18T15:43:15.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:43:15.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:43:15.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:43:15.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-18T15:43:15.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:43:15.291 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-eio-397939 2024-09-18T15:43:15.291 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:43:15.291 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-397939 2024-09-18T15:43:15.291 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-397939 2024-09-18T15:43:15.291 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:43:15.651 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:43:15.651 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:43:15.651 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:43:15.651 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:43:15.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:43:15.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:43:15.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-18T15:43:15.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:43:15.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:43:15.652 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-18T15:43:15.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:43:15.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-18T15:43:15.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:43:15.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-18T15:43:15.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:43:15.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:43:15.654 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:43:15.654 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-18T15:43:15.654 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:43:15.654 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:43:15.655 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:43:15.655 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:43:15.655 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:43:15.655 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.1.asok 2024-09-18T15:43:15.655 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:43:15.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:43:15.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:43:15.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:43:15.738 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-18T15:43:15.738 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:43:15.738 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:43:15.738 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:43:15.738 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:43:15.738 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:43:15.738 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.1.asok 2024-09-18T15:43:15.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:43:15.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.1.asok injectdataerr pool-rep obj-eio-397939 2024-09-18T15:43:15.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:155: rados_get_data: inject_eio rep data pool-rep obj-eio-397939 td/osd-rep-recov-eio.sh 2 2024-09-18T15:43:15.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:43:15.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:43:15.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:43:15.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:43:15.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:43:15.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:43:15.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-eio-397939 2024-09-18T15:43:15.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:43:15.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:43:15.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:43:15.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=2 2024-09-18T15:43:15.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:43:15.823 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-eio-397939 2024-09-18T15:43:15.823 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:43:15.823 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-397939 2024-09-18T15:43:15.823 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-397939 2024-09-18T15:43:15.823 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:43:16.220 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:43:16.220 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:43:16.220 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:43:16.220 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:43:16.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:43:16.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:43:16.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=0 2024-09-18T15:43:16.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:43:16.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:43:16.220 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/0/type 2024-09-18T15:43:16.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:43:16.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-09-18T15:43:16.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:43:16.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-09-18T15:43:16.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:43:16.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:43:16.222 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:43:16.223 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-09-18T15:43:16.223 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T15:43:16.223 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T15:43:16.223 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:43:16.223 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:43:16.223 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:43:16.223 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.0.asok 2024-09-18T15:43:16.223 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:43:16.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:43:16.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:43:16.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:43:16.307 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.0 2024-09-18T15:43:16.307 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T15:43:16.307 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T15:43:16.307 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:43:16.307 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:43:16.307 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:43:16.307 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.0.asok 2024-09-18T15:43:16.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:43:16.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.0.asok injectdataerr pool-rep obj-eio-397939 2024-09-18T15:43:16.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:156: rados_get_data: rados_get td/osd-rep-recov-eio.sh pool-rep obj-eio-397939 hang 2024-09-18T15:43:16.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:43:16.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-18T15:43:16.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-eio-397939 2024-09-18T15:43:16.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=hang 2024-09-18T15:43:16.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' hang = fail ']' 2024-09-18T15:43:16.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' hang = hang ']' 2024-09-18T15:43:16.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:98: rados_get: timeout 5 rados --pool pool-rep get obj-eio-397939 td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:43:21.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:99: rados_get: test 124 = 124 2024-09-18T15:43:21.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:100: rados_get: return 2024-09-18T15:43:21.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:158: rados_get_data: wait_for_clean 2024-09-18T15:43:21.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:43:21.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:43:21.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:43:21.398 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:43:21.398 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:43:21.398 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:43:21.398 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:43:21.398 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:43:21.398 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:43:21.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:43:21.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:43:21.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:43:21.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:43:21.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:43:21.496 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:43:21.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:43:21.897 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:43:21.897 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T15:43:21.897 INFO:tasks.workunit.client.0.smithi047.stderr:3' 2024-09-18T15:43:21.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:43:21.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:43:21.898 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:43:22.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803826 2024-09-18T15:43:22.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803826 2024-09-18T15:43:22.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803826' 2024-09-18T15:43:22.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:43:22.004 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:43:22.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=51539607595 2024-09-18T15:43:22.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 51539607595 2024-09-18T15:43:22.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803826 1-51539607595' 2024-09-18T15:43:22.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:43:22.114 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:43:22.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378660 2024-09-18T15:43:22.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378660 2024-09-18T15:43:22.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803826 1-51539607595 2-81604378660' 2024-09-18T15:43:22.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:43:22.221 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:43:22.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=137438953504 2024-09-18T15:43:22.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 137438953504 2024-09-18T15:43:22.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803826 1-51539607595 2-81604378660 3-137438953504' 2024-09-18T15:43:22.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:43:22.327 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803826 2024-09-18T15:43:22.327 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:43:22.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:43:22.329 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803826 2024-09-18T15:43:22.329 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:43:22.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803826 2024-09-18T15:43:22.330 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 25769803826 2024-09-18T15:43:22.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803826' 2024-09-18T15:43:22.330 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:43:22.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803823 -lt 25769803826 2024-09-18T15:43:22.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:43:23.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:43:23.735 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:43:24.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803826 -lt 25769803826 2024-09-18T15:43:24.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:43:24.144 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-51539607595 2024-09-18T15:43:24.145 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:43:24.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:43:24.146 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-51539607595 2024-09-18T15:43:24.146 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:43:24.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=51539607595 2024-09-18T15:43:24.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 51539607595' 2024-09-18T15:43:24.147 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 51539607595 2024-09-18T15:43:24.148 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:43:24.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 51539607595 -lt 51539607595 2024-09-18T15:43:24.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:43:24.516 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378660 2024-09-18T15:43:24.516 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:43:24.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:43:24.517 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378660 2024-09-18T15:43:24.517 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:43:24.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378660 2024-09-18T15:43:24.521 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 81604378660 2024-09-18T15:43:24.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378660' 2024-09-18T15:43:24.521 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:43:24.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378661 -lt 81604378660 2024-09-18T15:43:24.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:43:24.881 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-137438953504 2024-09-18T15:43:24.882 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:43:24.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:43:24.883 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-137438953504 2024-09-18T15:43:24.883 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:43:24.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=137438953504 2024-09-18T15:43:24.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 137438953504' 2024-09-18T15:43:24.884 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 137438953504 2024-09-18T15:43:24.885 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:43:25.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 137438953505 -lt 137438953504 2024-09-18T15:43:25.306 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:43:25.306 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:43:25.306 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:43:25.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-18T15:43:25.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:43:25.799 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:43:25.799 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:43:25.799 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:43:25.799 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:43:25.799 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:43:25.799 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:43:26.157 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-18T15:43:26.157 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:43:26.158 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:43:26.158 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:43:26.642 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-18T15:43:26.642 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' -1 2024-09-18T15:43:26.642 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T15:43:26.642 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=8 2024-09-18T15:43:26.642 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:43:26.642 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:43:26.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:43:26.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:43:26.743 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:43:26.744 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:43:26.744 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:43:26.744 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:43:26.744 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:43:26.744 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:43:27.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-18T15:43:27.100 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:43:27.101 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:43:27.101 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:43:27.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-18T15:43:27.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-18T15:43:27.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:43:27.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:43:27.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:43:27.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:43:27.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:43:27.619 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:43:27.619 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:43:28.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T15:43:28.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T15:43:28.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-09-18T15:43:28.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:43:28.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-18T15:43:28.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:43:28.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:43:28.361 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:43:28.361 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:43:28.361 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:43:28.361 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:43:28.361 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:43:28.361 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:43:28.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-18T15:43:28.721 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:43:28.721 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:43:28.721 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:43:29.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-18T15:43:29.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-18T15:43:29.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:43:29.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:43:29.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:43:29.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:43:29.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:43:29.219 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:43:29.219 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:43:29.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T15:43:29.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T15:43:29.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-09-18T15:43:29.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:43:29.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-18T15:43:30.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:43:30.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:43:30.127 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:43:30.127 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:43:30.127 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:43:30.127 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:43:30.127 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:43:30.127 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:43:30.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-18T15:43:30.485 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:43:30.485 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:43:30.485 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:43:31.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-18T15:43:31.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-18T15:43:31.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:43:31.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:43:31.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:43:31.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:43:31.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:43:31.015 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:43:31.015 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:43:31.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T15:43:31.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T15:43:31.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 13 )) 2024-09-18T15:43:31.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:43:31.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-09-18T15:43:32.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:43:32.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:43:32.357 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:43:32.357 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:43:32.358 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:43:32.358 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:43:32.358 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:43:32.358 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:43:32.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-18T15:43:32.725 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:43:32.725 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:43:32.725 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:43:33.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-18T15:43:33.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-18T15:43:33.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:43:33.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:43:33.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:43:33.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:43:33.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:43:33.219 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:43:33.219 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:43:33.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T15:43:33.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T15:43:33.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 13 )) 2024-09-18T15:43:33.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:43:33.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-09-18T15:43:35.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:43:35.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:43:35.389 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:43:35.389 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:43:35.389 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:43:35.389 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:43:35.389 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:43:35.390 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:43:35.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-18T15:43:35.757 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:43:35.757 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:43:35.757 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:43:36.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-18T15:43:36.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-18T15:43:36.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:43:36.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:43:36.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:43:36.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:43:36.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:43:36.254 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:43:36.254 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:43:36.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T15:43:36.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T15:43:36.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 13 )) 2024-09-18T15:43:36.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:43:36.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-09-18T15:43:39.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:43:39.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:43:39.971 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:43:39.971 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:43:39.972 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:43:39.972 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:43:39.972 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:43:39.972 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:43:40.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-18T15:43:40.341 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:43:40.341 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:43:40.342 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:43:40.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-18T15:43:40.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-18T15:43:40.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:43:40.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:43:40.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:43:40.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:43:40.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:43:40.830 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:43:40.830 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:43:41.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T15:43:41.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T15:43:41.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 13 )) 2024-09-18T15:43:41.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:43:41.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-09-18T15:43:47.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:43:47.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:43:47.732 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:43:47.732 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:43:47.732 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:43:47.733 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:43:47.733 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:43:47.733 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:43:48.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-18T15:43:48.108 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:43:48.108 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:43:48.108 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:43:48.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-18T15:43:48.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-18T15:43:48.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:43:48.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:43:48.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:43:48.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:43:48.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:43:48.623 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:43:48.623 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:43:49.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T15:43:49.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T15:43:49.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 13 )) 2024-09-18T15:43:49.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:43:49.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 12.8 2024-09-18T15:44:01.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:44:01.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:44:01.915 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:44:01.915 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:44:01.915 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:44:01.915 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:44:01.916 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:44:01.916 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:44:02.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-18T15:44:02.281 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:44:02.281 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:44:02.281 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:44:02.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-18T15:44:02.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-18T15:44:02.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:44:02.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:44:02.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:44:02.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:44:02.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:44:02.805 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:44:02.805 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:44:03.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T15:44:03.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T15:44:03.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 13 )) 2024-09-18T15:44:03.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:44:03.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 15 2024-09-18T15:44:18.343 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:44:18.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:44:18.344 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:44:18.344 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:44:18.344 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:44:18.344 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:44:18.344 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:44:18.344 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:44:18.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-18T15:44:18.710 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:44:18.710 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:44:18.710 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:44:19.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-18T15:44:19.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-18T15:44:19.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:44:19.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:44:19.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:44:19.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:44:19.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:44:19.203 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:44:19.203 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:44:19.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T15:44:19.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T15:44:19.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 9 >= 13 )) 2024-09-18T15:44:19.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:44:19.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 15 2024-09-18T15:44:34.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:44:34.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:44:34.706 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:44:34.706 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:44:34.706 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:44:34.706 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:44:34.706 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:44:34.706 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:44:35.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-18T15:44:35.076 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:44:35.076 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:44:35.076 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:44:35.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-18T15:44:35.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-18T15:44:35.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:44:35.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:44:35.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:44:35.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:44:35.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:44:35.605 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:44:35.605 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:44:36.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T15:44:36.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T15:44:36.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 10 >= 13 )) 2024-09-18T15:44:36.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:44:36.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 15 2024-09-18T15:44:51.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:44:51.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:44:51.098 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:44:51.098 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:44:51.098 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:44:51.098 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:44:51.098 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:44:51.098 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:44:51.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-18T15:44:51.465 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:44:51.466 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:44:51.466 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:44:51.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-18T15:44:51.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-18T15:44:51.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:44:51.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:44:51.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:44:51.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:44:51.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:44:51.971 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:44:51.971 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:44:52.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T15:44:52.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T15:44:52.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 11 >= 13 )) 2024-09-18T15:44:52.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:44:52.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 15 2024-09-18T15:45:07.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:45:07.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:45:07.465 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:45:07.465 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:45:07.466 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:45:07.466 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:45:07.466 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:45:07.466 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:45:07.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-18T15:45:07.830 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:45:07.831 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:45:07.831 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:45:08.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-18T15:45:08.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-18T15:45:08.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:45:08.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:45:08.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:45:08.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:45:08.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:45:08.324 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:45:08.324 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:45:08.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T15:45:08.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T15:45:08.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 12 >= 13 )) 2024-09-18T15:45:08.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:45:08.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 4.5 2024-09-18T15:45:13.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:45:13.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:45:13.318 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:45:13.318 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:45:13.318 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:45:13.318 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:45:13.318 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:45:13.319 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:45:13.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-18T15:45:13.713 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:45:13.714 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:45:13.714 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:45:14.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-18T15:45:14.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-18T15:45:14.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:45:14.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:45:14.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:45:14.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:45:14.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:45:14.202 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:45:14.202 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:45:14.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T15:45:14.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T15:45:14.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 13 >= 13 )) 2024-09-18T15:45:14.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-18T15:45:15.301 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:45:15.301 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_fingerprint": "0787f9dc-eda6-4b28-89af-80fcefc51ff5", 2024-09-18T15:45:15.301 INFO:tasks.workunit.client.0.smithi047.stdout: "version": "17.2.7-1645-g7061ea6c", 2024-09-18T15:45:15.301 INFO:tasks.workunit.client.0.smithi047.stdout: "commit": "7061ea6ca50908d104e9f3979dc615e118d9e153", 2024-09-18T15:45:15.301 INFO:tasks.workunit.client.0.smithi047.stdout: "timestamp": "2024-09-18T15:45:15.299689+0000", 2024-09-18T15:45:15.301 INFO:tasks.workunit.client.0.smithi047.stdout: "tag": "", 2024-09-18T15:45:15.301 INFO:tasks.workunit.client.0.smithi047.stdout: "health": { 2024-09-18T15:45:15.301 INFO:tasks.workunit.client.0.smithi047.stdout: "status": "HEALTH_ERR", 2024-09-18T15:45:15.301 INFO:tasks.workunit.client.0.smithi047.stdout: "checks": { 2024-09-18T15:45:15.302 INFO:tasks.workunit.client.0.smithi047.stdout: "OBJECT_UNFOUND": { 2024-09-18T15:45:15.302 INFO:tasks.workunit.client.0.smithi047.stdout: "severity": "HEALTH_WARN", 2024-09-18T15:45:15.302 INFO:tasks.workunit.client.0.smithi047.stdout: "summary": { 2024-09-18T15:45:15.302 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "1/1 objects unfound (100.000%)", 2024-09-18T15:45:15.302 INFO:tasks.workunit.client.0.smithi047.stdout: "count": 1 2024-09-18T15:45:15.302 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.302 INFO:tasks.workunit.client.0.smithi047.stdout: "detail": [ 2024-09-18T15:45:15.302 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.302 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 2.0 has 1 unfound objects" 2024-09-18T15:45:15.302 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.302 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:45:15.302 INFO:tasks.workunit.client.0.smithi047.stdout: "muted": false 2024-09-18T15:45:15.302 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.303 INFO:tasks.workunit.client.0.smithi047.stdout: "PG_DAMAGED": { 2024-09-18T15:45:15.303 INFO:tasks.workunit.client.0.smithi047.stdout: "severity": "HEALTH_ERR", 2024-09-18T15:45:15.303 INFO:tasks.workunit.client.0.smithi047.stdout: "summary": { 2024-09-18T15:45:15.303 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "Possible data damage: 1 pg recovery_unfound", 2024-09-18T15:45:15.303 INFO:tasks.workunit.client.0.smithi047.stdout: "count": 1 2024-09-18T15:45:15.303 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.303 INFO:tasks.workunit.client.0.smithi047.stdout: "detail": [ 2024-09-18T15:45:15.303 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.303 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 2.0 is active+recovery_unfound+degraded, acting [3,1,0], 1 unfound" 2024-09-18T15:45:15.303 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.303 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:45:15.303 INFO:tasks.workunit.client.0.smithi047.stdout: "muted": false 2024-09-18T15:45:15.303 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.303 INFO:tasks.workunit.client.0.smithi047.stdout: "PG_DEGRADED": { 2024-09-18T15:45:15.304 INFO:tasks.workunit.client.0.smithi047.stdout: "severity": "HEALTH_WARN", 2024-09-18T15:45:15.304 INFO:tasks.workunit.client.0.smithi047.stdout: "summary": { 2024-09-18T15:45:15.304 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "Degraded data redundancy: 3/3 objects degraded (100.000%), 1 pg degraded", 2024-09-18T15:45:15.304 INFO:tasks.workunit.client.0.smithi047.stdout: "count": 1 2024-09-18T15:45:15.304 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.304 INFO:tasks.workunit.client.0.smithi047.stdout: "detail": [ 2024-09-18T15:45:15.304 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.304 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "pg 2.0 is active+recovery_unfound+degraded, acting [3,1,0], 1 unfound" 2024-09-18T15:45:15.304 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.304 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:45:15.304 INFO:tasks.workunit.client.0.smithi047.stdout: "muted": false 2024-09-18T15:45:15.304 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.304 INFO:tasks.workunit.client.0.smithi047.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-18T15:45:15.304 INFO:tasks.workunit.client.0.smithi047.stdout: "severity": "HEALTH_WARN", 2024-09-18T15:45:15.305 INFO:tasks.workunit.client.0.smithi047.stdout: "summary": { 2024-09-18T15:45:15.305 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "2 pool(s) do not have an application enabled", 2024-09-18T15:45:15.305 INFO:tasks.workunit.client.0.smithi047.stdout: "count": 2 2024-09-18T15:45:15.305 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.305 INFO:tasks.workunit.client.0.smithi047.stdout: "detail": [ 2024-09-18T15:45:15.305 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.305 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "application not enabled on pool 'foo'" 2024-09-18T15:45:15.305 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.305 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.305 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "application not enabled on pool 'pool-rep'" 2024-09-18T15:45:15.305 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.305 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.305 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-18T15:45:15.306 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.306 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:45:15.306 INFO:tasks.workunit.client.0.smithi047.stdout: "muted": false 2024-09-18T15:45:15.306 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.306 INFO:tasks.workunit.client.0.smithi047.stdout: "SLOW_OPS": { 2024-09-18T15:45:15.306 INFO:tasks.workunit.client.0.smithi047.stdout: "severity": "HEALTH_WARN", 2024-09-18T15:45:15.306 INFO:tasks.workunit.client.0.smithi047.stdout: "summary": { 2024-09-18T15:45:15.306 INFO:tasks.workunit.client.0.smithi047.stdout: "message": "1 slow ops, oldest one blocked for 115 sec, osd.3 has slow ops", 2024-09-18T15:45:15.306 INFO:tasks.workunit.client.0.smithi047.stdout: "count": 1 2024-09-18T15:45:15.306 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.306 INFO:tasks.workunit.client.0.smithi047.stdout: "detail": [], 2024-09-18T15:45:15.306 INFO:tasks.workunit.client.0.smithi047.stdout: "muted": false 2024-09-18T15:45:15.306 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.306 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.306 INFO:tasks.workunit.client.0.smithi047.stdout: "mutes": [] 2024-09-18T15:45:15.307 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.307 INFO:tasks.workunit.client.0.smithi047.stdout: "monmap_first_committed": 1, 2024-09-18T15:45:15.307 INFO:tasks.workunit.client.0.smithi047.stdout: "monmap_last_committed": 1, 2024-09-18T15:45:15.307 INFO:tasks.workunit.client.0.smithi047.stdout: "monmap": { 2024-09-18T15:45:15.307 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 1, 2024-09-18T15:45:15.307 INFO:tasks.workunit.client.0.smithi047.stdout: "fsid": "19cc9b06-7e46-4b89-943e-ffc293e1c9de", 2024-09-18T15:45:15.307 INFO:tasks.workunit.client.0.smithi047.stdout: "modified": "2024-09-18T15:40:17.807270Z", 2024-09-18T15:45:15.307 INFO:tasks.workunit.client.0.smithi047.stdout: "created": "2024-09-18T15:40:17.807270Z", 2024-09-18T15:45:15.307 INFO:tasks.workunit.client.0.smithi047.stdout: "min_mon_release": 17, 2024-09-18T15:45:15.307 INFO:tasks.workunit.client.0.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-18T15:45:15.307 INFO:tasks.workunit.client.0.smithi047.stdout: "election_strategy": 1, 2024-09-18T15:45:15.307 INFO:tasks.workunit.client.0.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-18T15:45:15.307 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_mode": false, 2024-09-18T15:45:15.307 INFO:tasks.workunit.client.0.smithi047.stdout: "tiebreaker_mon": "", 2024-09-18T15:45:15.308 INFO:tasks.workunit.client.0.smithi047.stdout: "removed_ranks: ": "", 2024-09-18T15:45:15.308 INFO:tasks.workunit.client.0.smithi047.stdout: "features": { 2024-09-18T15:45:15.308 INFO:tasks.workunit.client.0.smithi047.stdout: "persistent": [ 2024-09-18T15:45:15.308 INFO:tasks.workunit.client.0.smithi047.stdout: "kraken", 2024-09-18T15:45:15.308 INFO:tasks.workunit.client.0.smithi047.stdout: "luminous", 2024-09-18T15:45:15.308 INFO:tasks.workunit.client.0.smithi047.stdout: "mimic", 2024-09-18T15:45:15.308 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap-prune", 2024-09-18T15:45:15.308 INFO:tasks.workunit.client.0.smithi047.stdout: "nautilus", 2024-09-18T15:45:15.308 INFO:tasks.workunit.client.0.smithi047.stdout: "octopus", 2024-09-18T15:45:15.308 INFO:tasks.workunit.client.0.smithi047.stdout: "pacific", 2024-09-18T15:45:15.308 INFO:tasks.workunit.client.0.smithi047.stdout: "elector-pinging", 2024-09-18T15:45:15.308 INFO:tasks.workunit.client.0.smithi047.stdout: "quincy" 2024-09-18T15:45:15.308 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:45:15.309 INFO:tasks.workunit.client.0.smithi047.stdout: "optional": [] 2024-09-18T15:45:15.309 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.309 INFO:tasks.workunit.client.0.smithi047.stdout: "mons": [ 2024-09-18T15:45:15.309 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.309 INFO:tasks.workunit.client.0.smithi047.stdout: "rank": 0, 2024-09-18T15:45:15.309 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "a", 2024-09-18T15:45:15.309 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addrs": { 2024-09-18T15:45:15.309 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T15:45:15.309 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.309 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T15:45:15.309 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:7140", 2024-09-18T15:45:15.309 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 0 2024-09-18T15:45:15.309 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.309 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:45:15.309 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.310 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:7140/0", 2024-09-18T15:45:15.310 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addr": "127.0.0.1:7140/0", 2024-09-18T15:45:15.310 INFO:tasks.workunit.client.0.smithi047.stdout: "priority": 0, 2024-09-18T15:45:15.310 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 0, 2024-09-18T15:45:15.310 INFO:tasks.workunit.client.0.smithi047.stdout: "crush_location": "{}" 2024-09-18T15:45:15.310 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.310 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:45:15.310 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.310 INFO:tasks.workunit.client.0.smithi047.stdout: "quorum": [ 2024-09-18T15:45:15.310 INFO:tasks.workunit.client.0.smithi047.stdout: 0 2024-09-18T15:45:15.310 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:45:15.310 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap": { 2024-09-18T15:45:15.310 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 39, 2024-09-18T15:45:15.310 INFO:tasks.workunit.client.0.smithi047.stdout: "fsid": "19cc9b06-7e46-4b89-943e-ffc293e1c9de", 2024-09-18T15:45:15.311 INFO:tasks.workunit.client.0.smithi047.stdout: "created": "2024-09-18T15:40:18.662251+0000", 2024-09-18T15:45:15.311 INFO:tasks.workunit.client.0.smithi047.stdout: "modified": "2024-09-18T15:42:41.284054+0000", 2024-09-18T15:45:15.311 INFO:tasks.workunit.client.0.smithi047.stdout: "last_up_change": "2024-09-18T15:42:32.486155+0000", 2024-09-18T15:45:15.311 INFO:tasks.workunit.client.0.smithi047.stdout: "last_in_change": "2024-09-18T15:41:58.119377+0000", 2024-09-18T15:45:15.311 INFO:tasks.workunit.client.0.smithi047.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-18T15:45:15.311 INFO:tasks.workunit.client.0.smithi047.stdout: "flags_num": 5799936, 2024-09-18T15:45:15.311 INFO:tasks.workunit.client.0.smithi047.stdout: "flags_set": [ 2024-09-18T15:45:15.311 INFO:tasks.workunit.client.0.smithi047.stdout: "pglog_hardlimit", 2024-09-18T15:45:15.311 INFO:tasks.workunit.client.0.smithi047.stdout: "purged_snapdirs", 2024-09-18T15:45:15.311 INFO:tasks.workunit.client.0.smithi047.stdout: "recovery_deletes", 2024-09-18T15:45:15.311 INFO:tasks.workunit.client.0.smithi047.stdout: "sortbitwise" 2024-09-18T15:45:15.311 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:45:15.311 INFO:tasks.workunit.client.0.smithi047.stdout: "crush_version": 9, 2024-09-18T15:45:15.311 INFO:tasks.workunit.client.0.smithi047.stdout: "full_ratio": 0.99000000953674316, 2024-09-18T15:45:15.311 INFO:tasks.workunit.client.0.smithi047.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-18T15:45:15.312 INFO:tasks.workunit.client.0.smithi047.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-18T15:45:15.312 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_snapshot": "", 2024-09-18T15:45:15.312 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_max": 2, 2024-09-18T15:45:15.312 INFO:tasks.workunit.client.0.smithi047.stdout: "max_osd": 4, 2024-09-18T15:45:15.312 INFO:tasks.workunit.client.0.smithi047.stdout: "require_min_compat_client": "luminous", 2024-09-18T15:45:15.312 INFO:tasks.workunit.client.0.smithi047.stdout: "min_compat_client": "luminous", 2024-09-18T15:45:15.312 INFO:tasks.workunit.client.0.smithi047.stdout: "require_osd_release": "quincy", 2024-09-18T15:45:15.312 INFO:tasks.workunit.client.0.smithi047.stdout: "pools": [ 2024-09-18T15:45:15.312 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.313 INFO:tasks.workunit.client.0.smithi047.stdout: "pool": 1, 2024-09-18T15:45:15.313 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_name": "foo", 2024-09-18T15:45:15.313 INFO:tasks.workunit.client.0.smithi047.stdout: "create_time": "2024-09-18T15:40:19.612056+0000", 2024-09-18T15:45:15.313 INFO:tasks.workunit.client.0.smithi047.stdout: "flags": 1, 2024-09-18T15:45:15.313 INFO:tasks.workunit.client.0.smithi047.stdout: "flags_names": "hashpspool", 2024-09-18T15:45:15.313 INFO:tasks.workunit.client.0.smithi047.stdout: "type": 1, 2024-09-18T15:45:15.313 INFO:tasks.workunit.client.0.smithi047.stdout: "size": 3, 2024-09-18T15:45:15.313 INFO:tasks.workunit.client.0.smithi047.stdout: "min_size": 2, 2024-09-18T15:45:15.313 INFO:tasks.workunit.client.0.smithi047.stdout: "crush_rule": 0, 2024-09-18T15:45:15.313 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_count": 0, 2024-09-18T15:45:15.313 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_target": 0, 2024-09-18T15:45:15.313 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T15:45:15.313 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T15:45:15.314 INFO:tasks.workunit.client.0.smithi047.stdout: "object_hash": 2, 2024-09-18T15:45:15.314 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_autoscale_mode": "off", 2024-09-18T15:45:15.314 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_num": 8, 2024-09-18T15:45:15.314 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_placement_num": 8, 2024-09-18T15:45:15.314 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_placement_num_target": 8, 2024-09-18T15:45:15.314 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_num_target": 8, 2024-09-18T15:45:15.314 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_num_pending": 8, 2024-09-18T15:45:15.314 INFO:tasks.workunit.client.0.smithi047.stdout: "last_pg_merge_meta": { 2024-09-18T15:45:15.314 INFO:tasks.workunit.client.0.smithi047.stdout: "source_pgid": "0.0", 2024-09-18T15:45:15.314 INFO:tasks.workunit.client.0.smithi047.stdout: "ready_epoch": 0, 2024-09-18T15:45:15.314 INFO:tasks.workunit.client.0.smithi047.stdout: "last_epoch_started": 0, 2024-09-18T15:45:15.314 INFO:tasks.workunit.client.0.smithi047.stdout: "last_epoch_clean": 0, 2024-09-18T15:45:15.314 INFO:tasks.workunit.client.0.smithi047.stdout: "source_version": "0'0", 2024-09-18T15:45:15.314 INFO:tasks.workunit.client.0.smithi047.stdout: "target_version": "0'0" 2024-09-18T15:45:15.315 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.315 INFO:tasks.workunit.client.0.smithi047.stdout: "last_change": "8", 2024-09-18T15:45:15.315 INFO:tasks.workunit.client.0.smithi047.stdout: "last_force_op_resend": "0", 2024-09-18T15:45:15.315 INFO:tasks.workunit.client.0.smithi047.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T15:45:15.315 INFO:tasks.workunit.client.0.smithi047.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T15:45:15.315 INFO:tasks.workunit.client.0.smithi047.stdout: "auid": 0, 2024-09-18T15:45:15.315 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_mode": "selfmanaged", 2024-09-18T15:45:15.315 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_seq": 0, 2024-09-18T15:45:15.315 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_epoch": 0, 2024-09-18T15:45:15.315 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_snaps": [], 2024-09-18T15:45:15.315 INFO:tasks.workunit.client.0.smithi047.stdout: "removed_snaps": "[]", 2024-09-18T15:45:15.315 INFO:tasks.workunit.client.0.smithi047.stdout: "quota_max_bytes": 0, 2024-09-18T15:45:15.316 INFO:tasks.workunit.client.0.smithi047.stdout: "quota_max_objects": 0, 2024-09-18T15:45:15.316 INFO:tasks.workunit.client.0.smithi047.stdout: "tiers": [], 2024-09-18T15:45:15.316 INFO:tasks.workunit.client.0.smithi047.stdout: "tier_of": -1, 2024-09-18T15:45:15.316 INFO:tasks.workunit.client.0.smithi047.stdout: "read_tier": -1, 2024-09-18T15:45:15.316 INFO:tasks.workunit.client.0.smithi047.stdout: "write_tier": -1, 2024-09-18T15:45:15.316 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_mode": "none", 2024-09-18T15:45:15.316 INFO:tasks.workunit.client.0.smithi047.stdout: "target_max_bytes": 0, 2024-09-18T15:45:15.316 INFO:tasks.workunit.client.0.smithi047.stdout: "target_max_objects": 0, 2024-09-18T15:45:15.316 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T15:45:15.316 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T15:45:15.316 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T15:45:15.316 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_min_flush_age": 0, 2024-09-18T15:45:15.316 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_min_evict_age": 0, 2024-09-18T15:45:15.317 INFO:tasks.workunit.client.0.smithi047.stdout: "erasure_code_profile": "", 2024-09-18T15:45:15.317 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_params": { 2024-09-18T15:45:15.317 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "none" 2024-09-18T15:45:15.317 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.317 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_period": 0, 2024-09-18T15:45:15.317 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_count": 0, 2024-09-18T15:45:15.317 INFO:tasks.workunit.client.0.smithi047.stdout: "use_gmt_hitset": true, 2024-09-18T15:45:15.317 INFO:tasks.workunit.client.0.smithi047.stdout: "min_read_recency_for_promote": 0, 2024-09-18T15:45:15.317 INFO:tasks.workunit.client.0.smithi047.stdout: "min_write_recency_for_promote": 0, 2024-09-18T15:45:15.317 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T15:45:15.317 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_search_last_n": 0, 2024-09-18T15:45:15.317 INFO:tasks.workunit.client.0.smithi047.stdout: "grade_table": [], 2024-09-18T15:45:15.317 INFO:tasks.workunit.client.0.smithi047.stdout: "stripe_width": 0, 2024-09-18T15:45:15.317 INFO:tasks.workunit.client.0.smithi047.stdout: "expected_num_objects": 0, 2024-09-18T15:45:15.318 INFO:tasks.workunit.client.0.smithi047.stdout: "fast_read": false, 2024-09-18T15:45:15.318 INFO:tasks.workunit.client.0.smithi047.stdout: "options": {}, 2024-09-18T15:45:15.318 INFO:tasks.workunit.client.0.smithi047.stdout: "application_metadata": {} 2024-09-18T15:45:15.318 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.318 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.318 INFO:tasks.workunit.client.0.smithi047.stdout: "pool": 2, 2024-09-18T15:45:15.318 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_name": "pool-rep", 2024-09-18T15:45:15.318 INFO:tasks.workunit.client.0.smithi047.stdout: "create_time": "2024-09-18T15:42:39.943703+0000", 2024-09-18T15:45:15.318 INFO:tasks.workunit.client.0.smithi047.stdout: "flags": 1, 2024-09-18T15:45:15.318 INFO:tasks.workunit.client.0.smithi047.stdout: "flags_names": "hashpspool", 2024-09-18T15:45:15.318 INFO:tasks.workunit.client.0.smithi047.stdout: "type": 1, 2024-09-18T15:45:15.318 INFO:tasks.workunit.client.0.smithi047.stdout: "size": 3, 2024-09-18T15:45:15.318 INFO:tasks.workunit.client.0.smithi047.stdout: "min_size": 2, 2024-09-18T15:45:15.318 INFO:tasks.workunit.client.0.smithi047.stdout: "crush_rule": 0, 2024-09-18T15:45:15.318 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_count": 0, 2024-09-18T15:45:15.319 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_target": 0, 2024-09-18T15:45:15.319 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T15:45:15.319 INFO:tasks.workunit.client.0.smithi047.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T15:45:15.319 INFO:tasks.workunit.client.0.smithi047.stdout: "object_hash": 2, 2024-09-18T15:45:15.319 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_autoscale_mode": "off", 2024-09-18T15:45:15.319 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_num": 1, 2024-09-18T15:45:15.319 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_placement_num": 1, 2024-09-18T15:45:15.319 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_placement_num_target": 1, 2024-09-18T15:45:15.319 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_num_target": 1, 2024-09-18T15:45:15.319 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_num_pending": 1, 2024-09-18T15:45:15.319 INFO:tasks.workunit.client.0.smithi047.stdout: "last_pg_merge_meta": { 2024-09-18T15:45:15.319 INFO:tasks.workunit.client.0.smithi047.stdout: "source_pgid": "0.0", 2024-09-18T15:45:15.319 INFO:tasks.workunit.client.0.smithi047.stdout: "ready_epoch": 0, 2024-09-18T15:45:15.319 INFO:tasks.workunit.client.0.smithi047.stdout: "last_epoch_started": 0, 2024-09-18T15:45:15.319 INFO:tasks.workunit.client.0.smithi047.stdout: "last_epoch_clean": 0, 2024-09-18T15:45:15.320 INFO:tasks.workunit.client.0.smithi047.stdout: "source_version": "0'0", 2024-09-18T15:45:15.320 INFO:tasks.workunit.client.0.smithi047.stdout: "target_version": "0'0" 2024-09-18T15:45:15.320 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.320 INFO:tasks.workunit.client.0.smithi047.stdout: "last_change": "39", 2024-09-18T15:45:15.320 INFO:tasks.workunit.client.0.smithi047.stdout: "last_force_op_resend": "0", 2024-09-18T15:45:15.320 INFO:tasks.workunit.client.0.smithi047.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T15:45:15.320 INFO:tasks.workunit.client.0.smithi047.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T15:45:15.320 INFO:tasks.workunit.client.0.smithi047.stdout: "auid": 0, 2024-09-18T15:45:15.320 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_mode": "selfmanaged", 2024-09-18T15:45:15.320 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_seq": 0, 2024-09-18T15:45:15.320 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_epoch": 0, 2024-09-18T15:45:15.320 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_snaps": [], 2024-09-18T15:45:15.320 INFO:tasks.workunit.client.0.smithi047.stdout: "removed_snaps": "[]", 2024-09-18T15:45:15.320 INFO:tasks.workunit.client.0.smithi047.stdout: "quota_max_bytes": 0, 2024-09-18T15:45:15.321 INFO:tasks.workunit.client.0.smithi047.stdout: "quota_max_objects": 0, 2024-09-18T15:45:15.321 INFO:tasks.workunit.client.0.smithi047.stdout: "tiers": [], 2024-09-18T15:45:15.321 INFO:tasks.workunit.client.0.smithi047.stdout: "tier_of": -1, 2024-09-18T15:45:15.321 INFO:tasks.workunit.client.0.smithi047.stdout: "read_tier": -1, 2024-09-18T15:45:15.321 INFO:tasks.workunit.client.0.smithi047.stdout: "write_tier": -1, 2024-09-18T15:45:15.321 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_mode": "none", 2024-09-18T15:45:15.321 INFO:tasks.workunit.client.0.smithi047.stdout: "target_max_bytes": 0, 2024-09-18T15:45:15.321 INFO:tasks.workunit.client.0.smithi047.stdout: "target_max_objects": 0, 2024-09-18T15:45:15.321 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T15:45:15.321 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T15:45:15.321 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T15:45:15.322 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_min_flush_age": 0, 2024-09-18T15:45:15.322 INFO:tasks.workunit.client.0.smithi047.stdout: "cache_min_evict_age": 0, 2024-09-18T15:45:15.322 INFO:tasks.workunit.client.0.smithi047.stdout: "erasure_code_profile": "", 2024-09-18T15:45:15.322 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_params": { 2024-09-18T15:45:15.322 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "none" 2024-09-18T15:45:15.322 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.322 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_period": 0, 2024-09-18T15:45:15.322 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_count": 0, 2024-09-18T15:45:15.322 INFO:tasks.workunit.client.0.smithi047.stdout: "use_gmt_hitset": true, 2024-09-18T15:45:15.322 INFO:tasks.workunit.client.0.smithi047.stdout: "min_read_recency_for_promote": 0, 2024-09-18T15:45:15.323 INFO:tasks.workunit.client.0.smithi047.stdout: "min_write_recency_for_promote": 0, 2024-09-18T15:45:15.323 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T15:45:15.323 INFO:tasks.workunit.client.0.smithi047.stdout: "hit_set_search_last_n": 0, 2024-09-18T15:45:15.323 INFO:tasks.workunit.client.0.smithi047.stdout: "grade_table": [], 2024-09-18T15:45:15.323 INFO:tasks.workunit.client.0.smithi047.stdout: "stripe_width": 0, 2024-09-18T15:45:15.323 INFO:tasks.workunit.client.0.smithi047.stdout: "expected_num_objects": 0, 2024-09-18T15:45:15.323 INFO:tasks.workunit.client.0.smithi047.stdout: "fast_read": false, 2024-09-18T15:45:15.323 INFO:tasks.workunit.client.0.smithi047.stdout: "options": {}, 2024-09-18T15:45:15.323 INFO:tasks.workunit.client.0.smithi047.stdout: "application_metadata": {} 2024-09-18T15:45:15.323 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.323 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:45:15.323 INFO:tasks.workunit.client.0.smithi047.stdout: "osds": [ 2024-09-18T15:45:15.323 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.323 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 0, 2024-09-18T15:45:15.323 INFO:tasks.workunit.client.0.smithi047.stdout: "uuid": "b8c05245-ffaa-452f-8a94-090c5b6bf6be", 2024-09-18T15:45:15.324 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 1, 2024-09-18T15:45:15.324 INFO:tasks.workunit.client.0.smithi047.stdout: "in": 1, 2024-09-18T15:45:15.324 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 1, 2024-09-18T15:45:15.324 INFO:tasks.workunit.client.0.smithi047.stdout: "primary_affinity": 1, 2024-09-18T15:45:15.324 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_begin": 0, 2024-09-18T15:45:15.324 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_end": 0, 2024-09-18T15:45:15.324 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 6, 2024-09-18T15:45:15.324 INFO:tasks.workunit.client.0.smithi047.stdout: "up_thru": 34, 2024-09-18T15:45:15.324 INFO:tasks.workunit.client.0.smithi047.stdout: "down_at": 0, 2024-09-18T15:45:15.324 INFO:tasks.workunit.client.0.smithi047.stdout: "lost_at": 0, 2024-09-18T15:45:15.324 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addrs": { 2024-09-18T15:45:15.324 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T15:45:15.324 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.324 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T15:45:15.324 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6802", 2024-09-18T15:45:15.325 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 399058 2024-09-18T15:45:15.325 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.325 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.325 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T15:45:15.325 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6803", 2024-09-18T15:45:15.325 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 399058 2024-09-18T15:45:15.325 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.325 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:45:15.325 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.325 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addrs": { 2024-09-18T15:45:15.325 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T15:45:15.325 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.325 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T15:45:15.325 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6804", 2024-09-18T15:45:15.325 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 399058 2024-09-18T15:45:15.326 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.326 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.326 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T15:45:15.326 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6805", 2024-09-18T15:45:15.326 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 399058 2024-09-18T15:45:15.326 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.326 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:45:15.326 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.326 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addrs": { 2024-09-18T15:45:15.326 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T15:45:15.326 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.326 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T15:45:15.326 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6808", 2024-09-18T15:45:15.326 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 399058 2024-09-18T15:45:15.327 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.327 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.327 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T15:45:15.327 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6809", 2024-09-18T15:45:15.327 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 399058 2024-09-18T15:45:15.327 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.327 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:45:15.327 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.327 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addrs": { 2024-09-18T15:45:15.327 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T15:45:15.327 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.327 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T15:45:15.327 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6806", 2024-09-18T15:45:15.327 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 399058 2024-09-18T15:45:15.328 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.328 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.328 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T15:45:15.328 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6807", 2024-09-18T15:45:15.328 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 399058 2024-09-18T15:45:15.328 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.328 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:45:15.328 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.328 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addr": "127.0.0.1:6803/399058", 2024-09-18T15:45:15.328 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addr": "127.0.0.1:6805/399058", 2024-09-18T15:45:15.328 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addr": "127.0.0.1:6809/399058", 2024-09-18T15:45:15.328 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addr": "127.0.0.1:6807/399058", 2024-09-18T15:45:15.328 INFO:tasks.workunit.client.0.smithi047.stdout: "state": [ 2024-09-18T15:45:15.328 INFO:tasks.workunit.client.0.smithi047.stdout: "exists", 2024-09-18T15:45:15.329 INFO:tasks.workunit.client.0.smithi047.stdout: "up" 2024-09-18T15:45:15.329 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:45:15.329 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.329 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.329 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 1, 2024-09-18T15:45:15.329 INFO:tasks.workunit.client.0.smithi047.stdout: "uuid": "bd0ccc6d-3562-4f16-8161-163dafa47ebf", 2024-09-18T15:45:15.329 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 1, 2024-09-18T15:45:15.329 INFO:tasks.workunit.client.0.smithi047.stdout: "in": 1, 2024-09-18T15:45:15.329 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 1, 2024-09-18T15:45:15.329 INFO:tasks.workunit.client.0.smithi047.stdout: "primary_affinity": 1, 2024-09-18T15:45:15.329 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_begin": 0, 2024-09-18T15:45:15.329 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_end": 0, 2024-09-18T15:45:15.329 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 12, 2024-09-18T15:45:15.329 INFO:tasks.workunit.client.0.smithi047.stdout: "up_thru": 34, 2024-09-18T15:45:15.329 INFO:tasks.workunit.client.0.smithi047.stdout: "down_at": 0, 2024-09-18T15:45:15.330 INFO:tasks.workunit.client.0.smithi047.stdout: "lost_at": 0, 2024-09-18T15:45:15.330 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addrs": { 2024-09-18T15:45:15.330 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T15:45:15.330 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.330 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T15:45:15.330 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6810", 2024-09-18T15:45:15.330 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 400786 2024-09-18T15:45:15.330 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.330 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.330 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T15:45:15.330 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6811", 2024-09-18T15:45:15.330 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 400786 2024-09-18T15:45:15.330 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.330 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:45:15.330 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.331 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addrs": { 2024-09-18T15:45:15.331 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T15:45:15.331 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.331 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T15:45:15.331 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6812", 2024-09-18T15:45:15.331 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 400786 2024-09-18T15:45:15.331 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.331 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.331 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T15:45:15.331 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6813", 2024-09-18T15:45:15.331 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 400786 2024-09-18T15:45:15.331 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.331 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:45:15.331 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.331 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addrs": { 2024-09-18T15:45:15.332 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T15:45:15.332 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.332 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T15:45:15.332 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6816", 2024-09-18T15:45:15.332 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 400786 2024-09-18T15:45:15.332 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.332 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.332 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T15:45:15.332 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6817", 2024-09-18T15:45:15.332 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 400786 2024-09-18T15:45:15.332 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.332 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:45:15.332 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.332 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addrs": { 2024-09-18T15:45:15.333 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T15:45:15.333 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.333 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T15:45:15.333 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6814", 2024-09-18T15:45:15.333 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 400786 2024-09-18T15:45:15.333 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.333 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.333 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T15:45:15.333 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6815", 2024-09-18T15:45:15.333 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 400786 2024-09-18T15:45:15.333 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.333 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:45:15.333 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.333 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addr": "127.0.0.1:6811/400786", 2024-09-18T15:45:15.333 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addr": "127.0.0.1:6813/400786", 2024-09-18T15:45:15.334 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addr": "127.0.0.1:6817/400786", 2024-09-18T15:45:15.334 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addr": "127.0.0.1:6815/400786", 2024-09-18T15:45:15.334 INFO:tasks.workunit.client.0.smithi047.stdout: "state": [ 2024-09-18T15:45:15.334 INFO:tasks.workunit.client.0.smithi047.stdout: "exists", 2024-09-18T15:45:15.334 INFO:tasks.workunit.client.0.smithi047.stdout: "up" 2024-09-18T15:45:15.334 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:45:15.334 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.334 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.334 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 2, 2024-09-18T15:45:15.334 INFO:tasks.workunit.client.0.smithi047.stdout: "uuid": "f3a2ea0b-7193-4b81-b586-b965dec55c24", 2024-09-18T15:45:15.334 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 1, 2024-09-18T15:45:15.334 INFO:tasks.workunit.client.0.smithi047.stdout: "in": 1, 2024-09-18T15:45:15.334 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 1, 2024-09-18T15:45:15.334 INFO:tasks.workunit.client.0.smithi047.stdout: "primary_affinity": 1, 2024-09-18T15:45:15.334 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_begin": 0, 2024-09-18T15:45:15.335 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_end": 0, 2024-09-18T15:45:15.335 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 19, 2024-09-18T15:45:15.335 INFO:tasks.workunit.client.0.smithi047.stdout: "up_thru": 34, 2024-09-18T15:45:15.335 INFO:tasks.workunit.client.0.smithi047.stdout: "down_at": 0, 2024-09-18T15:45:15.335 INFO:tasks.workunit.client.0.smithi047.stdout: "lost_at": 0, 2024-09-18T15:45:15.335 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addrs": { 2024-09-18T15:45:15.335 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T15:45:15.335 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.335 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T15:45:15.335 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6818", 2024-09-18T15:45:15.335 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 402516 2024-09-18T15:45:15.335 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.335 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.335 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T15:45:15.335 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6819", 2024-09-18T15:45:15.335 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 402516 2024-09-18T15:45:15.336 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.336 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:45:15.336 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.336 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addrs": { 2024-09-18T15:45:15.336 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T15:45:15.336 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.336 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T15:45:15.336 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6820", 2024-09-18T15:45:15.336 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 402516 2024-09-18T15:45:15.336 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.336 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.336 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T15:45:15.336 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6821", 2024-09-18T15:45:15.336 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 402516 2024-09-18T15:45:15.337 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.337 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:45:15.337 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.337 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addrs": { 2024-09-18T15:45:15.337 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T15:45:15.337 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.337 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T15:45:15.337 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6824", 2024-09-18T15:45:15.337 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 402516 2024-09-18T15:45:15.337 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.337 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.337 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T15:45:15.337 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6825", 2024-09-18T15:45:15.337 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 402516 2024-09-18T15:45:15.337 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.338 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:45:15.338 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.338 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addrs": { 2024-09-18T15:45:15.338 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T15:45:15.338 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.338 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T15:45:15.338 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6822", 2024-09-18T15:45:15.338 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 402516 2024-09-18T15:45:15.338 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.338 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.338 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T15:45:15.338 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6823", 2024-09-18T15:45:15.338 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 402516 2024-09-18T15:45:15.338 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.338 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:45:15.339 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.339 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addr": "127.0.0.1:6819/402516", 2024-09-18T15:45:15.339 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addr": "127.0.0.1:6821/402516", 2024-09-18T15:45:15.339 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addr": "127.0.0.1:6825/402516", 2024-09-18T15:45:15.339 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addr": "127.0.0.1:6823/402516", 2024-09-18T15:45:15.339 INFO:tasks.workunit.client.0.smithi047.stdout: "state": [ 2024-09-18T15:45:15.339 INFO:tasks.workunit.client.0.smithi047.stdout: "exists", 2024-09-18T15:45:15.339 INFO:tasks.workunit.client.0.smithi047.stdout: "up" 2024-09-18T15:45:15.339 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:45:15.339 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.339 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.339 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 3, 2024-09-18T15:45:15.339 INFO:tasks.workunit.client.0.smithi047.stdout: "uuid": "70534b9c-354d-4434-9368-9c1a1a5e9c06", 2024-09-18T15:45:15.339 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 1, 2024-09-18T15:45:15.339 INFO:tasks.workunit.client.0.smithi047.stdout: "in": 1, 2024-09-18T15:45:15.340 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 1, 2024-09-18T15:45:15.340 INFO:tasks.workunit.client.0.smithi047.stdout: "primary_affinity": 1, 2024-09-18T15:45:15.340 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_begin": 0, 2024-09-18T15:45:15.340 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_end": 0, 2024-09-18T15:45:15.340 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 32, 2024-09-18T15:45:15.340 INFO:tasks.workunit.client.0.smithi047.stdout: "up_thru": 37, 2024-09-18T15:45:15.340 INFO:tasks.workunit.client.0.smithi047.stdout: "down_at": 0, 2024-09-18T15:45:15.340 INFO:tasks.workunit.client.0.smithi047.stdout: "lost_at": 0, 2024-09-18T15:45:15.340 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addrs": { 2024-09-18T15:45:15.340 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T15:45:15.340 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.340 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T15:45:15.340 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6826", 2024-09-18T15:45:15.340 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 404245 2024-09-18T15:45:15.340 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.341 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.341 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T15:45:15.341 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6827", 2024-09-18T15:45:15.341 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 404245 2024-09-18T15:45:15.341 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.341 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:45:15.341 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.341 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addrs": { 2024-09-18T15:45:15.341 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T15:45:15.341 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.341 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T15:45:15.341 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6828", 2024-09-18T15:45:15.341 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 404245 2024-09-18T15:45:15.341 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.342 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.342 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T15:45:15.342 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6829", 2024-09-18T15:45:15.342 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 404245 2024-09-18T15:45:15.342 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.342 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:45:15.342 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.342 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addrs": { 2024-09-18T15:45:15.342 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T15:45:15.342 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.342 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T15:45:15.342 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6832", 2024-09-18T15:45:15.342 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 404245 2024-09-18T15:45:15.342 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.342 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.343 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T15:45:15.343 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6833", 2024-09-18T15:45:15.343 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 404245 2024-09-18T15:45:15.343 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.343 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:45:15.343 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.343 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addrs": { 2024-09-18T15:45:15.343 INFO:tasks.workunit.client.0.smithi047.stdout: "addrvec": [ 2024-09-18T15:45:15.343 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.343 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v2", 2024-09-18T15:45:15.343 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6830", 2024-09-18T15:45:15.343 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 404245 2024-09-18T15:45:15.343 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.343 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.344 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "v1", 2024-09-18T15:45:15.344 INFO:tasks.workunit.client.0.smithi047.stdout: "addr": "127.0.0.1:6831", 2024-09-18T15:45:15.344 INFO:tasks.workunit.client.0.smithi047.stdout: "nonce": 404245 2024-09-18T15:45:15.344 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.344 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:45:15.344 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.344 INFO:tasks.workunit.client.0.smithi047.stdout: "public_addr": "127.0.0.1:6827/404245", 2024-09-18T15:45:15.344 INFO:tasks.workunit.client.0.smithi047.stdout: "cluster_addr": "127.0.0.1:6829/404245", 2024-09-18T15:45:15.344 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_back_addr": "127.0.0.1:6833/404245", 2024-09-18T15:45:15.344 INFO:tasks.workunit.client.0.smithi047.stdout: "heartbeat_front_addr": "127.0.0.1:6831/404245", 2024-09-18T15:45:15.344 INFO:tasks.workunit.client.0.smithi047.stdout: "state": [ 2024-09-18T15:45:15.344 INFO:tasks.workunit.client.0.smithi047.stdout: "exists", 2024-09-18T15:45:15.345 INFO:tasks.workunit.client.0.smithi047.stdout: "up" 2024-09-18T15:45:15.345 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:45:15.345 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.345 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:45:15.345 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_xinfo": [ 2024-09-18T15:45:15.345 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.345 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 0, 2024-09-18T15:45:15.345 INFO:tasks.workunit.client.0.smithi047.stdout: "down_stamp": "0.000000", 2024-09-18T15:45:15.345 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_probability": 0, 2024-09-18T15:45:15.345 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_interval": 0, 2024-09-18T15:45:15.345 INFO:tasks.workunit.client.0.smithi047.stdout: "features": 4540138320759226367, 2024-09-18T15:45:15.345 INFO:tasks.workunit.client.0.smithi047.stdout: "old_weight": 0, 2024-09-18T15:45:15.345 INFO:tasks.workunit.client.0.smithi047.stdout: "last_purged_snaps_scrub": "2024-09-18T15:40:50.193002+0000", 2024-09-18T15:45:15.345 INFO:tasks.workunit.client.0.smithi047.stdout: "dead_epoch": 0 2024-09-18T15:45:15.345 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.346 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.346 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 1, 2024-09-18T15:45:15.346 INFO:tasks.workunit.client.0.smithi047.stdout: "down_stamp": "0.000000", 2024-09-18T15:45:15.346 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_probability": 0, 2024-09-18T15:45:15.346 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_interval": 0, 2024-09-18T15:45:15.346 INFO:tasks.workunit.client.0.smithi047.stdout: "features": 4540138320759226367, 2024-09-18T15:45:15.346 INFO:tasks.workunit.client.0.smithi047.stdout: "old_weight": 0, 2024-09-18T15:45:15.346 INFO:tasks.workunit.client.0.smithi047.stdout: "last_purged_snaps_scrub": "2024-09-18T15:41:22.677167+0000", 2024-09-18T15:45:15.346 INFO:tasks.workunit.client.0.smithi047.stdout: "dead_epoch": 0 2024-09-18T15:45:15.346 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.346 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.346 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 2, 2024-09-18T15:45:15.346 INFO:tasks.workunit.client.0.smithi047.stdout: "down_stamp": "0.000000", 2024-09-18T15:45:15.346 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_probability": 0, 2024-09-18T15:45:15.346 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_interval": 0, 2024-09-18T15:45:15.347 INFO:tasks.workunit.client.0.smithi047.stdout: "features": 4540138320759226367, 2024-09-18T15:45:15.347 INFO:tasks.workunit.client.0.smithi047.stdout: "old_weight": 0, 2024-09-18T15:45:15.347 INFO:tasks.workunit.client.0.smithi047.stdout: "last_purged_snaps_scrub": "2024-09-18T15:41:56.750292+0000", 2024-09-18T15:45:15.347 INFO:tasks.workunit.client.0.smithi047.stdout: "dead_epoch": 0 2024-09-18T15:45:15.347 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.347 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.347 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 3, 2024-09-18T15:45:15.347 INFO:tasks.workunit.client.0.smithi047.stdout: "down_stamp": "0.000000", 2024-09-18T15:45:15.347 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_probability": 0, 2024-09-18T15:45:15.347 INFO:tasks.workunit.client.0.smithi047.stdout: "laggy_interval": 0, 2024-09-18T15:45:15.347 INFO:tasks.workunit.client.0.smithi047.stdout: "features": 4540138320759226367, 2024-09-18T15:45:15.347 INFO:tasks.workunit.client.0.smithi047.stdout: "old_weight": 0, 2024-09-18T15:45:15.347 INFO:tasks.workunit.client.0.smithi047.stdout: "last_purged_snaps_scrub": "2024-09-18T15:42:32.059341+0000", 2024-09-18T15:45:15.347 INFO:tasks.workunit.client.0.smithi047.stdout: "dead_epoch": 0 2024-09-18T15:45:15.348 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.348 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:45:15.348 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_upmap": [], 2024-09-18T15:45:15.348 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_upmap_items": [ 2024-09-18T15:45:15.348 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.348 INFO:tasks.workunit.client.0.smithi047.stdout: "pgid": "1.3", 2024-09-18T15:45:15.348 INFO:tasks.workunit.client.0.smithi047.stdout: "mappings": [ 2024-09-18T15:45:15.348 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.348 INFO:tasks.workunit.client.0.smithi047.stdout: "from": 0, 2024-09-18T15:45:15.348 INFO:tasks.workunit.client.0.smithi047.stdout: "to": 3 2024-09-18T15:45:15.348 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.348 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:45:15.348 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.348 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:45:15.348 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_temp": [], 2024-09-18T15:45:15.349 INFO:tasks.workunit.client.0.smithi047.stdout: "primary_temp": [], 2024-09-18T15:45:15.349 INFO:tasks.workunit.client.0.smithi047.stdout: "blocklist": {}, 2024-09-18T15:45:15.349 INFO:tasks.workunit.client.0.smithi047.stdout: "range_blocklist": {}, 2024-09-18T15:45:15.349 INFO:tasks.workunit.client.0.smithi047.stdout: "erasure_code_profiles": { 2024-09-18T15:45:15.349 INFO:tasks.workunit.client.0.smithi047.stdout: "default": { 2024-09-18T15:45:15.349 INFO:tasks.workunit.client.0.smithi047.stdout: "k": "2", 2024-09-18T15:45:15.349 INFO:tasks.workunit.client.0.smithi047.stdout: "m": "2", 2024-09-18T15:45:15.349 INFO:tasks.workunit.client.0.smithi047.stdout: "plugin": "jerasure", 2024-09-18T15:45:15.349 INFO:tasks.workunit.client.0.smithi047.stdout: "technique": "reed_sol_van" 2024-09-18T15:45:15.349 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.349 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.349 INFO:tasks.workunit.client.0.smithi047.stdout: "removed_snaps_queue": [], 2024-09-18T15:45:15.349 INFO:tasks.workunit.client.0.smithi047.stdout: "new_removed_snaps": [], 2024-09-18T15:45:15.349 INFO:tasks.workunit.client.0.smithi047.stdout: "new_purged_snaps": [], 2024-09-18T15:45:15.349 INFO:tasks.workunit.client.0.smithi047.stdout: "crush_node_flags": {}, 2024-09-18T15:45:15.350 INFO:tasks.workunit.client.0.smithi047.stdout: "device_class_flags": {}, 2024-09-18T15:45:15.350 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_mode": { 2024-09-18T15:45:15.350 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_mode_enabled": false, 2024-09-18T15:45:15.350 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_bucket_count": 0, 2024-09-18T15:45:15.350 INFO:tasks.workunit.client.0.smithi047.stdout: "degraded_stretch_mode": 0, 2024-09-18T15:45:15.350 INFO:tasks.workunit.client.0.smithi047.stdout: "recovering_stretch_mode": 0, 2024-09-18T15:45:15.350 INFO:tasks.workunit.client.0.smithi047.stdout: "stretch_mode_bucket": 0 2024-09-18T15:45:15.350 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.350 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.350 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_metadata": [ 2024-09-18T15:45:15.350 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.350 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T15:45:15.350 INFO:tasks.workunit.client.0.smithi047.stdout: "arch": "x86_64", 2024-09-18T15:45:15.350 INFO:tasks.workunit.client.0.smithi047.stdout: "back_addr": "[v2:127.0.0.1:6804/399058,v1:127.0.0.1:6805/399058]", 2024-09-18T15:45:15.350 INFO:tasks.workunit.client.0.smithi047.stdout: "back_iface": "", 2024-09-18T15:45:15.351 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs": "1", 2024-09-18T15:45:15.351 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_db": "0", 2024-09-18T15:45:15.351 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_wal": "0", 2024-09-18T15:45:15.351 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_single_shared_device": "1", 2024-09-18T15:45:15.351 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_allocation_from_file": "0", 2024-09-18T15:45:15.351 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_access_mode": "file", 2024-09-18T15:45:15.351 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_block_size": "4096", 2024-09-18T15:45:15.351 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_devices": "sda", 2024-09-18T15:45:15.351 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-18T15:45:15.351 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-18T15:45:15.351 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_path": "td/osd-rep-recov-eio.sh/0/block", 2024-09-18T15:45:15.351 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_rotational": "1", 2024-09-18T15:45:15.351 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_size": "107374182400", 2024-09-18T15:45:15.351 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_support_discard": "1", 2024-09-18T15:45:15.352 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_type": "hdd", 2024-09-18T15:45:15.352 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_min_alloc_size": "4096", 2024-09-18T15:45:15.352 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_release": "quincy", 2024-09-18T15:45:15.352 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T15:45:15.352 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_short": "17.2.7-1645-g7061ea6c", 2024-09-18T15:45:15.352 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_when_created": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T15:45:15.352 INFO:tasks.workunit.client.0.smithi047.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-18T15:45:15.352 INFO:tasks.workunit.client.0.smithi047.stdout: "created_at": "2024-09-18T15:40:24.737096Z", 2024-09-18T15:45:15.352 INFO:tasks.workunit.client.0.smithi047.stdout: "default_device_class": "hdd", 2024-09-18T15:45:15.352 INFO:tasks.workunit.client.0.smithi047.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4HN80", 2024-09-18T15:45:15.352 INFO:tasks.workunit.client.0.smithi047.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-18T15:45:15.352 INFO:tasks.workunit.client.0.smithi047.stdout: "devices": "sda", 2024-09-18T15:45:15.352 INFO:tasks.workunit.client.0.smithi047.stdout: "distro": "centos", 2024-09-18T15:45:15.352 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_description": "CentOS Stream 9", 2024-09-18T15:45:15.352 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_version": "9", 2024-09-18T15:45:15.353 INFO:tasks.workunit.client.0.smithi047.stdout: "front_addr": "[v2:127.0.0.1:6802/399058,v1:127.0.0.1:6803/399058]", 2024-09-18T15:45:15.353 INFO:tasks.workunit.client.0.smithi047.stdout: "front_iface": "", 2024-09-18T15:45:15.353 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/399058,v1:127.0.0.1:6809/399058]", 2024-09-18T15:45:15.353 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/399058,v1:127.0.0.1:6807/399058]", 2024-09-18T15:45:15.353 INFO:tasks.workunit.client.0.smithi047.stdout: "hostname": "smithi047", 2024-09-18T15:45:15.353 INFO:tasks.workunit.client.0.smithi047.stdout: "journal_rotational": "1", 2024-09-18T15:45:15.353 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-18T15:45:15.353 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-18T15:45:15.353 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_swap_kb": "0", 2024-09-18T15:45:15.353 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_total_kb": "32493492", 2024-09-18T15:45:15.353 INFO:tasks.workunit.client.0.smithi047.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-18T15:45:15.353 INFO:tasks.workunit.client.0.smithi047.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-18T15:45:15.353 INFO:tasks.workunit.client.0.smithi047.stdout: "os": "Linux", 2024-09-18T15:45:15.353 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_data": "td/osd-rep-recov-eio.sh/0", 2024-09-18T15:45:15.353 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_objectstore": "bluestore", 2024-09-18T15:45:15.354 INFO:tasks.workunit.client.0.smithi047.stdout: "osdspec_affinity": "", 2024-09-18T15:45:15.354 INFO:tasks.workunit.client.0.smithi047.stdout: "rotational": "1" 2024-09-18T15:45:15.354 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.354 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.354 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 1, 2024-09-18T15:45:15.354 INFO:tasks.workunit.client.0.smithi047.stdout: "arch": "x86_64", 2024-09-18T15:45:15.354 INFO:tasks.workunit.client.0.smithi047.stdout: "back_addr": "[v2:127.0.0.1:6812/400786,v1:127.0.0.1:6813/400786]", 2024-09-18T15:45:15.354 INFO:tasks.workunit.client.0.smithi047.stdout: "back_iface": "", 2024-09-18T15:45:15.354 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs": "1", 2024-09-18T15:45:15.354 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_db": "0", 2024-09-18T15:45:15.354 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_wal": "0", 2024-09-18T15:45:15.354 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_single_shared_device": "1", 2024-09-18T15:45:15.354 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_allocation_from_file": "0", 2024-09-18T15:45:15.354 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_access_mode": "file", 2024-09-18T15:45:15.355 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_block_size": "4096", 2024-09-18T15:45:15.355 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_devices": "sda", 2024-09-18T15:45:15.355 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-18T15:45:15.355 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-18T15:45:15.355 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_path": "td/osd-rep-recov-eio.sh/1/block", 2024-09-18T15:45:15.355 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_rotational": "1", 2024-09-18T15:45:15.355 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_size": "107374182400", 2024-09-18T15:45:15.355 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_support_discard": "1", 2024-09-18T15:45:15.355 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_type": "hdd", 2024-09-18T15:45:15.355 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_min_alloc_size": "4096", 2024-09-18T15:45:15.355 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_release": "quincy", 2024-09-18T15:45:15.355 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T15:45:15.355 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_short": "17.2.7-1645-g7061ea6c", 2024-09-18T15:45:15.355 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_when_created": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T15:45:15.355 INFO:tasks.workunit.client.0.smithi047.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-18T15:45:15.356 INFO:tasks.workunit.client.0.smithi047.stdout: "created_at": "2024-09-18T15:40:56.868419Z", 2024-09-18T15:45:15.356 INFO:tasks.workunit.client.0.smithi047.stdout: "default_device_class": "hdd", 2024-09-18T15:45:15.356 INFO:tasks.workunit.client.0.smithi047.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4HN80", 2024-09-18T15:45:15.356 INFO:tasks.workunit.client.0.smithi047.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-18T15:45:15.356 INFO:tasks.workunit.client.0.smithi047.stdout: "devices": "sda", 2024-09-18T15:45:15.356 INFO:tasks.workunit.client.0.smithi047.stdout: "distro": "centos", 2024-09-18T15:45:15.356 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_description": "CentOS Stream 9", 2024-09-18T15:45:15.356 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_version": "9", 2024-09-18T15:45:15.356 INFO:tasks.workunit.client.0.smithi047.stdout: "front_addr": "[v2:127.0.0.1:6810/400786,v1:127.0.0.1:6811/400786]", 2024-09-18T15:45:15.356 INFO:tasks.workunit.client.0.smithi047.stdout: "front_iface": "", 2024-09-18T15:45:15.356 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/400786,v1:127.0.0.1:6817/400786]", 2024-09-18T15:45:15.356 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/400786,v1:127.0.0.1:6815/400786]", 2024-09-18T15:45:15.356 INFO:tasks.workunit.client.0.smithi047.stdout: "hostname": "smithi047", 2024-09-18T15:45:15.356 INFO:tasks.workunit.client.0.smithi047.stdout: "journal_rotational": "1", 2024-09-18T15:45:15.356 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-18T15:45:15.357 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-18T15:45:15.357 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_swap_kb": "0", 2024-09-18T15:45:15.357 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_total_kb": "32493492", 2024-09-18T15:45:15.357 INFO:tasks.workunit.client.0.smithi047.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-18T15:45:15.357 INFO:tasks.workunit.client.0.smithi047.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-18T15:45:15.357 INFO:tasks.workunit.client.0.smithi047.stdout: "os": "Linux", 2024-09-18T15:45:15.357 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_data": "td/osd-rep-recov-eio.sh/1", 2024-09-18T15:45:15.357 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_objectstore": "bluestore", 2024-09-18T15:45:15.357 INFO:tasks.workunit.client.0.smithi047.stdout: "osdspec_affinity": "", 2024-09-18T15:45:15.357 INFO:tasks.workunit.client.0.smithi047.stdout: "rotational": "1" 2024-09-18T15:45:15.357 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.357 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.357 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T15:45:15.357 INFO:tasks.workunit.client.0.smithi047.stdout: "arch": "x86_64", 2024-09-18T15:45:15.357 INFO:tasks.workunit.client.0.smithi047.stdout: "back_addr": "[v2:127.0.0.1:6820/402516,v1:127.0.0.1:6821/402516]", 2024-09-18T15:45:15.358 INFO:tasks.workunit.client.0.smithi047.stdout: "back_iface": "", 2024-09-18T15:45:15.358 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs": "1", 2024-09-18T15:45:15.358 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_db": "0", 2024-09-18T15:45:15.358 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_wal": "0", 2024-09-18T15:45:15.358 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_single_shared_device": "1", 2024-09-18T15:45:15.358 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_allocation_from_file": "0", 2024-09-18T15:45:15.358 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_access_mode": "file", 2024-09-18T15:45:15.358 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_block_size": "4096", 2024-09-18T15:45:15.358 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_devices": "sda", 2024-09-18T15:45:15.358 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-18T15:45:15.358 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-18T15:45:15.358 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_path": "td/osd-rep-recov-eio.sh/2/block", 2024-09-18T15:45:15.358 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_rotational": "1", 2024-09-18T15:45:15.358 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_size": "107374182400", 2024-09-18T15:45:15.359 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_support_discard": "1", 2024-09-18T15:45:15.359 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_type": "hdd", 2024-09-18T15:45:15.359 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_min_alloc_size": "4096", 2024-09-18T15:45:15.359 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_release": "quincy", 2024-09-18T15:45:15.359 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T15:45:15.359 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_short": "17.2.7-1645-g7061ea6c", 2024-09-18T15:45:15.359 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_when_created": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T15:45:15.359 INFO:tasks.workunit.client.0.smithi047.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-18T15:45:15.359 INFO:tasks.workunit.client.0.smithi047.stdout: "created_at": "2024-09-18T15:41:30.085827Z", 2024-09-18T15:45:15.359 INFO:tasks.workunit.client.0.smithi047.stdout: "default_device_class": "hdd", 2024-09-18T15:45:15.359 INFO:tasks.workunit.client.0.smithi047.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4HN80", 2024-09-18T15:45:15.359 INFO:tasks.workunit.client.0.smithi047.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-18T15:45:15.359 INFO:tasks.workunit.client.0.smithi047.stdout: "devices": "sda", 2024-09-18T15:45:15.359 INFO:tasks.workunit.client.0.smithi047.stdout: "distro": "centos", 2024-09-18T15:45:15.360 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_description": "CentOS Stream 9", 2024-09-18T15:45:15.360 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_version": "9", 2024-09-18T15:45:15.360 INFO:tasks.workunit.client.0.smithi047.stdout: "front_addr": "[v2:127.0.0.1:6818/402516,v1:127.0.0.1:6819/402516]", 2024-09-18T15:45:15.360 INFO:tasks.workunit.client.0.smithi047.stdout: "front_iface": "", 2024-09-18T15:45:15.360 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_back_addr": "[v2:127.0.0.1:6824/402516,v1:127.0.0.1:6825/402516]", 2024-09-18T15:45:15.360 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_front_addr": "[v2:127.0.0.1:6822/402516,v1:127.0.0.1:6823/402516]", 2024-09-18T15:45:15.360 INFO:tasks.workunit.client.0.smithi047.stdout: "hostname": "smithi047", 2024-09-18T15:45:15.360 INFO:tasks.workunit.client.0.smithi047.stdout: "journal_rotational": "1", 2024-09-18T15:45:15.360 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-18T15:45:15.360 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-18T15:45:15.360 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_swap_kb": "0", 2024-09-18T15:45:15.360 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_total_kb": "32493492", 2024-09-18T15:45:15.360 INFO:tasks.workunit.client.0.smithi047.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-18T15:45:15.360 INFO:tasks.workunit.client.0.smithi047.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-18T15:45:15.360 INFO:tasks.workunit.client.0.smithi047.stdout: "os": "Linux", 2024-09-18T15:45:15.361 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_data": "td/osd-rep-recov-eio.sh/2", 2024-09-18T15:45:15.361 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_objectstore": "bluestore", 2024-09-18T15:45:15.361 INFO:tasks.workunit.client.0.smithi047.stdout: "osdspec_affinity": "", 2024-09-18T15:45:15.361 INFO:tasks.workunit.client.0.smithi047.stdout: "rotational": "1" 2024-09-18T15:45:15.361 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.361 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.361 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 3, 2024-09-18T15:45:15.361 INFO:tasks.workunit.client.0.smithi047.stdout: "arch": "x86_64", 2024-09-18T15:45:15.361 INFO:tasks.workunit.client.0.smithi047.stdout: "back_addr": "[v2:127.0.0.1:6828/404245,v1:127.0.0.1:6829/404245]", 2024-09-18T15:45:15.361 INFO:tasks.workunit.client.0.smithi047.stdout: "back_iface": "", 2024-09-18T15:45:15.361 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs": "1", 2024-09-18T15:45:15.361 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_db": "0", 2024-09-18T15:45:15.362 INFO:tasks.workunit.client.0.smithi047.stderr:report 1840512297 2024-09-18T15:45:15.362 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_dedicated_wal": "0", 2024-09-18T15:45:15.362 INFO:tasks.workunit.client.0.smithi047.stdout: "bluefs_single_shared_device": "1", 2024-09-18T15:45:15.362 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_allocation_from_file": "0", 2024-09-18T15:45:15.362 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_access_mode": "file", 2024-09-18T15:45:15.363 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_block_size": "4096", 2024-09-18T15:45:15.363 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_devices": "sda", 2024-09-18T15:45:15.363 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-18T15:45:15.363 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-18T15:45:15.363 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_path": "td/osd-rep-recov-eio.sh/3/block", 2024-09-18T15:45:15.363 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_rotational": "1", 2024-09-18T15:45:15.363 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_size": "107374182400", 2024-09-18T15:45:15.363 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_support_discard": "1", 2024-09-18T15:45:15.363 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_bdev_type": "hdd", 2024-09-18T15:45:15.363 INFO:tasks.workunit.client.0.smithi047.stdout: "bluestore_min_alloc_size": "4096", 2024-09-18T15:45:15.363 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_release": "quincy", 2024-09-18T15:45:15.363 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T15:45:15.363 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_short": "17.2.7-1645-g7061ea6c", 2024-09-18T15:45:15.363 INFO:tasks.workunit.client.0.smithi047.stdout: "ceph_version_when_created": "ceph version 17.2.7-1645-g7061ea6c (7061ea6ca50908d104e9f3979dc615e118d9e153) quincy (stable)", 2024-09-18T15:45:15.364 INFO:tasks.workunit.client.0.smithi047.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-18T15:45:15.364 INFO:tasks.workunit.client.0.smithi047.stdout: "created_at": "2024-09-18T15:42:03.935533Z", 2024-09-18T15:45:15.364 INFO:tasks.workunit.client.0.smithi047.stdout: "default_device_class": "hdd", 2024-09-18T15:45:15.364 INFO:tasks.workunit.client.0.smithi047.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W4HN80", 2024-09-18T15:45:15.364 INFO:tasks.workunit.client.0.smithi047.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-18T15:45:15.364 INFO:tasks.workunit.client.0.smithi047.stdout: "devices": "sda", 2024-09-18T15:45:15.364 INFO:tasks.workunit.client.0.smithi047.stdout: "distro": "centos", 2024-09-18T15:45:15.364 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_description": "CentOS Stream 9", 2024-09-18T15:45:15.364 INFO:tasks.workunit.client.0.smithi047.stdout: "distro_version": "9", 2024-09-18T15:45:15.364 INFO:tasks.workunit.client.0.smithi047.stdout: "front_addr": "[v2:127.0.0.1:6826/404245,v1:127.0.0.1:6827/404245]", 2024-09-18T15:45:15.364 INFO:tasks.workunit.client.0.smithi047.stdout: "front_iface": "", 2024-09-18T15:45:15.364 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_back_addr": "[v2:127.0.0.1:6832/404245,v1:127.0.0.1:6833/404245]", 2024-09-18T15:45:15.364 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_front_addr": "[v2:127.0.0.1:6830/404245,v1:127.0.0.1:6831/404245]", 2024-09-18T15:45:15.364 INFO:tasks.workunit.client.0.smithi047.stdout: "hostname": "smithi047", 2024-09-18T15:45:15.364 INFO:tasks.workunit.client.0.smithi047.stdout: "journal_rotational": "1", 2024-09-18T15:45:15.365 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-18T15:45:15.365 INFO:tasks.workunit.client.0.smithi047.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-18T15:45:15.365 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_swap_kb": "0", 2024-09-18T15:45:15.365 INFO:tasks.workunit.client.0.smithi047.stdout: "mem_total_kb": "32493492", 2024-09-18T15:45:15.365 INFO:tasks.workunit.client.0.smithi047.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-18T15:45:15.365 INFO:tasks.workunit.client.0.smithi047.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-18T15:45:15.365 INFO:tasks.workunit.client.0.smithi047.stdout: "os": "Linux", 2024-09-18T15:45:15.365 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_data": "td/osd-rep-recov-eio.sh/3", 2024-09-18T15:45:15.365 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_objectstore": "bluestore", 2024-09-18T15:45:15.365 INFO:tasks.workunit.client.0.smithi047.stdout: "osdspec_affinity": "", 2024-09-18T15:45:15.365 INFO:tasks.workunit.client.0.smithi047.stdout: "rotational": "1" 2024-09-18T15:45:15.365 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.365 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:45:15.365 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap_clean_epochs": { 2024-09-18T15:45:15.366 INFO:tasks.workunit.client.0.smithi047.stdout: "min_last_epoch_clean": 0, 2024-09-18T15:45:15.366 INFO:tasks.workunit.client.0.smithi047.stdout: "last_epoch_clean": { 2024-09-18T15:45:15.366 INFO:tasks.workunit.client.0.smithi047.stdout: "per_pool": [ 2024-09-18T15:45:15.366 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.366 INFO:tasks.workunit.client.0.smithi047.stdout: "poolid": 1, 2024-09-18T15:45:15.366 INFO:tasks.workunit.client.0.smithi047.stdout: "floor": 4294967295 2024-09-18T15:45:15.366 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.366 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:45:15.366 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.366 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_epochs": [ 2024-09-18T15:45:15.366 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.366 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T15:45:15.366 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 7 2024-09-18T15:45:15.366 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.366 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.367 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 1, 2024-09-18T15:45:15.367 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 13 2024-09-18T15:45:15.367 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.367 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.367 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T15:45:15.367 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 19 2024-09-18T15:45:15.367 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.367 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.367 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 3, 2024-09-18T15:45:15.367 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 33 2024-09-18T15:45:15.367 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.367 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:45:15.367 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.367 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap_first_committed": 1, 2024-09-18T15:45:15.367 INFO:tasks.workunit.client.0.smithi047.stdout: "osdmap_last_committed": 39, 2024-09-18T15:45:15.368 INFO:tasks.workunit.client.0.smithi047.stdout: "crushmap": { 2024-09-18T15:45:15.368 INFO:tasks.workunit.client.0.smithi047.stdout: "devices": [ 2024-09-18T15:45:15.368 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.368 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T15:45:15.368 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "osd.0", 2024-09-18T15:45:15.368 INFO:tasks.workunit.client.0.smithi047.stdout: "class": "hdd" 2024-09-18T15:45:15.368 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.368 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.368 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 1, 2024-09-18T15:45:15.368 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "osd.1", 2024-09-18T15:45:15.368 INFO:tasks.workunit.client.0.smithi047.stdout: "class": "hdd" 2024-09-18T15:45:15.368 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.368 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.368 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T15:45:15.369 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "osd.2", 2024-09-18T15:45:15.369 INFO:tasks.workunit.client.0.smithi047.stdout: "class": "hdd" 2024-09-18T15:45:15.369 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.369 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.369 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 3, 2024-09-18T15:45:15.369 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "osd.3", 2024-09-18T15:45:15.369 INFO:tasks.workunit.client.0.smithi047.stdout: "class": "hdd" 2024-09-18T15:45:15.369 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.369 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:45:15.369 INFO:tasks.workunit.client.0.smithi047.stdout: "types": [ 2024-09-18T15:45:15.369 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.369 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 0, 2024-09-18T15:45:15.369 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "osd" 2024-09-18T15:45:15.369 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.369 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.370 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 1, 2024-09-18T15:45:15.370 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "host" 2024-09-18T15:45:15.370 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.370 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.370 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 2, 2024-09-18T15:45:15.370 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "chassis" 2024-09-18T15:45:15.370 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.370 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.370 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 3, 2024-09-18T15:45:15.370 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "rack" 2024-09-18T15:45:15.370 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.370 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.370 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 4, 2024-09-18T15:45:15.370 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "row" 2024-09-18T15:45:15.370 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.371 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.371 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 5, 2024-09-18T15:45:15.371 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "pdu" 2024-09-18T15:45:15.371 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.371 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.371 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 6, 2024-09-18T15:45:15.371 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "pod" 2024-09-18T15:45:15.371 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.371 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.371 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 7, 2024-09-18T15:45:15.371 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "room" 2024-09-18T15:45:15.371 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.371 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.371 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 8, 2024-09-18T15:45:15.372 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "datacenter" 2024-09-18T15:45:15.372 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.372 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.372 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 9, 2024-09-18T15:45:15.372 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "zone" 2024-09-18T15:45:15.372 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.372 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.372 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 10, 2024-09-18T15:45:15.372 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "region" 2024-09-18T15:45:15.372 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.372 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.372 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 11, 2024-09-18T15:45:15.372 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "root" 2024-09-18T15:45:15.372 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.373 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:45:15.373 INFO:tasks.workunit.client.0.smithi047.stdout: "buckets": [ 2024-09-18T15:45:15.373 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.373 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -1, 2024-09-18T15:45:15.373 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "default", 2024-09-18T15:45:15.373 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 11, 2024-09-18T15:45:15.373 INFO:tasks.workunit.client.0.smithi047.stdout: "type_name": "root", 2024-09-18T15:45:15.373 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 25608, 2024-09-18T15:45:15.373 INFO:tasks.workunit.client.0.smithi047.stdout: "alg": "straw2", 2024-09-18T15:45:15.373 INFO:tasks.workunit.client.0.smithi047.stdout: "hash": "rjenkins1", 2024-09-18T15:45:15.373 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:45:15.373 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.373 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -3, 2024-09-18T15:45:15.373 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 25608, 2024-09-18T15:45:15.373 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 0 2024-09-18T15:45:15.374 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.374 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:45:15.374 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.374 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.374 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -2, 2024-09-18T15:45:15.374 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "default~hdd", 2024-09-18T15:45:15.374 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 11, 2024-09-18T15:45:15.374 INFO:tasks.workunit.client.0.smithi047.stdout: "type_name": "root", 2024-09-18T15:45:15.374 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 25608, 2024-09-18T15:45:15.374 INFO:tasks.workunit.client.0.smithi047.stdout: "alg": "straw2", 2024-09-18T15:45:15.374 INFO:tasks.workunit.client.0.smithi047.stdout: "hash": "rjenkins1", 2024-09-18T15:45:15.374 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:45:15.374 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.374 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -4, 2024-09-18T15:45:15.374 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 25608, 2024-09-18T15:45:15.375 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 0 2024-09-18T15:45:15.375 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.375 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:45:15.375 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.375 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.375 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -3, 2024-09-18T15:45:15.375 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "smithi047", 2024-09-18T15:45:15.375 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 1, 2024-09-18T15:45:15.375 INFO:tasks.workunit.client.0.smithi047.stdout: "type_name": "host", 2024-09-18T15:45:15.375 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 25608, 2024-09-18T15:45:15.375 INFO:tasks.workunit.client.0.smithi047.stdout: "alg": "straw2", 2024-09-18T15:45:15.375 INFO:tasks.workunit.client.0.smithi047.stdout: "hash": "rjenkins1", 2024-09-18T15:45:15.375 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:45:15.375 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.376 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T15:45:15.376 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T15:45:15.376 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 0 2024-09-18T15:45:15.376 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.376 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.376 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 1, 2024-09-18T15:45:15.376 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T15:45:15.376 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 1 2024-09-18T15:45:15.376 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.376 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.376 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T15:45:15.376 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T15:45:15.376 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 2 2024-09-18T15:45:15.376 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.376 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.377 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 3, 2024-09-18T15:45:15.377 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T15:45:15.377 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 3 2024-09-18T15:45:15.377 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.377 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:45:15.377 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.377 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.377 INFO:tasks.workunit.client.0.smithi047.stdout: "id": -4, 2024-09-18T15:45:15.377 INFO:tasks.workunit.client.0.smithi047.stdout: "name": "smithi047~hdd", 2024-09-18T15:45:15.377 INFO:tasks.workunit.client.0.smithi047.stdout: "type_id": 1, 2024-09-18T15:45:15.377 INFO:tasks.workunit.client.0.smithi047.stdout: "type_name": "host", 2024-09-18T15:45:15.377 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 25608, 2024-09-18T15:45:15.377 INFO:tasks.workunit.client.0.smithi047.stdout: "alg": "straw2", 2024-09-18T15:45:15.377 INFO:tasks.workunit.client.0.smithi047.stdout: "hash": "rjenkins1", 2024-09-18T15:45:15.377 INFO:tasks.workunit.client.0.smithi047.stdout: "items": [ 2024-09-18T15:45:15.378 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.378 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 0, 2024-09-18T15:45:15.378 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T15:45:15.378 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 0 2024-09-18T15:45:15.378 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.378 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.378 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 1, 2024-09-18T15:45:15.378 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T15:45:15.378 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 1 2024-09-18T15:45:15.378 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.378 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.378 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 2, 2024-09-18T15:45:15.378 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T15:45:15.378 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 2 2024-09-18T15:45:15.379 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.379 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.379 INFO:tasks.workunit.client.0.smithi047.stdout: "id": 3, 2024-09-18T15:45:15.379 INFO:tasks.workunit.client.0.smithi047.stdout: "weight": 6402, 2024-09-18T15:45:15.379 INFO:tasks.workunit.client.0.smithi047.stdout: "pos": 3 2024-09-18T15:45:15.379 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.379 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:45:15.379 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.379 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:45:15.379 INFO:tasks.workunit.client.0.smithi047.stdout: "rules": [ 2024-09-18T15:45:15.379 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.379 INFO:tasks.workunit.client.0.smithi047.stdout: "rule_id": 0, 2024-09-18T15:45:15.379 INFO:tasks.workunit.client.0.smithi047.stdout: "rule_name": "replicated_rule", 2024-09-18T15:45:15.379 INFO:tasks.workunit.client.0.smithi047.stdout: "type": 1, 2024-09-18T15:45:15.379 INFO:tasks.workunit.client.0.smithi047.stdout: "steps": [ 2024-09-18T15:45:15.380 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.380 INFO:tasks.workunit.client.0.smithi047.stdout: "op": "take", 2024-09-18T15:45:15.380 INFO:tasks.workunit.client.0.smithi047.stdout: "item": -1, 2024-09-18T15:45:15.380 INFO:tasks.workunit.client.0.smithi047.stdout: "item_name": "default" 2024-09-18T15:45:15.380 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.380 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.380 INFO:tasks.workunit.client.0.smithi047.stdout: "op": "choose_firstn", 2024-09-18T15:45:15.380 INFO:tasks.workunit.client.0.smithi047.stdout: "num": 0, 2024-09-18T15:45:15.380 INFO:tasks.workunit.client.0.smithi047.stdout: "type": "osd" 2024-09-18T15:45:15.380 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.380 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.380 INFO:tasks.workunit.client.0.smithi047.stdout: "op": "emit" 2024-09-18T15:45:15.380 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.380 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:45:15.381 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.381 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:45:15.381 INFO:tasks.workunit.client.0.smithi047.stdout: "tunables": { 2024-09-18T15:45:15.381 INFO:tasks.workunit.client.0.smithi047.stdout: "choose_local_tries": 0, 2024-09-18T15:45:15.381 INFO:tasks.workunit.client.0.smithi047.stdout: "choose_local_fallback_tries": 0, 2024-09-18T15:45:15.381 INFO:tasks.workunit.client.0.smithi047.stdout: "choose_total_tries": 50, 2024-09-18T15:45:15.381 INFO:tasks.workunit.client.0.smithi047.stdout: "chooseleaf_descend_once": 1, 2024-09-18T15:45:15.381 INFO:tasks.workunit.client.0.smithi047.stdout: "chooseleaf_vary_r": 1, 2024-09-18T15:45:15.381 INFO:tasks.workunit.client.0.smithi047.stdout: "chooseleaf_stable": 1, 2024-09-18T15:45:15.381 INFO:tasks.workunit.client.0.smithi047.stdout: "straw_calc_version": 1, 2024-09-18T15:45:15.381 INFO:tasks.workunit.client.0.smithi047.stdout: "allowed_bucket_algs": 54, 2024-09-18T15:45:15.381 INFO:tasks.workunit.client.0.smithi047.stdout: "profile": "jewel", 2024-09-18T15:45:15.381 INFO:tasks.workunit.client.0.smithi047.stdout: "optimal_tunables": 1, 2024-09-18T15:45:15.381 INFO:tasks.workunit.client.0.smithi047.stdout: "legacy_tunables": 0, 2024-09-18T15:45:15.381 INFO:tasks.workunit.client.0.smithi047.stdout: "minimum_required_version": "jewel", 2024-09-18T15:45:15.382 INFO:tasks.workunit.client.0.smithi047.stdout: "require_feature_tunables": 1, 2024-09-18T15:45:15.382 INFO:tasks.workunit.client.0.smithi047.stdout: "require_feature_tunables2": 1, 2024-09-18T15:45:15.382 INFO:tasks.workunit.client.0.smithi047.stdout: "has_v2_rules": 0, 2024-09-18T15:45:15.382 INFO:tasks.workunit.client.0.smithi047.stdout: "require_feature_tunables3": 1, 2024-09-18T15:45:15.382 INFO:tasks.workunit.client.0.smithi047.stdout: "has_v3_rules": 0, 2024-09-18T15:45:15.382 INFO:tasks.workunit.client.0.smithi047.stdout: "has_v4_buckets": 1, 2024-09-18T15:45:15.382 INFO:tasks.workunit.client.0.smithi047.stdout: "require_feature_tunables5": 1, 2024-09-18T15:45:15.382 INFO:tasks.workunit.client.0.smithi047.stdout: "has_v5_rules": 0 2024-09-18T15:45:15.382 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.382 INFO:tasks.workunit.client.0.smithi047.stdout: "choose_args": {} 2024-09-18T15:45:15.382 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.382 INFO:tasks.workunit.client.0.smithi047.stdout: "fsmap": { 2024-09-18T15:45:15.382 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 1, 2024-09-18T15:45:15.382 INFO:tasks.workunit.client.0.smithi047.stdout: "default_fscid": -1, 2024-09-18T15:45:15.382 INFO:tasks.workunit.client.0.smithi047.stdout: "compat": { 2024-09-18T15:45:15.383 INFO:tasks.workunit.client.0.smithi047.stdout: "compat": {}, 2024-09-18T15:45:15.383 INFO:tasks.workunit.client.0.smithi047.stdout: "ro_compat": {}, 2024-09-18T15:45:15.383 INFO:tasks.workunit.client.0.smithi047.stdout: "incompat": { 2024-09-18T15:45:15.383 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_1": "base v0.20", 2024-09-18T15:45:15.383 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_2": "client writeable ranges", 2024-09-18T15:45:15.383 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_3": "default file layouts on dirs", 2024-09-18T15:45:15.383 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_4": "dir inode in separate object", 2024-09-18T15:45:15.383 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_5": "mds uses versioned encoding", 2024-09-18T15:45:15.383 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-18T15:45:15.383 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_8": "no anchor table", 2024-09-18T15:45:15.383 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_9": "file layout v2", 2024-09-18T15:45:15.383 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_10": "snaprealm v2" 2024-09-18T15:45:15.383 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.383 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.383 INFO:tasks.workunit.client.0.smithi047.stdout: "feature_flags": { 2024-09-18T15:45:15.384 INFO:tasks.workunit.client.0.smithi047.stdout: "enable_multiple": true, 2024-09-18T15:45:15.384 INFO:tasks.workunit.client.0.smithi047.stdout: "ever_enabled_multiple": true 2024-09-18T15:45:15.384 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.384 INFO:tasks.workunit.client.0.smithi047.stdout: "standbys": [], 2024-09-18T15:45:15.384 INFO:tasks.workunit.client.0.smithi047.stdout: "filesystems": [] 2024-09-18T15:45:15.384 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.384 INFO:tasks.workunit.client.0.smithi047.stdout: "mdsmap_first_committed": 1, 2024-09-18T15:45:15.384 INFO:tasks.workunit.client.0.smithi047.stdout: "mdsmap_last_committed": 1, 2024-09-18T15:45:15.384 INFO:tasks.workunit.client.0.smithi047.stdout: "auth": { 2024-09-18T15:45:15.384 INFO:tasks.workunit.client.0.smithi047.stdout: "first_committed": 1, 2024-09-18T15:45:15.384 INFO:tasks.workunit.client.0.smithi047.stdout: "last_committed": 6, 2024-09-18T15:45:15.384 INFO:tasks.workunit.client.0.smithi047.stdout: "num_secrets": 4 2024-09-18T15:45:15.384 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.384 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg": 9, 2024-09-18T15:45:15.385 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg_active": 9, 2024-09-18T15:45:15.385 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg_unknown": 0, 2024-09-18T15:45:15.385 INFO:tasks.workunit.client.0.smithi047.stdout: "num_osd": 4, 2024-09-18T15:45:15.385 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_sum": { 2024-09-18T15:45:15.385 INFO:tasks.workunit.client.0.smithi047.stdout: "stat_sum": { 2024-09-18T15:45:15.385 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes": 4096, 2024-09-18T15:45:15.385 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects": 1, 2024-09-18T15:45:15.385 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_clones": 0, 2024-09-18T15:45:15.385 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_copies": 3, 2024-09-18T15:45:15.385 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing_on_primary": 1, 2024-09-18T15:45:15.385 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing": 0, 2024-09-18T15:45:15.385 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_degraded": 3, 2024-09-18T15:45:15.385 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_misplaced": 0, 2024-09-18T15:45:15.385 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_unfound": 1, 2024-09-18T15:45:15.385 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_dirty": 1, 2024-09-18T15:45:15.386 INFO:tasks.workunit.client.0.smithi047.stdout: "num_whiteouts": 0, 2024-09-18T15:45:15.386 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read": 3, 2024-09-18T15:45:15.386 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read_kb": 12, 2024-09-18T15:45:15.386 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write": 1, 2024-09-18T15:45:15.386 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write_kb": 4, 2024-09-18T15:45:15.386 INFO:tasks.workunit.client.0.smithi047.stdout: "num_scrub_errors": 0, 2024-09-18T15:45:15.386 INFO:tasks.workunit.client.0.smithi047.stdout: "num_shallow_scrub_errors": 0, 2024-09-18T15:45:15.386 INFO:tasks.workunit.client.0.smithi047.stdout: "num_deep_scrub_errors": 0, 2024-09-18T15:45:15.386 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_recovered": 3, 2024-09-18T15:45:15.386 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_recovered": 12288, 2024-09-18T15:45:15.386 INFO:tasks.workunit.client.0.smithi047.stdout: "num_keys_recovered": 0, 2024-09-18T15:45:15.386 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_omap": 0, 2024-09-18T15:45:15.386 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_hit_set_archive": 0, 2024-09-18T15:45:15.386 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_hit_set_archive": 0, 2024-09-18T15:45:15.386 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush": 0, 2024-09-18T15:45:15.387 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_kb": 0, 2024-09-18T15:45:15.387 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict": 0, 2024-09-18T15:45:15.387 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_kb": 0, 2024-09-18T15:45:15.387 INFO:tasks.workunit.client.0.smithi047.stdout: "num_promote": 0, 2024-09-18T15:45:15.387 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_high": 0, 2024-09-18T15:45:15.387 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_low": 0, 2024-09-18T15:45:15.387 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_some": 0, 2024-09-18T15:45:15.387 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_full": 0, 2024-09-18T15:45:15.387 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_pinned": 0, 2024-09-18T15:45:15.387 INFO:tasks.workunit.client.0.smithi047.stdout: "num_legacy_snapsets": 0, 2024-09-18T15:45:15.387 INFO:tasks.workunit.client.0.smithi047.stdout: "num_large_omap_objects": 0, 2024-09-18T15:45:15.387 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_manifest": 0, 2024-09-18T15:45:15.387 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_bytes": 0, 2024-09-18T15:45:15.387 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_keys": 0, 2024-09-18T15:45:15.388 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_repaired": 3 2024-09-18T15:45:15.388 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.388 INFO:tasks.workunit.client.0.smithi047.stdout: "store_stats": { 2024-09-18T15:45:15.388 INFO:tasks.workunit.client.0.smithi047.stdout: "total": 0, 2024-09-18T15:45:15.388 INFO:tasks.workunit.client.0.smithi047.stdout: "available": 0, 2024-09-18T15:45:15.388 INFO:tasks.workunit.client.0.smithi047.stdout: "internally_reserved": 0, 2024-09-18T15:45:15.388 INFO:tasks.workunit.client.0.smithi047.stdout: "allocated": 0, 2024-09-18T15:45:15.388 INFO:tasks.workunit.client.0.smithi047.stdout: "data_stored": 0, 2024-09-18T15:45:15.388 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed": 0, 2024-09-18T15:45:15.388 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_allocated": 0, 2024-09-18T15:45:15.388 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_original": 0, 2024-09-18T15:45:15.388 INFO:tasks.workunit.client.0.smithi047.stdout: "omap_allocated": 0, 2024-09-18T15:45:15.388 INFO:tasks.workunit.client.0.smithi047.stdout: "internal_metadata": 0 2024-09-18T15:45:15.388 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.389 INFO:tasks.workunit.client.0.smithi047.stdout: "log_size": 1, 2024-09-18T15:45:15.389 INFO:tasks.workunit.client.0.smithi047.stdout: "ondisk_log_size": 1, 2024-09-18T15:45:15.389 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 27, 2024-09-18T15:45:15.389 INFO:tasks.workunit.client.0.smithi047.stdout: "acting": 27, 2024-09-18T15:45:15.389 INFO:tasks.workunit.client.0.smithi047.stdout: "num_store_stats": 0 2024-09-18T15:45:15.389 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.389 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_sum": { 2024-09-18T15:45:15.389 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 0, 2024-09-18T15:45:15.389 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 0, 2024-09-18T15:45:15.389 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pgs": 27, 2024-09-18T15:45:15.389 INFO:tasks.workunit.client.0.smithi047.stdout: "num_osds": 4, 2024-09-18T15:45:15.389 INFO:tasks.workunit.client.0.smithi047.stdout: "num_per_pool_osds": 4, 2024-09-18T15:45:15.389 INFO:tasks.workunit.client.0.smithi047.stdout: "num_per_pool_omap_osds": 4, 2024-09-18T15:45:15.389 INFO:tasks.workunit.client.0.smithi047.stdout: "kb": 419430400, 2024-09-18T15:45:15.389 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used": 1189948, 2024-09-18T15:45:15.390 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_data": 1292, 2024-09-18T15:45:15.390 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_omap": 6, 2024-09-18T15:45:15.390 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_meta": 1188601, 2024-09-18T15:45:15.390 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_avail": 418240452, 2024-09-18T15:45:15.390 INFO:tasks.workunit.client.0.smithi047.stdout: "statfs": { 2024-09-18T15:45:15.390 INFO:tasks.workunit.client.0.smithi047.stdout: "total": 429496729600, 2024-09-18T15:45:15.390 INFO:tasks.workunit.client.0.smithi047.stdout: "available": 428278222848, 2024-09-18T15:45:15.390 INFO:tasks.workunit.client.0.smithi047.stdout: "internally_reserved": 0, 2024-09-18T15:45:15.390 INFO:tasks.workunit.client.0.smithi047.stdout: "allocated": 1323008, 2024-09-18T15:45:15.390 INFO:tasks.workunit.client.0.smithi047.stdout: "data_stored": 449160, 2024-09-18T15:45:15.390 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed": 0, 2024-09-18T15:45:15.390 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_allocated": 0, 2024-09-18T15:45:15.390 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_original": 0, 2024-09-18T15:45:15.390 INFO:tasks.workunit.client.0.smithi047.stdout: "omap_allocated": 6816, 2024-09-18T15:45:15.391 INFO:tasks.workunit.client.0.smithi047.stdout: "internal_metadata": 1217127776 2024-09-18T15:45:15.391 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.391 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_peers": [], 2024-09-18T15:45:15.391 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_trim_queue_len": 0, 2024-09-18T15:45:15.391 INFO:tasks.workunit.client.0.smithi047.stdout: "num_snap_trimming": 0, 2024-09-18T15:45:15.391 INFO:tasks.workunit.client.0.smithi047.stdout: "num_shards_repaired": 4, 2024-09-18T15:45:15.391 INFO:tasks.workunit.client.0.smithi047.stdout: "op_queue_age_hist": { 2024-09-18T15:45:15.391 INFO:tasks.workunit.client.0.smithi047.stdout: "histogram": [ 2024-09-18T15:45:15.391 INFO:tasks.workunit.client.0.smithi047.stdout: 0, 2024-09-18T15:45:15.391 INFO:tasks.workunit.client.0.smithi047.stdout: 0, 2024-09-18T15:45:15.391 INFO:tasks.workunit.client.0.smithi047.stdout: 0, 2024-09-18T15:45:15.392 INFO:tasks.workunit.client.0.smithi047.stdout: 0, 2024-09-18T15:45:15.392 INFO:tasks.workunit.client.0.smithi047.stdout: 0, 2024-09-18T15:45:15.392 INFO:tasks.workunit.client.0.smithi047.stdout: 0, 2024-09-18T15:45:15.392 INFO:tasks.workunit.client.0.smithi047.stdout: 0, 2024-09-18T15:45:15.392 INFO:tasks.workunit.client.0.smithi047.stdout: 0, 2024-09-18T15:45:15.392 INFO:tasks.workunit.client.0.smithi047.stdout: 0, 2024-09-18T15:45:15.392 INFO:tasks.workunit.client.0.smithi047.stdout: 0, 2024-09-18T15:45:15.392 INFO:tasks.workunit.client.0.smithi047.stdout: 0, 2024-09-18T15:45:15.392 INFO:tasks.workunit.client.0.smithi047.stdout: 0, 2024-09-18T15:45:15.392 INFO:tasks.workunit.client.0.smithi047.stdout: 0, 2024-09-18T15:45:15.392 INFO:tasks.workunit.client.0.smithi047.stdout: 0, 2024-09-18T15:45:15.393 INFO:tasks.workunit.client.0.smithi047.stdout: 0, 2024-09-18T15:45:15.393 INFO:tasks.workunit.client.0.smithi047.stdout: 0, 2024-09-18T15:45:15.393 INFO:tasks.workunit.client.0.smithi047.stdout: 0, 2024-09-18T15:45:15.393 INFO:tasks.workunit.client.0.smithi047.stdout: 1 2024-09-18T15:45:15.393 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:45:15.393 INFO:tasks.workunit.client.0.smithi047.stdout: "upper_bound": 262144 2024-09-18T15:45:15.393 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.393 INFO:tasks.workunit.client.0.smithi047.stdout: "perf_stat": { 2024-09-18T15:45:15.393 INFO:tasks.workunit.client.0.smithi047.stdout: "commit_latency_ms": 0, 2024-09-18T15:45:15.393 INFO:tasks.workunit.client.0.smithi047.stdout: "apply_latency_ms": 0, 2024-09-18T15:45:15.393 INFO:tasks.workunit.client.0.smithi047.stdout: "commit_latency_ns": 0, 2024-09-18T15:45:15.393 INFO:tasks.workunit.client.0.smithi047.stdout: "apply_latency_ns": 0 2024-09-18T15:45:15.393 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.393 INFO:tasks.workunit.client.0.smithi047.stdout: "alerts": [], 2024-09-18T15:45:15.394 INFO:tasks.workunit.client.0.smithi047.stdout: "network_ping_times": [] 2024-09-18T15:45:15.394 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.394 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_sum_by_class": { 2024-09-18T15:45:15.394 INFO:tasks.workunit.client.0.smithi047.stdout: "hdd": { 2024-09-18T15:45:15.394 INFO:tasks.workunit.client.0.smithi047.stdout: "up_from": 0, 2024-09-18T15:45:15.394 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 0, 2024-09-18T15:45:15.394 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pgs": 27, 2024-09-18T15:45:15.394 INFO:tasks.workunit.client.0.smithi047.stdout: "num_osds": 4, 2024-09-18T15:45:15.394 INFO:tasks.workunit.client.0.smithi047.stdout: "num_per_pool_osds": 4, 2024-09-18T15:45:15.394 INFO:tasks.workunit.client.0.smithi047.stdout: "num_per_pool_omap_osds": 4, 2024-09-18T15:45:15.394 INFO:tasks.workunit.client.0.smithi047.stdout: "kb": 419430400, 2024-09-18T15:45:15.394 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used": 1189948, 2024-09-18T15:45:15.394 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_data": 1292, 2024-09-18T15:45:15.394 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_omap": 6, 2024-09-18T15:45:15.394 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_used_meta": 1188601, 2024-09-18T15:45:15.395 INFO:tasks.workunit.client.0.smithi047.stdout: "kb_avail": 418240452, 2024-09-18T15:45:15.395 INFO:tasks.workunit.client.0.smithi047.stdout: "statfs": { 2024-09-18T15:45:15.395 INFO:tasks.workunit.client.0.smithi047.stdout: "total": 429496729600, 2024-09-18T15:45:15.395 INFO:tasks.workunit.client.0.smithi047.stdout: "available": 428278222848, 2024-09-18T15:45:15.395 INFO:tasks.workunit.client.0.smithi047.stdout: "internally_reserved": 0, 2024-09-18T15:45:15.395 INFO:tasks.workunit.client.0.smithi047.stdout: "allocated": 1323008, 2024-09-18T15:45:15.395 INFO:tasks.workunit.client.0.smithi047.stdout: "data_stored": 449160, 2024-09-18T15:45:15.395 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed": 0, 2024-09-18T15:45:15.395 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_allocated": 0, 2024-09-18T15:45:15.395 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_original": 0, 2024-09-18T15:45:15.395 INFO:tasks.workunit.client.0.smithi047.stdout: "omap_allocated": 6816, 2024-09-18T15:45:15.395 INFO:tasks.workunit.client.0.smithi047.stdout: "internal_metadata": 1217127776 2024-09-18T15:45:15.395 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.395 INFO:tasks.workunit.client.0.smithi047.stdout: "hb_peers": [], 2024-09-18T15:45:15.395 INFO:tasks.workunit.client.0.smithi047.stdout: "snap_trim_queue_len": 0, 2024-09-18T15:45:15.396 INFO:tasks.workunit.client.0.smithi047.stdout: "num_snap_trimming": 0, 2024-09-18T15:45:15.396 INFO:tasks.workunit.client.0.smithi047.stdout: "num_shards_repaired": 4, 2024-09-18T15:45:15.396 INFO:tasks.workunit.client.0.smithi047.stdout: "op_queue_age_hist": { 2024-09-18T15:45:15.396 INFO:tasks.workunit.client.0.smithi047.stdout: "histogram": [ 2024-09-18T15:45:15.396 INFO:tasks.workunit.client.0.smithi047.stdout: 0, 2024-09-18T15:45:15.396 INFO:tasks.workunit.client.0.smithi047.stdout: 0, 2024-09-18T15:45:15.396 INFO:tasks.workunit.client.0.smithi047.stdout: 0, 2024-09-18T15:45:15.396 INFO:tasks.workunit.client.0.smithi047.stdout: 0, 2024-09-18T15:45:15.396 INFO:tasks.workunit.client.0.smithi047.stdout: 0, 2024-09-18T15:45:15.396 INFO:tasks.workunit.client.0.smithi047.stdout: 0, 2024-09-18T15:45:15.396 INFO:tasks.workunit.client.0.smithi047.stdout: 0, 2024-09-18T15:45:15.396 INFO:tasks.workunit.client.0.smithi047.stdout: 0, 2024-09-18T15:45:15.396 INFO:tasks.workunit.client.0.smithi047.stdout: 0, 2024-09-18T15:45:15.396 INFO:tasks.workunit.client.0.smithi047.stdout: 0, 2024-09-18T15:45:15.397 INFO:tasks.workunit.client.0.smithi047.stdout: 0, 2024-09-18T15:45:15.397 INFO:tasks.workunit.client.0.smithi047.stdout: 0, 2024-09-18T15:45:15.397 INFO:tasks.workunit.client.0.smithi047.stdout: 0, 2024-09-18T15:45:15.397 INFO:tasks.workunit.client.0.smithi047.stdout: 0, 2024-09-18T15:45:15.397 INFO:tasks.workunit.client.0.smithi047.stdout: 0, 2024-09-18T15:45:15.397 INFO:tasks.workunit.client.0.smithi047.stdout: 0, 2024-09-18T15:45:15.397 INFO:tasks.workunit.client.0.smithi047.stdout: 0, 2024-09-18T15:45:15.397 INFO:tasks.workunit.client.0.smithi047.stdout: 1 2024-09-18T15:45:15.397 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:45:15.397 INFO:tasks.workunit.client.0.smithi047.stdout: "upper_bound": 262144 2024-09-18T15:45:15.397 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.397 INFO:tasks.workunit.client.0.smithi047.stdout: "perf_stat": { 2024-09-18T15:45:15.397 INFO:tasks.workunit.client.0.smithi047.stdout: "commit_latency_ms": 0, 2024-09-18T15:45:15.397 INFO:tasks.workunit.client.0.smithi047.stdout: "apply_latency_ms": 0, 2024-09-18T15:45:15.397 INFO:tasks.workunit.client.0.smithi047.stdout: "commit_latency_ns": 0, 2024-09-18T15:45:15.398 INFO:tasks.workunit.client.0.smithi047.stdout: "apply_latency_ns": 0 2024-09-18T15:45:15.398 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.398 INFO:tasks.workunit.client.0.smithi047.stdout: "alerts": [], 2024-09-18T15:45:15.398 INFO:tasks.workunit.client.0.smithi047.stdout: "network_ping_times": [] 2024-09-18T15:45:15.398 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.398 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.398 INFO:tasks.workunit.client.0.smithi047.stdout: "pool_stats": [ 2024-09-18T15:45:15.398 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.398 INFO:tasks.workunit.client.0.smithi047.stdout: "poolid": 2, 2024-09-18T15:45:15.398 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg": 1, 2024-09-18T15:45:15.398 INFO:tasks.workunit.client.0.smithi047.stdout: "stat_sum": { 2024-09-18T15:45:15.398 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes": 4096, 2024-09-18T15:45:15.398 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects": 1, 2024-09-18T15:45:15.398 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_clones": 0, 2024-09-18T15:45:15.398 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_copies": 3, 2024-09-18T15:45:15.399 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing_on_primary": 1, 2024-09-18T15:45:15.399 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing": 0, 2024-09-18T15:45:15.399 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_degraded": 3, 2024-09-18T15:45:15.399 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_misplaced": 0, 2024-09-18T15:45:15.399 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_unfound": 1, 2024-09-18T15:45:15.399 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_dirty": 1, 2024-09-18T15:45:15.399 INFO:tasks.workunit.client.0.smithi047.stdout: "num_whiteouts": 0, 2024-09-18T15:45:15.399 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read": 3, 2024-09-18T15:45:15.399 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read_kb": 12, 2024-09-18T15:45:15.399 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write": 1, 2024-09-18T15:45:15.399 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write_kb": 4, 2024-09-18T15:45:15.400 INFO:tasks.workunit.client.0.smithi047.stdout: "num_scrub_errors": 0, 2024-09-18T15:45:15.400 INFO:tasks.workunit.client.0.smithi047.stdout: "num_shallow_scrub_errors": 0, 2024-09-18T15:45:15.400 INFO:tasks.workunit.client.0.smithi047.stdout: "num_deep_scrub_errors": 0, 2024-09-18T15:45:15.400 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_recovered": 3, 2024-09-18T15:45:15.400 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_recovered": 12288, 2024-09-18T15:45:15.400 INFO:tasks.workunit.client.0.smithi047.stdout: "num_keys_recovered": 0, 2024-09-18T15:45:15.400 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_omap": 0, 2024-09-18T15:45:15.400 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_hit_set_archive": 0, 2024-09-18T15:45:15.400 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_hit_set_archive": 0, 2024-09-18T15:45:15.400 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush": 0, 2024-09-18T15:45:15.400 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_kb": 0, 2024-09-18T15:45:15.400 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict": 0, 2024-09-18T15:45:15.400 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_kb": 0, 2024-09-18T15:45:15.400 INFO:tasks.workunit.client.0.smithi047.stdout: "num_promote": 0, 2024-09-18T15:45:15.401 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_high": 0, 2024-09-18T15:45:15.401 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_low": 0, 2024-09-18T15:45:15.401 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_some": 0, 2024-09-18T15:45:15.401 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_full": 0, 2024-09-18T15:45:15.401 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_pinned": 0, 2024-09-18T15:45:15.401 INFO:tasks.workunit.client.0.smithi047.stdout: "num_legacy_snapsets": 0, 2024-09-18T15:45:15.401 INFO:tasks.workunit.client.0.smithi047.stdout: "num_large_omap_objects": 0, 2024-09-18T15:45:15.401 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_manifest": 0, 2024-09-18T15:45:15.401 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_bytes": 0, 2024-09-18T15:45:15.401 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_keys": 0, 2024-09-18T15:45:15.401 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_repaired": 3 2024-09-18T15:45:15.401 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.401 INFO:tasks.workunit.client.0.smithi047.stdout: "store_stats": { 2024-09-18T15:45:15.401 INFO:tasks.workunit.client.0.smithi047.stdout: "total": 0, 2024-09-18T15:45:15.402 INFO:tasks.workunit.client.0.smithi047.stdout: "available": 0, 2024-09-18T15:45:15.402 INFO:tasks.workunit.client.0.smithi047.stdout: "internally_reserved": 0, 2024-09-18T15:45:15.402 INFO:tasks.workunit.client.0.smithi047.stdout: "allocated": 12288, 2024-09-18T15:45:15.402 INFO:tasks.workunit.client.0.smithi047.stdout: "data_stored": 12288, 2024-09-18T15:45:15.402 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed": 0, 2024-09-18T15:45:15.402 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_allocated": 0, 2024-09-18T15:45:15.402 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_original": 0, 2024-09-18T15:45:15.402 INFO:tasks.workunit.client.0.smithi047.stdout: "omap_allocated": 0, 2024-09-18T15:45:15.402 INFO:tasks.workunit.client.0.smithi047.stdout: "internal_metadata": 0 2024-09-18T15:45:15.402 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.402 INFO:tasks.workunit.client.0.smithi047.stdout: "log_size": 1, 2024-09-18T15:45:15.402 INFO:tasks.workunit.client.0.smithi047.stdout: "ondisk_log_size": 1, 2024-09-18T15:45:15.402 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 3, 2024-09-18T15:45:15.402 INFO:tasks.workunit.client.0.smithi047.stdout: "acting": 3, 2024-09-18T15:45:15.402 INFO:tasks.workunit.client.0.smithi047.stdout: "num_store_stats": 3 2024-09-18T15:45:15.402 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.403 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.403 INFO:tasks.workunit.client.0.smithi047.stdout: "poolid": 1, 2024-09-18T15:45:15.403 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg": 8, 2024-09-18T15:45:15.403 INFO:tasks.workunit.client.0.smithi047.stdout: "stat_sum": { 2024-09-18T15:45:15.403 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes": 0, 2024-09-18T15:45:15.403 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects": 0, 2024-09-18T15:45:15.403 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_clones": 0, 2024-09-18T15:45:15.403 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_copies": 0, 2024-09-18T15:45:15.403 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing_on_primary": 0, 2024-09-18T15:45:15.403 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing": 0, 2024-09-18T15:45:15.403 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_degraded": 0, 2024-09-18T15:45:15.403 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_misplaced": 0, 2024-09-18T15:45:15.403 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_unfound": 0, 2024-09-18T15:45:15.403 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_dirty": 0, 2024-09-18T15:45:15.404 INFO:tasks.workunit.client.0.smithi047.stdout: "num_whiteouts": 0, 2024-09-18T15:45:15.404 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read": 0, 2024-09-18T15:45:15.404 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read_kb": 0, 2024-09-18T15:45:15.404 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write": 0, 2024-09-18T15:45:15.404 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write_kb": 0, 2024-09-18T15:45:15.404 INFO:tasks.workunit.client.0.smithi047.stdout: "num_scrub_errors": 0, 2024-09-18T15:45:15.404 INFO:tasks.workunit.client.0.smithi047.stdout: "num_shallow_scrub_errors": 0, 2024-09-18T15:45:15.404 INFO:tasks.workunit.client.0.smithi047.stdout: "num_deep_scrub_errors": 0, 2024-09-18T15:45:15.404 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_recovered": 0, 2024-09-18T15:45:15.404 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_recovered": 0, 2024-09-18T15:45:15.404 INFO:tasks.workunit.client.0.smithi047.stdout: "num_keys_recovered": 0, 2024-09-18T15:45:15.404 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_omap": 0, 2024-09-18T15:45:15.404 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_hit_set_archive": 0, 2024-09-18T15:45:15.404 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_hit_set_archive": 0, 2024-09-18T15:45:15.404 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush": 0, 2024-09-18T15:45:15.405 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_kb": 0, 2024-09-18T15:45:15.405 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict": 0, 2024-09-18T15:45:15.405 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_kb": 0, 2024-09-18T15:45:15.405 INFO:tasks.workunit.client.0.smithi047.stdout: "num_promote": 0, 2024-09-18T15:45:15.405 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_high": 0, 2024-09-18T15:45:15.405 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_low": 0, 2024-09-18T15:45:15.405 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_some": 0, 2024-09-18T15:45:15.405 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_full": 0, 2024-09-18T15:45:15.405 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_pinned": 0, 2024-09-18T15:45:15.405 INFO:tasks.workunit.client.0.smithi047.stdout: "num_legacy_snapsets": 0, 2024-09-18T15:45:15.405 INFO:tasks.workunit.client.0.smithi047.stdout: "num_large_omap_objects": 0, 2024-09-18T15:45:15.405 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_manifest": 0, 2024-09-18T15:45:15.405 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_bytes": 0, 2024-09-18T15:45:15.405 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_keys": 0, 2024-09-18T15:45:15.405 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_repaired": 0 2024-09-18T15:45:15.406 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.406 INFO:tasks.workunit.client.0.smithi047.stdout: "store_stats": { 2024-09-18T15:45:15.406 INFO:tasks.workunit.client.0.smithi047.stdout: "total": 0, 2024-09-18T15:45:15.406 INFO:tasks.workunit.client.0.smithi047.stdout: "available": 0, 2024-09-18T15:45:15.406 INFO:tasks.workunit.client.0.smithi047.stdout: "internally_reserved": 0, 2024-09-18T15:45:15.406 INFO:tasks.workunit.client.0.smithi047.stdout: "allocated": 0, 2024-09-18T15:45:15.406 INFO:tasks.workunit.client.0.smithi047.stdout: "data_stored": 0, 2024-09-18T15:45:15.406 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed": 0, 2024-09-18T15:45:15.406 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_allocated": 0, 2024-09-18T15:45:15.406 INFO:tasks.workunit.client.0.smithi047.stdout: "data_compressed_original": 0, 2024-09-18T15:45:15.406 INFO:tasks.workunit.client.0.smithi047.stdout: "omap_allocated": 0, 2024-09-18T15:45:15.406 INFO:tasks.workunit.client.0.smithi047.stdout: "internal_metadata": 0 2024-09-18T15:45:15.406 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.406 INFO:tasks.workunit.client.0.smithi047.stdout: "log_size": 0, 2024-09-18T15:45:15.407 INFO:tasks.workunit.client.0.smithi047.stdout: "ondisk_log_size": 0, 2024-09-18T15:45:15.407 INFO:tasks.workunit.client.0.smithi047.stdout: "up": 24, 2024-09-18T15:45:15.407 INFO:tasks.workunit.client.0.smithi047.stdout: "acting": 24, 2024-09-18T15:45:15.407 INFO:tasks.workunit.client.0.smithi047.stdout: "num_store_stats": 4 2024-09-18T15:45:15.407 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.407 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:45:15.407 INFO:tasks.workunit.client.0.smithi047.stdout: "osd_stats": [ 2024-09-18T15:45:15.407 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.407 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 0, 2024-09-18T15:45:15.407 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 25769803848 2024-09-18T15:45:15.407 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.407 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.407 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 1, 2024-09-18T15:45:15.407 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 51539607617 2024-09-18T15:45:15.407 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.408 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.408 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 2, 2024-09-18T15:45:15.408 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 81604378683 2024-09-18T15:45:15.408 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.408 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.408 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 3, 2024-09-18T15:45:15.408 INFO:tasks.workunit.client.0.smithi047.stdout: "seq": 137438953527 2024-09-18T15:45:15.408 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.408 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:45:15.408 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg_by_state": [ 2024-09-18T15:45:15.408 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.408 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "active+recovery_unfound+degraded", 2024-09-18T15:45:15.408 INFO:tasks.workunit.client.0.smithi047.stdout: "num": 1 2024-09-18T15:45:15.408 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.408 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.409 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "active+clean", 2024-09-18T15:45:15.409 INFO:tasks.workunit.client.0.smithi047.stdout: "num": 8 2024-09-18T15:45:15.409 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.409 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:45:15.409 INFO:tasks.workunit.client.0.smithi047.stdout: "num_pg_by_osd": [ 2024-09-18T15:45:15.409 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.409 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 3, 2024-09-18T15:45:15.409 INFO:tasks.workunit.client.0.smithi047.stdout: "num_primary_pg": 4, 2024-09-18T15:45:15.409 INFO:tasks.workunit.client.0.smithi047.stdout: "num_acting_pg": 8, 2024-09-18T15:45:15.409 INFO:tasks.workunit.client.0.smithi047.stdout: "num_up_not_acting_pg": 0 2024-09-18T15:45:15.409 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.409 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.409 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 2, 2024-09-18T15:45:15.409 INFO:tasks.workunit.client.0.smithi047.stdout: "num_primary_pg": 1, 2024-09-18T15:45:15.409 INFO:tasks.workunit.client.0.smithi047.stdout: "num_acting_pg": 5, 2024-09-18T15:45:15.410 INFO:tasks.workunit.client.0.smithi047.stdout: "num_up_not_acting_pg": 0 2024-09-18T15:45:15.410 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.410 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.410 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 1, 2024-09-18T15:45:15.410 INFO:tasks.workunit.client.0.smithi047.stdout: "num_primary_pg": 3, 2024-09-18T15:45:15.410 INFO:tasks.workunit.client.0.smithi047.stdout: "num_acting_pg": 6, 2024-09-18T15:45:15.410 INFO:tasks.workunit.client.0.smithi047.stdout: "num_up_not_acting_pg": 0 2024-09-18T15:45:15.410 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.410 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.410 INFO:tasks.workunit.client.0.smithi047.stdout: "osd": 0, 2024-09-18T15:45:15.410 INFO:tasks.workunit.client.0.smithi047.stdout: "num_primary_pg": 1, 2024-09-18T15:45:15.410 INFO:tasks.workunit.client.0.smithi047.stdout: "num_acting_pg": 8, 2024-09-18T15:45:15.410 INFO:tasks.workunit.client.0.smithi047.stdout: "num_up_not_acting_pg": 0 2024-09-18T15:45:15.410 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.410 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:45:15.411 INFO:tasks.workunit.client.0.smithi047.stdout: "purged_snaps": [ 2024-09-18T15:45:15.411 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.411 INFO:tasks.workunit.client.0.smithi047.stdout: "pool": 1, 2024-09-18T15:45:15.411 INFO:tasks.workunit.client.0.smithi047.stdout: "purged_snaps": {} 2024-09-18T15:45:15.411 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.411 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:45:15.411 INFO:tasks.workunit.client.0.smithi047.stdout: "pool": 2, 2024-09-18T15:45:15.411 INFO:tasks.workunit.client.0.smithi047.stdout: "purged_snaps": {} 2024-09-18T15:45:15.411 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.411 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:45:15.411 INFO:tasks.workunit.client.0.smithi047.stdout: "servicemap": { 2024-09-18T15:45:15.411 INFO:tasks.workunit.client.0.smithi047.stdout: "epoch": 6, 2024-09-18T15:45:15.411 INFO:tasks.workunit.client.0.smithi047.stdout: "modified": "2024-09-18T15:43:33.115545+0000", 2024-09-18T15:45:15.411 INFO:tasks.workunit.client.0.smithi047.stdout: "services": {} 2024-09-18T15:45:15.411 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:45:15.412 INFO:tasks.workunit.client.0.smithi047.stdout: "paxos": { 2024-09-18T15:45:15.412 INFO:tasks.workunit.client.0.smithi047.stdout: "first_committed": 252, 2024-09-18T15:45:15.412 INFO:tasks.workunit.client.0.smithi047.stdout: "last_committed": 790, 2024-09-18T15:45:15.412 INFO:tasks.workunit.client.0.smithi047.stdout: "last_pn": 0, 2024-09-18T15:45:15.412 INFO:tasks.workunit.client.0.smithi047.stdout: "accepted_pn": 0 2024-09-18T15:45:15.412 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:45:15.412 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:45:15.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-18T15:45:15.413 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:160: rados_get_data: ceph pg 2.0 query 2024-09-18T15:45:15.413 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:160: rados_get_data: jq .info.stats.stat_sum.num_objects_repaired 2024-09-18T15:45:15.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:160: rados_get_data: COUNT=3 2024-09-18T15:45:15.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:161: rados_get_data: test 3 = 3 2024-09-18T15:45:15.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:162: rados_get_data: flush_pg_stats 2024-09-18T15:45:15.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:45:15.449 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:45:15.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:45:15.838 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:45:15.838 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T15:45:15.839 INFO:tasks.workunit.client.0.smithi047.stderr:3' 2024-09-18T15:45:15.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:45:15.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:45:15.839 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:45:15.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803850 2024-09-18T15:45:15.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803850 2024-09-18T15:45:15.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803850' 2024-09-18T15:45:15.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:45:15.945 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:45:16.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=51539607620 2024-09-18T15:45:16.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 51539607620 2024-09-18T15:45:16.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803850 1-51539607620' 2024-09-18T15:45:16.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:45:16.051 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:45:16.157 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378685 2024-09-18T15:45:16.157 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378685 2024-09-18T15:45:16.157 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803850 1-51539607620 2-81604378685' 2024-09-18T15:45:16.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:45:16.158 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:45:16.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=137438953529 2024-09-18T15:45:16.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 137438953529 2024-09-18T15:45:16.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803850 1-51539607620 2-81604378685 3-137438953529' 2024-09-18T15:45:16.264 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:45:16.264 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803850 2024-09-18T15:45:16.264 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:45:16.265 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:45:16.269 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803850 2024-09-18T15:45:16.269 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:45:16.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803850 2024-09-18T15:45:16.269 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 25769803850 2024-09-18T15:45:16.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803850' 2024-09-18T15:45:16.270 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:45:16.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803848 -lt 25769803850 2024-09-18T15:45:16.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:45:17.666 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:45:17.667 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:45:18.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803851 -lt 25769803850 2024-09-18T15:45:18.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:45:18.085 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-51539607620 2024-09-18T15:45:18.085 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:45:18.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:45:18.087 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-51539607620 2024-09-18T15:45:18.087 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:45:18.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=51539607620 2024-09-18T15:45:18.088 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 51539607620 2024-09-18T15:45:18.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 51539607620' 2024-09-18T15:45:18.088 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:45:18.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 51539607620 -lt 51539607620 2024-09-18T15:45:18.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:45:18.487 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378685 2024-09-18T15:45:18.487 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:45:18.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:45:18.489 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378685 2024-09-18T15:45:18.489 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:45:18.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378685 2024-09-18T15:45:18.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378685' 2024-09-18T15:45:18.490 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 81604378685 2024-09-18T15:45:18.490 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:45:18.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378685 -lt 81604378685 2024-09-18T15:45:18.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:45:18.880 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-137438953529 2024-09-18T15:45:18.880 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:45:18.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:45:18.882 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-137438953529 2024-09-18T15:45:18.882 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:45:18.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=137438953529 2024-09-18T15:45:18.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 137438953529' 2024-09-18T15:45:18.883 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 137438953529 2024-09-18T15:45:18.883 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:45:19.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 137438953529 -lt 137438953529 2024-09-18T15:45:19.247 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:163: rados_get_data: ceph pg dump --format=json-pretty 2024-09-18T15:45:19.247 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:163: rados_get_data: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-09-18T15:45:19.595 INFO:tasks.workunit.client.0.smithi047.stderr:dumped all 2024-09-18T15:45:19.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:163: rados_get_data: COUNT=4 2024-09-18T15:45:19.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:164: rados_get_data: test 4 = 4 2024-09-18T15:45:19.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:177: TEST_rados_get_with_eio: delete_pool pool-rep 2024-09-18T15:45:19.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-rep 2024-09-18T15:45:19.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-rep pool-rep --yes-i-really-really-mean-it 2024-09-18T15:45:20.213 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'pool-rep' does not exist 2024-09-18T15:45:20.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:43: run: teardown td/osd-rep-recov-eio.sh 2024-09-18T15:45:20.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:45:20.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:45:20.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-09-18T15:45:20.225 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:45:20.225 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:45:20.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:45:20.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:45:20.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:45:20.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:45:20.334 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:45:20.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:45:20.334 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:45:20.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:45:20.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:45:20.336 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:45:20.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:45:20.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:45:20.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:45:20.337 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:45:20.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:45:20.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:45:20.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-09-18T15:45:20.365 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:45:20.365 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:45:20.365 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:45:20.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.397939 2024-09-18T15:45:20.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:45:20.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:45:20.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:35: run: for func in $funcs 2024-09-18T15:45:20.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:36: run: setup td/osd-rep-recov-eio.sh 2024-09-18T15:45:20.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:45:20.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-rep-recov-eio.sh 2024-09-18T15:45:20.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:45:20.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:45:20.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-09-18T15:45:20.367 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:45:20.367 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:45:20.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:45:20.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:45:20.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:45:20.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:45:20.368 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:45:20.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:45:20.369 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:45:20.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:45:20.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:45:20.371 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:45:20.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:45:20.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:45:20.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:45:20.372 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:45:20.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:45:20.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:45:20.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-09-18T15:45:20.375 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:45:20.375 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:45:20.375 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:45:20.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.397939 2024-09-18T15:45:20.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:45:20.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:45:20.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-rep-recov-eio.sh 2024-09-18T15:45:20.377 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T15:45:20.377 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:45:20.377 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:45:20.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.397939 2024-09-18T15:45:20.379 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T15:45:20.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T15:45:20.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T15:45:20.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-rep-recov-eio.sh 1' TERM HUP INT 2024-09-18T15:45:20.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:38: run: run_mon td/osd-rep-recov-eio.sh a --mon_osd_warn_num_repaired=10 2024-09-18T15:45:20.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:45:20.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T15:45:20.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T15:45:20.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T15:45:20.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-rep-recov-eio.sh/a 2024-09-18T15:45:20.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-rep-recov-eio.sh/a --run-dir=td/osd-rep-recov-eio.sh --mon_osd_warn_num_repaired=10 2024-09-18T15:45:20.755 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T15:45:20.755 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:45:20.755 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:45:20.755 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:45:20.755 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:45:20.755 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:45:20.755 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:45:20.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-rep-recov-eio.sh/a '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --mon-cluster-log-file=td/osd-rep-recov-eio.sh/log --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_osd_warn_num_repaired=10 2024-09-18T15:45:21.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T15:45:21.006 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T15:45:21.006 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:45:21.006 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:45:21.006 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T15:45:21.007 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T15:45:21.007 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:45:21.008 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:45:21.008 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:45:21.008 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:45:21.008 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:45:21.008 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:45:21.008 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-mon.a.asok 2024-09-18T15:45:21.008 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:45:21.008 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.397939/ceph-mon.a.asok config get fsid 2024-09-18T15:45:21.085 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T15:45:21.085 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:45:21.085 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:45:21.085 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T15:45:21.086 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T15:45:21.086 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:45:21.086 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:45:21.086 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:45:21.086 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:45:21.086 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:45:21.086 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:45:21.086 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-mon.a.asok 2024-09-18T15:45:21.086 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:45:21.086 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.397939/ceph-mon.a.asok config get mon_host 2024-09-18T15:45:21.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:39: run: run_mgr td/osd-rep-recov-eio.sh x 2024-09-18T15:45:21.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:45:21.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T15:45:21.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T15:45:21.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T15:45:21.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-rep-recov-eio.sh/x 2024-09-18T15:45:21.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T15:45:22.004 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T15:45:22.004 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:45:22.004 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:45:22.004 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:45:22.004 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:45:22.004 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:45:22.004 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:45:22.005 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:45:22.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-rep-recov-eio.sh/x '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:45:22.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:40: run: ceph osd pool create foo 8 2024-09-18T15:45:22.334 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'foo' already exists 2024-09-18T15:45:22.343 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:42: run: TEST_rados_repair_warning td/osd-rep-recov-eio.sh 2024-09-18T15:45:22.343 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:181: TEST_rados_repair_warning: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:45:22.344 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:182: TEST_rados_repair_warning: expr 10 + 1 2024-09-18T15:45:22.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:182: TEST_rados_repair_warning: local OBJS=11 2024-09-18T15:45:22.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:184: TEST_rados_repair_warning: setup_osds 4 2024-09-18T15:45:22.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:48: setup_osds: local count=4 2024-09-18T15:45:22.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:49: setup_osds: shift 2024-09-18T15:45:22.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:50: setup_osds: local type= 2024-09-18T15:45:22.346 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: expr 4 - 1 2024-09-18T15:45:22.346 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: seq 0 3 2024-09-18T15:45:22.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T15:45:22.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 0 2024-09-18T15:45:22.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:45:22.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:45:22.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T15:45:22.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:45:22.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-09-18T15:45:22.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-09-18T15:45:22.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:45:22.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:45:22.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:45:22.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-09-18T15:45:22.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-09-18T15:45:22.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:45:22.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:45:22.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-18T15:45:22.351 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:45:22.351 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:45:22.352 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:45:22.352 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:45:22.352 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:45:22.352 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:45:22.352 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:45:22.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:45:22.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:45:22.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:45:22.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:45:22.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-18T15:45:22.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-18T15:45:22.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:45:22.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:45:22.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:45:22.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:45:22.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:45:22.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:45:22.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-09-18T15:45:22.353 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:45:22.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0f03780e-fa59-45f2-a39f-c32e81bce400 2024-09-18T15:45:22.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0f03780e-fa59-45f2-a39f-c32e81bce400' 2024-09-18T15:45:22.354 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 0f03780e-fa59-45f2-a39f-c32e81bce400 2024-09-18T15:45:22.354 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:45:22.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCS9epmBNSrFRAA5v/68dU/BGd+MKSrWd5zHw== 2024-09-18T15:45:22.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCS9epmBNSrFRAA5v/68dU/BGd+MKSrWd5zHw=="}' 2024-09-18T15:45:22.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0f03780e-fa59-45f2-a39f-c32e81bce400 -i td/osd-rep-recov-eio.sh/0/new.json 2024-09-18T15:45:22.663 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:45:22.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/0/new.json 2024-09-18T15:45:22.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCS9epmBNSrFRAA5v/68dU/BGd+MKSrWd5zHw== --osd-uuid 0f03780e-fa59-45f2-a39f-c32e81bce400 2024-09-18T15:45:22.696 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:45:22.695+0000 7fa42be3c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:45:22.707 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:45:22.707+0000 7fa42be3c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:45:22.708 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:45:22.708+0000 7fa42be3c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:45:22.709 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:45:22.708+0000 7fa42be3c440 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-18T15:45:22.709 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:45:22.708+0000 7fa42be3c440 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-18T15:45:22.709 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:45:22.708+0000 7fa42be3c440 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-18T15:45:22.748 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:45:22.747+0000 7fa42be3c440 -1 bluestore(td/osd-rep-recov-eio.sh/0) _read_fsid unparsable uuid 2024-09-18T15:45:28.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/0/keyring 2024-09-18T15:45:28.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:45:28.057 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T15:45:28.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T15:45:28.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:45:28.569 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T15:45:28.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T15:45:28.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:45:28.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:45:28.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:45:28.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:45:28.590 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:45:28.589+0000 7f6426e0c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:45:28.612 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:45:28.612+0000 7f6426e0c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:45:28.613 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:45:28.613+0000 7f6426e0c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:45:29.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T15:45:29.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:45:29.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:45:29.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:45:29.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:45:29.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:45:29.010 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:45:29.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:45:29.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:45:29.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:45:29.152 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:45:29.151+0000 7f6426e0c440 -1 Falling back to public interface 2024-09-18T15:45:29.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:45:30.419 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:45:30.420 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:45:30.420 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:45:30.420 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:45:30.420 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:45:30.420 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:45:30.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:45:31.633 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:45:31.633+0000 7f6426e0c440 -1 osd.0 0 log_to_monitors true 2024-09-18T15:45:31.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:45:31.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:45:31.796 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:45:31.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:45:31.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:45:31.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:45:32.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:45:33.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:45:33.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:45:33.201 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:45:33.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:45:33.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:45:33.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:45:33.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:45:34.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:45:34.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:45:34.597 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:45:34.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:45:34.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:45:34.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:45:34.993 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:45:35.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:45:35.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:45:35.994 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:45:35.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:45:35.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:45:35.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:45:36.432 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:45:37.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:45:37.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:45:37.434 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:45:37.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:45:37.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:45:37.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:45:37.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:45:38.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:45:38.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:45:38.842 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:45:38.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:45:38.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:45:38.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:45:39.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:45:40.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:45:40.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:45:40.305 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:45:40.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:45:40.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:45:40.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:45:40.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:45:41.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:45:41.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:45:41.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:45:41.718 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:45:41.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:45:41.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:45:42.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:45:43.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:45:43.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:45:43.124 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:45:43.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:45:43.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:45:43.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:45:44.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:45:45.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:45:45.061 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:45:45.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:45:45.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:45:45.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:45:45.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:45:45.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:45:46.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:45:46.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:45:46.573 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:45:46.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:45:46.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:45:46.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:45:47.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:45:48.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:45:48.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:45:48.298 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:45:48.299 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:45:48.299 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:45:48.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:45:49.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:45:50.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:45:50.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:45:50.010 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:45:50.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:45:50.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:45:50.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:45:50.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:45:51.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:45:51.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:45:51.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:45:51.779 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:45:51.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:45:51.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:45:52.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:45:53.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:45:53.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:45:53.867 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:45:53.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:45:53.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:45:53.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:45:54.363 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/411094,v1:127.0.0.1:6803/411094] [v2:127.0.0.1:6804/411094,v1:127.0.0.1:6805/411094] exists,up 0f03780e-fa59-45f2-a39f-c32e81bce400 2024-09-18T15:45:54.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:45:54.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:45:54.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:45:54.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T15:45:54.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 1 2024-09-18T15:45:54.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:45:54.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:45:54.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T15:45:54.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:45:54.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2024-09-18T15:45:54.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-09-18T15:45:54.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:45:54.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:45:54.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:45:54.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2024-09-18T15:45:54.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2024-09-18T15:45:54.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:45:54.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:45:54.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-18T15:45:54.369 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:45:54.369 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:45:54.369 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:45:54.369 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:45:54.370 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:45:54.370 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:45:54.370 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:45:54.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:45:54.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:45:54.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:45:54.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:45:54.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-18T15:45:54.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-18T15:45:54.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:45:54.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:45:54.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:45:54.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:45:54.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:45:54.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:45:54.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2024-09-18T15:45:54.371 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:45:54.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2bd4d6c9-1410-458b-b4f9-9ad419246843 2024-09-18T15:45:54.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 2bd4d6c9-1410-458b-b4f9-9ad419246843' 2024-09-18T15:45:54.371 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 2bd4d6c9-1410-458b-b4f9-9ad419246843 2024-09-18T15:45:54.371 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:45:54.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCy9epmKcGiFhAAD2CaaF/rJwCYypqrGFf7uA== 2024-09-18T15:45:54.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCy9epmKcGiFhAAD2CaaF/rJwCYypqrGFf7uA=="}' 2024-09-18T15:45:54.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2bd4d6c9-1410-458b-b4f9-9ad419246843 -i td/osd-rep-recov-eio.sh/1/new.json 2024-09-18T15:45:54.923 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:45:54.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/1/new.json 2024-09-18T15:45:54.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCy9epmKcGiFhAAD2CaaF/rJwCYypqrGFf7uA== --osd-uuid 2bd4d6c9-1410-458b-b4f9-9ad419246843 2024-09-18T15:45:54.957 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:45:54.956+0000 7f02f4834440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:45:54.959 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:45:54.958+0000 7f02f4834440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:45:54.960 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:45:54.959+0000 7f02f4834440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:45:54.960 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:45:54.959+0000 7f02f4834440 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-18T15:45:54.960 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:45:54.959+0000 7f02f4834440 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-18T15:45:54.960 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:45:54.959+0000 7f02f4834440 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-18T15:45:55.038 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:45:55.037+0000 7f02f4834440 -1 bluestore(td/osd-rep-recov-eio.sh/1) _read_fsid unparsable uuid 2024-09-18T15:46:00.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/1/keyring 2024-09-18T15:46:00.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:46:00.711 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T15:46:00.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T15:46:00.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:46:01.225 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T15:46:01.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T15:46:01.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:46:01.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:46:01.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:46:01.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:46:01.246 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:46:01.245+0000 7fa593e4f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:46:01.321 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:46:01.320+0000 7fa593e4f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:46:01.322 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:46:01.322+0000 7fa593e4f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:46:01.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T15:46:01.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:46:01.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:46:01.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:46:01.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:46:01.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:01.611 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:46:01.612 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:46:01.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:01.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:46:01.851 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:46:01.850+0000 7fa593e4f440 -1 Falling back to public interface 2024-09-18T15:46:01.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:02.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:02.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:02.982 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:46:02.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:46:02.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:02.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:46:03.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:04.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:04.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:04.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:46:04.403 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:46:04.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:04.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:46:04.449 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:46:04.448+0000 7fa593e4f440 -1 osd.1 0 log_to_monitors true 2024-09-18T15:46:04.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:05.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:05.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:05.800 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:46:05.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:46:05.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:05.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:46:06.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:07.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:07.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:07.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:46:07.225 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:46:07.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:07.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:46:07.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:08.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:08.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:08.605 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:46:08.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:46:08.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:08.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:46:08.989 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:09.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:09.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:09.990 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:46:09.991 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:46:09.991 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:09.991 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:46:10.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:11.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:11.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:11.533 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:46:11.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:46:11.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:11.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:46:11.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:12.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:12.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:12.928 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:46:12.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:46:12.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:12.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:46:13.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:14.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:14.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:14.359 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:46:14.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:46:14.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:14.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:46:14.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:15.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:15.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:15.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:46:15.737 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:46:15.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:15.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:46:16.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:17.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:17.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:17.200 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:46:17.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:46:17.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:17.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:46:17.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:18.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:18.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:18.882 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:46:18.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:46:18.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:18.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:46:19.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:20.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:20.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:20.676 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:46:20.676 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:46:20.676 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:20.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:46:21.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:22.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:22.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:22.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:46:22.456 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:46:22.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:22.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:46:23.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:24.491 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:24.491 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:24.491 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:46:24.491 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:46:24.492 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:24.492 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:46:24.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:25.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:25.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:25.911 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:46:25.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:46:25.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:25.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:46:26.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:27.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:27.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:27.315 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T15:46:27.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:46:27.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:27.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:46:27.848 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/412840,v1:127.0.0.1:6811/412840] [v2:127.0.0.1:6812/412840,v1:127.0.0.1:6813/412840] exists,up 2bd4d6c9-1410-458b-b4f9-9ad419246843 2024-09-18T15:46:27.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:46:27.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:46:27.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:46:27.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T15:46:27.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 2 2024-09-18T15:46:27.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:46:27.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:46:27.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T15:46:27.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:46:27.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/2 2024-09-18T15:46:27.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-09-18T15:46:27.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:46:27.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:46:27.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:46:27.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2024-09-18T15:46:27.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2024-09-18T15:46:27.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:46:27.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:46:27.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-18T15:46:27.854 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:46:27.855 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:46:27.855 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:46:27.855 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:46:27.855 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:46:27.855 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:46:27.855 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:46:27.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:46:27.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:46:27.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:46:27.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:46:27.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-18T15:46:27.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-18T15:46:27.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:46:27.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:46:27.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:46:27.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:46:27.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:46:27.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:46:27.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/2 2024-09-18T15:46:27.856 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:46:27.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cf3f67da-ea12-4f0f-b986-9cd11c766b0a 2024-09-18T15:46:27.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 cf3f67da-ea12-4f0f-b986-9cd11c766b0a' 2024-09-18T15:46:27.857 INFO:tasks.workunit.client.0.smithi047.stdout:add osd2 cf3f67da-ea12-4f0f-b986-9cd11c766b0a 2024-09-18T15:46:27.857 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:46:27.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDT9epmSReQMxAA9HjNfdawiJLxb6bogbOz9Q== 2024-09-18T15:46:27.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDT9epmSReQMxAA9HjNfdawiJLxb6bogbOz9Q=="}' 2024-09-18T15:46:27.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cf3f67da-ea12-4f0f-b986-9cd11c766b0a -i td/osd-rep-recov-eio.sh/2/new.json 2024-09-18T15:46:28.468 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:46:28.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/2/new.json 2024-09-18T15:46:28.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDT9epmSReQMxAA9HjNfdawiJLxb6bogbOz9Q== --osd-uuid cf3f67da-ea12-4f0f-b986-9cd11c766b0a 2024-09-18T15:46:28.503 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:46:28.502+0000 7f9cd8a08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:46:28.505 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:46:28.504+0000 7f9cd8a08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:46:28.506 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:46:28.505+0000 7f9cd8a08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:46:28.507 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:46:28.506+0000 7f9cd8a08440 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-18T15:46:28.507 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:46:28.506+0000 7f9cd8a08440 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-18T15:46:28.507 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:46:28.506+0000 7f9cd8a08440 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-18T15:46:28.600 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:46:28.600+0000 7f9cd8a08440 -1 bluestore(td/osd-rep-recov-eio.sh/2) _read_fsid unparsable uuid 2024-09-18T15:46:34.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/2/keyring 2024-09-18T15:46:34.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:46:34.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T15:46:34.884 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd2 key to auth repository 2024-09-18T15:46:34.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:46:35.386 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T15:46:35.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T15:46:35.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:46:35.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:46:35.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:46:35.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:46:35.407 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:46:35.407+0000 7f7f82b27440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:46:35.433 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:46:35.432+0000 7f7f82b27440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:46:35.434 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:46:35.433+0000 7f7f82b27440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:46:35.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T15:46:35.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:46:35.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:46:35.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:46:35.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:46:35.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:35.803 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:46:35.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:46:35.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:35.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:46:35.967 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:46:35.966+0000 7f7f82b27440 -1 Falling back to public interface 2024-09-18T15:46:36.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:37.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:37.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:37.190 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:46:37.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:46:37.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:37.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:46:37.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:38.538 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:46:38.537+0000 7f7f82b27440 -1 osd.2 0 log_to_monitors true 2024-09-18T15:46:38.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:38.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:38.564 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:46:38.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:46:38.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:38.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:46:39.021 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:40.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:40.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:40.023 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:46:40.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:46:40.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:40.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:46:40.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:41.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:41.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:41.428 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:46:41.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:46:41.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:41.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:46:41.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:42.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:42.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:42.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:46:42.808 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:46:42.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:42.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:46:43.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:44.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:44.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:44.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:46:44.344 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:46:44.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:44.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:46:44.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:45.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:45.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:45.756 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:46:45.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:46:45.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:45.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:46:46.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:47.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:47.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:47.142 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:46:47.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:46:47.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:47.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:46:47.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:48.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:48.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:48.619 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:46:48.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:46:48.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:48.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:46:48.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:49.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:49.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:49.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:46:49.996 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:46:49.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:49.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:46:50.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:51.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:51.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:51.512 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:46:51.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:46:51.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:51.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:46:51.956 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:52.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:52.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:52.958 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:46:52.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:46:52.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:52.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:46:53.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:54.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:54.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:54.688 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:46:54.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:46:54.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:54.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:46:55.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:56.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:56.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:56.577 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:46:56.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:46:56.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:56.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:46:57.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:46:58.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:46:58.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:46:58.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:46:58.640 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:46:58.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:46:58.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:46:59.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:00.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:00.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:00.146 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:47:00.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:47:00.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:00.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:47:01.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:02.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:02.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:02.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:47:02.113 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T15:47:02.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:02.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:47:02.531 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/414577,v1:127.0.0.1:6819/414577] [v2:127.0.0.1:6820/414577,v1:127.0.0.1:6821/414577] exists,up cf3f67da-ea12-4f0f-b986-9cd11c766b0a 2024-09-18T15:47:02.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:47:02.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:47:02.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:47:02.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T15:47:02.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 3 2024-09-18T15:47:02.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:47:02.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:47:02.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-18T15:47:02.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:47:02.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/3 2024-09-18T15:47:02.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-09-18T15:47:02.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:47:02.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:47:02.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:47:02.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/3' 2024-09-18T15:47:02.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/3/journal' 2024-09-18T15:47:02.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:47:02.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:47:02.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-18T15:47:02.537 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:47:02.538 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:47:02.538 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:47:02.538 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:47:02.538 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:47:02.538 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:47:02.538 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:47:02.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:47:02.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:47:02.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:47:02.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:47:02.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-18T15:47:02.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-18T15:47:02.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:47:02.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:47:02.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:47:02.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:47:02.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:47:02.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:47:02.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/3 2024-09-18T15:47:02.539 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:47:02.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c89de21f-1ef7-4559-ab6e-b55cc737c522 2024-09-18T15:47:02.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 c89de21f-1ef7-4559-ab6e-b55cc737c522' 2024-09-18T15:47:02.539 INFO:tasks.workunit.client.0.smithi047.stdout:add osd3 c89de21f-1ef7-4559-ab6e-b55cc737c522 2024-09-18T15:47:02.540 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:47:02.549 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD29epmmkmzIBAAB3+/em2EwpoKPnzR5BIf7A== 2024-09-18T15:47:02.549 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD29epmmkmzIBAAB3+/em2EwpoKPnzR5BIf7A=="}' 2024-09-18T15:47:02.549 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c89de21f-1ef7-4559-ab6e-b55cc737c522 -i td/osd-rep-recov-eio.sh/3/new.json 2024-09-18T15:47:03.460 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:47:03.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/3/new.json 2024-09-18T15:47:03.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/3 --osd-journal=td/osd-rep-recov-eio.sh/3/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD29epmmkmzIBAAB3+/em2EwpoKPnzR5BIf7A== --osd-uuid c89de21f-1ef7-4559-ab6e-b55cc737c522 2024-09-18T15:47:03.498 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:47:03.497+0000 7f677d74a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:47:03.518 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:47:03.517+0000 7f677d74a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:47:03.519 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:47:03.518+0000 7f677d74a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:47:03.519 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:47:03.518+0000 7f677d74a440 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-09-18T15:47:03.519 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:47:03.518+0000 7f677d74a440 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-09-18T15:47:03.519 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:47:03.518+0000 7f677d74a440 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-09-18T15:47:03.651 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:47:03.651+0000 7f677d74a440 -1 bluestore(td/osd-rep-recov-eio.sh/3) _read_fsid unparsable uuid 2024-09-18T15:47:09.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/3/keyring 2024-09-18T15:47:09.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:47:09.701 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd3 key to auth repository 2024-09-18T15:47:09.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-18T15:47:09.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:47:10.208 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.3 2024-09-18T15:47:10.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-18T15:47:10.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/3 --osd-journal=td/osd-rep-recov-eio.sh/3/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:47:10.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:47:10.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:47:10.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:47:10.230 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:47:10.229+0000 7f4efcc0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:47:10.232 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:47:10.231+0000 7f4efcc0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:47:10.233 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:47:10.232+0000 7f4efcc0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:47:10.648 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-18T15:47:10.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:47:10.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T15:47:10.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:47:10.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:47:10.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:10.649 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:47:10.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:47:10.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:10.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:47:10.749 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:47:10.748+0000 7f4efcc0b440 -1 Falling back to public interface 2024-09-18T15:47:11.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:12.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:12.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:12.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:47:12.011 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:47:12.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:12.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:47:12.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:13.248 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:47:13.247+0000 7f4efcc0b440 -1 osd.3 0 log_to_monitors true 2024-09-18T15:47:13.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:13.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:13.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:47:13.418 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:47:13.419 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:13.419 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:47:14.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:15.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:15.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:15.018 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:47:15.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:47:15.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:15.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:47:15.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:16.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:16.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:16.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:47:16.617 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:47:16.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:16.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:47:17.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:18.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:18.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:18.027 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:47:18.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:47:18.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:18.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:47:18.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:19.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:19.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:19.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:47:19.468 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:47:19.469 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:19.469 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:47:19.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:20.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:20.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:20.962 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:47:20.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:47:20.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:20.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:47:21.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:22.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:22.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:22.360 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:47:22.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:47:22.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:22.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:47:22.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:23.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:23.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:23.757 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:47:23.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:47:23.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:23.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:47:24.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:25.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:25.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:25.176 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:47:25.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:47:25.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:25.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:47:25.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:26.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:26.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:26.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:47:26.559 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:47:26.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:26.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:47:27.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:28.676 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:28.676 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:28.676 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:47:28.676 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:47:28.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:28.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:47:29.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:30.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:30.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:30.064 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:47:30.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:47:30.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:30.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:47:30.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:31.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:31.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:31.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:47:31.973 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:47:31.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:31.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:47:32.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:33.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:33.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:33.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:47:33.571 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:47:33.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:33.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:47:34.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:35.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:35.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:35.455 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:47:35.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:47:35.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:35.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:47:36.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:37.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:37.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:37.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:47:37.227 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T15:47:37.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:37.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:47:37.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:47:38.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:47:38.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:47:38.898 INFO:tasks.workunit.client.0.smithi047.stdout:18 2024-09-18T15:47:38.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T15:47:38.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:47:38.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T15:47:39.389 INFO:tasks.workunit.client.0.smithi047.stdout:osd.3 up in weight 1 up_from 32 up_thru 34 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/416333,v1:127.0.0.1:6827/416333] [v2:127.0.0.1:6828/416333,v1:127.0.0.1:6829/416333] exists,up c89de21f-1ef7-4559-ab6e-b55cc737c522 2024-09-18T15:47:39.390 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:47:39.390 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:47:39.390 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:47:39.390 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:55: setup_osds: wait_for_clean 2024-09-18T15:47:39.390 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:47:39.390 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:47:39.390 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:47:39.391 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:47:39.391 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:47:39.391 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:47:39.391 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:47:39.391 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:47:39.391 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:47:39.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:47:39.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:47:39.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:47:39.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:47:39.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:47:39.480 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:47:40.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:47:40.066 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:47:40.066 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T15:47:40.067 INFO:tasks.workunit.client.0.smithi047.stderr:3' 2024-09-18T15:47:40.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:47:40.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:47:40.067 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:47:40.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803805 2024-09-18T15:47:40.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803805 2024-09-18T15:47:40.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803805' 2024-09-18T15:47:40.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:47:40.251 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:47:40.477 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574870 2024-09-18T15:47:40.477 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574870 2024-09-18T15:47:40.477 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803805 1-55834574870' 2024-09-18T15:47:40.477 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:47:40.477 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:47:40.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345935 2024-09-18T15:47:40.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345935 2024-09-18T15:47:40.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803805 1-55834574870 2-85899345935' 2024-09-18T15:47:40.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:47:40.584 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:47:40.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=137438953480 2024-09-18T15:47:40.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 137438953480 2024-09-18T15:47:40.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803805 1-55834574870 2-85899345935 3-137438953480' 2024-09-18T15:47:40.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:47:40.689 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803805 2024-09-18T15:47:40.689 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:47:40.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:47:40.690 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803805 2024-09-18T15:47:40.690 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:47:40.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803805 2024-09-18T15:47:40.691 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 25769803805 2024-09-18T15:47:40.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803805' 2024-09-18T15:47:40.691 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:47:41.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803803 -lt 25769803805 2024-09-18T15:47:41.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:47:42.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:47:42.059 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:47:42.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803805 -lt 25769803805 2024-09-18T15:47:42.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:47:42.473 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574870 2024-09-18T15:47:42.473 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:47:42.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:47:42.474 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574870 2024-09-18T15:47:42.475 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:47:42.476 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574870 2024-09-18T15:47:42.476 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574870' 2024-09-18T15:47:42.476 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 55834574870 2024-09-18T15:47:42.476 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:47:42.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574870 -lt 55834574870 2024-09-18T15:47:42.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:47:42.836 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345935 2024-09-18T15:47:42.837 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:47:42.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:47:42.838 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345935 2024-09-18T15:47:42.838 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:47:42.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345935 2024-09-18T15:47:42.839 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 85899345935 2024-09-18T15:47:42.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345935' 2024-09-18T15:47:42.839 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:47:43.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345935 -lt 85899345935 2024-09-18T15:47:43.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:47:43.207 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-137438953480 2024-09-18T15:47:43.207 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:47:43.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:47:43.208 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-137438953480 2024-09-18T15:47:43.208 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:47:43.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=137438953480 2024-09-18T15:47:43.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 137438953480' 2024-09-18T15:47:43.210 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 137438953480 2024-09-18T15:47:43.210 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:47:43.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 137438953480 -lt 137438953480 2024-09-18T15:47:43.618 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:47:43.618 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:47:43.618 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:47:44.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-18T15:47:44.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:47:44.126 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:47:44.126 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:47:44.126 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:47:44.126 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:47:44.126 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:47:44.126 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:47:44.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-18T15:47:44.522 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:47:44.522 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:47:44.522 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:47:45.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-18T15:47:45.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:47:45.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:47:45.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:186: TEST_rados_repair_warning: local poolname=pool-rep 2024-09-18T15:47:45.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:187: TEST_rados_repair_warning: create_pool pool-rep 1 1 2024-09-18T15:47:45.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-rep 1 1 2024-09-18T15:47:45.614 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'pool-rep' already exists 2024-09-18T15:47:45.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:47:46.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:188: TEST_rados_repair_warning: wait_for_clean 2024-09-18T15:47:46.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:47:46.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:47:46.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:47:46.627 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:47:46.627 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:47:46.627 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:47:46.627 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:47:46.628 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:47:46.628 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:47:46.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:47:46.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:47:46.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:47:46.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:47:46.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:47:46.729 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:47:47.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:47:47.124 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:47:47.125 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T15:47:47.125 INFO:tasks.workunit.client.0.smithi047.stderr:3' 2024-09-18T15:47:47.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:47:47.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:47:47.125 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:47:47.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803808 2024-09-18T15:47:47.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803808 2024-09-18T15:47:47.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803808' 2024-09-18T15:47:47.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:47:47.229 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:47:47.343 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574873 2024-09-18T15:47:47.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574873 2024-09-18T15:47:47.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803808 1-55834574873' 2024-09-18T15:47:47.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:47:47.344 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:47:47.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345938 2024-09-18T15:47:47.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345938 2024-09-18T15:47:47.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803808 1-55834574873 2-85899345938' 2024-09-18T15:47:47.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:47:47.450 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:47:47.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=137438953483 2024-09-18T15:47:47.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 137438953483 2024-09-18T15:47:47.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803808 1-55834574873 2-85899345938 3-137438953483' 2024-09-18T15:47:47.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:47:47.556 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803808 2024-09-18T15:47:47.556 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:47:47.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:47:47.558 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803808 2024-09-18T15:47:47.558 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:47:47.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803808 2024-09-18T15:47:47.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803808' 2024-09-18T15:47:47.559 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 25769803808 2024-09-18T15:47:47.559 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:47:47.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803808 -lt 25769803808 2024-09-18T15:47:47.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:47:47.966 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574873 2024-09-18T15:47:47.966 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:47:47.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:47:47.967 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574873 2024-09-18T15:47:47.967 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:47:47.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574873 2024-09-18T15:47:47.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574873' 2024-09-18T15:47:47.969 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 55834574873 2024-09-18T15:47:47.969 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:47:48.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574873 -lt 55834574873 2024-09-18T15:47:48.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:47:48.369 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345938 2024-09-18T15:47:48.369 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:47:48.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:47:48.370 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345938 2024-09-18T15:47:48.371 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:47:48.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345938 2024-09-18T15:47:48.372 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 85899345938 2024-09-18T15:47:48.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345938' 2024-09-18T15:47:48.372 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:47:48.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345938 -lt 85899345938 2024-09-18T15:47:48.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:47:48.795 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-137438953483 2024-09-18T15:47:48.795 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:47:48.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:47:48.796 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-137438953483 2024-09-18T15:47:48.796 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:47:48.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=137438953483 2024-09-18T15:47:48.797 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 137438953483 2024-09-18T15:47:48.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 137438953483' 2024-09-18T15:47:48.798 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:47:49.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 137438953483 -lt 137438953483 2024-09-18T15:47:49.160 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:47:49.160 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:47:49.160 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:47:49.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-18T15:47:49.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:47:49.657 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:47:49.657 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:47:49.657 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:47:49.658 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:47:49.658 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:47:49.658 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:47:50.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-09-18T15:47:50.012 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:47:50.012 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:47:50.013 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:47:50.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-09-18T15:47:50.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:47:50.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:47:50.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:190: TEST_rados_repair_warning: local poolname=pool-rep 2024-09-18T15:47:50.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:191: TEST_rados_repair_warning: local objbase=obj-warn 2024-09-18T15:47:50.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:192: TEST_rados_repair_warning: local inject=eio 2024-09-18T15:47:50.505 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: seq 1 11 2024-09-18T15:47:50.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-18T15:47:50.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-1 2024-09-18T15:47:50.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:47:50.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-18T15:47:50.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-1 2024-09-18T15:47:50.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:50.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-18T15:47:50.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:50.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-18T15:47:50.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:50.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-18T15:47:50.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:50.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-18T15:47:50.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-1 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:47:50.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-1 td/osd-rep-recov-eio.sh 0 2024-09-18T15:47:50.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:47:50.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:47:50.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:47:50.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:47:50.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:47:50.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:47:50.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-1 2024-09-18T15:47:50.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:47:50.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:47:50.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:47:50.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T15:47:50.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:47:50.650 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-1 2024-09-18T15:47:50.650 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:47:50.650 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-1 2024-09-18T15:47:50.650 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-1 2024-09-18T15:47:50.650 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:47:51.015 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:47:51.016 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:47:51.016 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:47:51.016 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:47:51.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:47:51.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:47:51.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-18T15:47:51.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:47:51.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:47:51.016 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-18T15:47:51.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:47:51.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-18T15:47:51.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:47:51.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:47:51.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:47:51.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:47:51.018 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:47:51.018 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:47:51.019 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:47:51.019 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:47:51.019 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:47:51.019 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:47:51.019 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:47:51.019 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:47:51.019 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:47:51.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:47:51.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:47:51.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:47:51.102 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-18T15:47:51.102 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:47:51.103 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:47:51.103 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:47:51.103 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:47:51.103 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:47:51.103 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:47:51.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:47:51.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.3.asok injectdataerr pool-rep obj-warn-1 2024-09-18T15:47:51.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-1 2024-09-18T15:47:51.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:47:51.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-18T15:47:51.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-1 2024-09-18T15:47:51.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-18T15:47:51.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-18T15:47:51.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-18T15:47:51.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-1 td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:47:51.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:47:51.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:47:51.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-18T15:47:51.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-2 2024-09-18T15:47:51.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:47:51.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-18T15:47:51.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-2 2024-09-18T15:47:51.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:51.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-18T15:47:51.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:51.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-18T15:47:51.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:51.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-18T15:47:51.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:51.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-18T15:47:51.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-2 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:47:51.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-2 td/osd-rep-recov-eio.sh 0 2024-09-18T15:47:51.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:47:51.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:47:51.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:47:51.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:47:51.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:47:51.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:47:51.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-2 2024-09-18T15:47:51.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:47:51.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:47:51.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:47:51.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T15:47:51.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:47:51.370 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-2 2024-09-18T15:47:51.371 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:47:51.371 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-2 2024-09-18T15:47:51.371 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-2 2024-09-18T15:47:51.371 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:47:51.829 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:47:51.829 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:47:51.830 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:47:51.830 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:47:51.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:47:51.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:47:51.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-18T15:47:51.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:47:51.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:47:51.830 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-18T15:47:51.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:47:51.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-18T15:47:51.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:47:51.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:47:51.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:47:51.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:47:51.832 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:47:51.832 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:47:51.832 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:47:51.832 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:47:51.832 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:47:51.832 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:47:51.832 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:47:51.833 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:47:51.833 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:47:51.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:47:51.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:47:51.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:47:51.916 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-18T15:47:51.916 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:47:51.916 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:47:51.916 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:47:51.916 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:47:51.916 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:47:51.916 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:47:51.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:47:51.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.3.asok injectdataerr pool-rep obj-warn-2 2024-09-18T15:47:51.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-2 2024-09-18T15:47:52.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:47:52.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-18T15:47:52.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-2 2024-09-18T15:47:52.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-18T15:47:52.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-18T15:47:52.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-18T15:47:52.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-2 td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:47:52.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:47:52.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:47:52.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-18T15:47:52.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-3 2024-09-18T15:47:52.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:47:52.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-18T15:47:52.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-3 2024-09-18T15:47:52.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:52.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-18T15:47:52.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:52.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-18T15:47:52.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:52.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-18T15:47:52.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:52.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-18T15:47:52.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-3 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:47:52.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-3 td/osd-rep-recov-eio.sh 0 2024-09-18T15:47:52.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:47:52.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:47:52.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:47:52.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:47:52.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:47:52.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:47:52.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-3 2024-09-18T15:47:52.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:47:52.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:47:52.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:47:52.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T15:47:52.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:47:52.190 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-3 2024-09-18T15:47:52.190 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:47:52.190 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-3 2024-09-18T15:47:52.190 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-3 2024-09-18T15:47:52.190 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:47:52.556 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:47:52.556 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:47:52.557 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:47:52.557 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:47:52.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:47:52.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:47:52.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-18T15:47:52.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:47:52.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:47:52.557 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-18T15:47:52.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:47:52.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-18T15:47:52.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:47:52.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:47:52.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:47:52.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:47:52.559 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:47:52.559 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:47:52.559 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:47:52.559 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:47:52.559 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:47:52.559 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:47:52.560 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:47:52.560 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:47:52.560 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:47:52.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:47:52.642 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:47:52.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:47:52.643 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-18T15:47:52.643 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:47:52.643 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:47:52.644 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:47:52.644 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:47:52.644 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:47:52.644 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:47:52.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:47:52.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.3.asok injectdataerr pool-rep obj-warn-3 2024-09-18T15:47:52.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-3 2024-09-18T15:47:52.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:47:52.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-18T15:47:52.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-3 2024-09-18T15:47:52.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-18T15:47:52.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-18T15:47:52.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-18T15:47:52.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-3 td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:47:52.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:47:52.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:47:52.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-18T15:47:52.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-4 2024-09-18T15:47:52.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:47:52.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-18T15:47:52.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-4 2024-09-18T15:47:52.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:52.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-18T15:47:52.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:52.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-18T15:47:52.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:52.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-18T15:47:52.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:52.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-18T15:47:52.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-4 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:47:52.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-4 td/osd-rep-recov-eio.sh 0 2024-09-18T15:47:52.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:47:52.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:47:52.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:47:52.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:47:52.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:47:52.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:47:52.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-4 2024-09-18T15:47:52.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:47:52.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:47:52.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:47:52.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T15:47:52.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:47:52.928 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-4 2024-09-18T15:47:52.928 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:47:52.928 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-4 2024-09-18T15:47:52.928 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-4 2024-09-18T15:47:52.928 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:47:53.329 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:47:53.329 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:47:53.329 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:47:53.329 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:47:53.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:47:53.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:47:53.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-18T15:47:53.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:47:53.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:47:53.329 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-18T15:47:53.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:47:53.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-18T15:47:53.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:47:53.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:47:53.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:47:53.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:47:53.332 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:47:53.332 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:47:53.332 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:47:53.332 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:47:53.332 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:47:53.332 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:47:53.332 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:47:53.332 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:47:53.332 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:47:53.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:47:53.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:47:53.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:47:53.415 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-18T15:47:53.415 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:47:53.415 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:47:53.415 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:47:53.415 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:47:53.415 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:47:53.415 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:47:53.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:47:53.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.3.asok injectdataerr pool-rep obj-warn-4 2024-09-18T15:47:53.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-4 2024-09-18T15:47:53.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:47:53.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-18T15:47:53.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-4 2024-09-18T15:47:53.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-18T15:47:53.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-18T15:47:53.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-18T15:47:53.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-4 td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:47:53.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:47:53.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:47:53.527 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-18T15:47:53.527 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-5 2024-09-18T15:47:53.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:47:53.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-18T15:47:53.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-5 2024-09-18T15:47:53.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:53.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-18T15:47:53.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:53.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-18T15:47:53.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:53.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-18T15:47:53.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:53.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-18T15:47:53.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-5 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:47:53.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-5 td/osd-rep-recov-eio.sh 0 2024-09-18T15:47:53.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:47:53.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:47:53.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:47:53.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:47:53.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:47:53.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:47:53.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-5 2024-09-18T15:47:53.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:47:53.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:47:53.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:47:53.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T15:47:53.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:47:53.721 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-5 2024-09-18T15:47:53.721 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:47:53.721 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-5 2024-09-18T15:47:53.722 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-5 2024-09-18T15:47:53.722 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:47:54.123 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:47:54.123 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:47:54.123 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:47:54.123 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:47:54.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:47:54.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:47:54.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-18T15:47:54.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:47:54.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:47:54.124 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-18T15:47:54.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:47:54.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-18T15:47:54.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:47:54.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:47:54.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:47:54.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:47:54.126 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:47:54.126 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:47:54.126 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:47:54.126 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:47:54.126 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:47:54.126 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:47:54.126 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:47:54.126 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:47:54.126 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:47:54.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:47:54.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:47:54.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:47:54.210 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-18T15:47:54.210 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:47:54.210 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:47:54.210 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:47:54.210 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:47:54.210 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:47:54.210 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:47:54.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:47:54.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.3.asok injectdataerr pool-rep obj-warn-5 2024-09-18T15:47:54.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-5 2024-09-18T15:47:54.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:47:54.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-18T15:47:54.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-5 2024-09-18T15:47:54.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-18T15:47:54.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-18T15:47:54.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-18T15:47:54.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-5 td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:47:54.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:47:54.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:47:54.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-18T15:47:54.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-6 2024-09-18T15:47:54.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:47:54.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-18T15:47:54.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-6 2024-09-18T15:47:54.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:54.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-18T15:47:54.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:54.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-18T15:47:54.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:54.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-18T15:47:54.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:54.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-18T15:47:54.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-6 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:47:54.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-6 td/osd-rep-recov-eio.sh 0 2024-09-18T15:47:54.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:47:54.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:47:54.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:47:54.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:47:54.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:47:54.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:47:54.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-6 2024-09-18T15:47:54.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:47:54.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:47:54.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:47:54.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T15:47:54.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:47:54.567 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-6 2024-09-18T15:47:54.567 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:47:54.567 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-6 2024-09-18T15:47:54.567 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-6 2024-09-18T15:47:54.567 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:47:54.932 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:47:54.932 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:47:54.932 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:47:54.932 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:47:54.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:47:54.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:47:54.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-18T15:47:54.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:47:54.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:47:54.933 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-18T15:47:54.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:47:54.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-18T15:47:54.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:47:54.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:47:54.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:47:54.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:47:54.935 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:47:54.935 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:47:54.935 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:47:54.935 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:47:54.935 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:47:54.935 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:47:54.935 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:47:54.935 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:47:54.935 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:47:55.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:47:55.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:47:55.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:47:55.018 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-18T15:47:55.018 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:47:55.018 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:47:55.018 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:47:55.018 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:47:55.018 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:47:55.018 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:47:55.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:47:55.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.3.asok injectdataerr pool-rep obj-warn-6 2024-09-18T15:47:55.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-6 2024-09-18T15:47:55.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:47:55.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-18T15:47:55.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-6 2024-09-18T15:47:55.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-18T15:47:55.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-18T15:47:55.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-18T15:47:55.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-6 td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:47:55.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:47:55.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:47:55.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-18T15:47:55.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-7 2024-09-18T15:47:55.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:47:55.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-18T15:47:55.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-7 2024-09-18T15:47:55.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:55.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-18T15:47:55.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:55.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-18T15:47:55.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:55.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-18T15:47:55.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:55.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-18T15:47:55.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-7 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:47:55.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-7 td/osd-rep-recov-eio.sh 0 2024-09-18T15:47:55.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:47:55.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:47:55.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:47:55.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:47:55.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:47:55.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:47:55.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-7 2024-09-18T15:47:55.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:47:55.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:47:55.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:47:55.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T15:47:55.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:47:55.311 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-7 2024-09-18T15:47:55.311 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:47:55.311 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-7 2024-09-18T15:47:55.311 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-7 2024-09-18T15:47:55.311 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:47:55.755 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:47:55.755 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:47:55.756 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:47:55.756 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:47:55.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:47:55.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:47:55.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-18T15:47:55.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:47:55.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:47:55.756 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-18T15:47:55.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:47:55.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-18T15:47:55.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:47:55.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:47:55.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:47:55.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:47:55.758 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:47:55.758 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:47:55.758 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:47:55.758 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:47:55.758 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:47:55.758 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:47:55.758 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:47:55.759 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:47:55.759 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:47:55.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:47:55.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:47:55.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:47:55.842 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-18T15:47:55.842 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:47:55.842 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:47:55.843 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:47:55.843 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:47:55.843 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:47:55.843 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:47:55.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:47:55.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.3.asok injectdataerr pool-rep obj-warn-7 2024-09-18T15:47:55.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-7 2024-09-18T15:47:55.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:47:55.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-18T15:47:55.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-7 2024-09-18T15:47:55.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-18T15:47:55.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-18T15:47:55.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-18T15:47:55.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-7 td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:47:55.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:47:55.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:47:55.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-18T15:47:55.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-8 2024-09-18T15:47:55.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:47:55.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-18T15:47:55.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-8 2024-09-18T15:47:55.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:55.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-18T15:47:55.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:55.956 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-18T15:47:55.956 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:55.956 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-18T15:47:55.956 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:55.956 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-18T15:47:55.956 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-8 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:47:56.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-8 td/osd-rep-recov-eio.sh 0 2024-09-18T15:47:56.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:47:56.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:47:56.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:47:56.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:47:56.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:47:56.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:47:56.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-8 2024-09-18T15:47:56.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:47:56.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:47:56.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:47:56.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T15:47:56.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:47:56.099 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-8 2024-09-18T15:47:56.099 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:47:56.099 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-8 2024-09-18T15:47:56.099 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-8 2024-09-18T15:47:56.099 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:47:56.485 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:47:56.486 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:47:56.486 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:47:56.486 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:47:56.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:47:56.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:47:56.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-18T15:47:56.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:47:56.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:47:56.486 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-18T15:47:56.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:47:56.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-18T15:47:56.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:47:56.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:47:56.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:47:56.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:47:56.488 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:47:56.488 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:47:56.488 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:47:56.489 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:47:56.489 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:47:56.489 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:47:56.489 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:47:56.489 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:47:56.489 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:47:56.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:47:56.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:47:56.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:47:56.572 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-18T15:47:56.572 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:47:56.572 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:47:56.572 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:47:56.572 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:47:56.572 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:47:56.572 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:47:56.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:47:56.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.3.asok injectdataerr pool-rep obj-warn-8 2024-09-18T15:47:56.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-8 2024-09-18T15:47:56.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:47:56.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-18T15:47:56.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-8 2024-09-18T15:47:56.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-18T15:47:56.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-18T15:47:56.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-18T15:47:56.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-8 td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:47:56.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:47:56.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:47:56.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-18T15:47:56.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-9 2024-09-18T15:47:56.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:47:56.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-18T15:47:56.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-9 2024-09-18T15:47:56.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:56.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-18T15:47:56.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:56.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-18T15:47:56.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:56.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-18T15:47:56.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:56.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-18T15:47:56.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-9 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:47:56.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-9 td/osd-rep-recov-eio.sh 0 2024-09-18T15:47:56.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:47:56.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:47:56.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:47:56.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:47:56.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:47:56.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:47:56.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-9 2024-09-18T15:47:56.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:47:56.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:47:56.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:47:56.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T15:47:56.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:47:56.843 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-9 2024-09-18T15:47:56.843 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:47:56.843 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-9 2024-09-18T15:47:56.843 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-9 2024-09-18T15:47:56.843 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:47:57.227 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:47:57.228 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:47:57.228 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:47:57.228 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:47:57.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:47:57.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:47:57.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-18T15:47:57.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:47:57.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:47:57.228 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-18T15:47:57.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:47:57.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-18T15:47:57.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:47:57.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:47:57.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:47:57.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:47:57.230 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:47:57.230 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:47:57.230 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:47:57.230 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:47:57.230 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:47:57.231 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:47:57.231 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:47:57.231 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:47:57.231 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:47:57.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:47:57.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:47:57.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:47:57.314 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-18T15:47:57.314 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:47:57.314 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:47:57.314 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:47:57.314 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:47:57.314 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:47:57.314 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:47:57.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:47:57.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.3.asok injectdataerr pool-rep obj-warn-9 2024-09-18T15:47:57.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-9 2024-09-18T15:47:57.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:47:57.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-18T15:47:57.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-9 2024-09-18T15:47:57.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-18T15:47:57.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-18T15:47:57.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-18T15:47:57.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-9 td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:47:57.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:47:57.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:47:57.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-18T15:47:57.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-10 2024-09-18T15:47:57.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:47:57.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-18T15:47:57.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-10 2024-09-18T15:47:57.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:57.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-18T15:47:57.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:57.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-18T15:47:57.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:57.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-18T15:47:57.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:57.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-18T15:47:57.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-10 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:47:57.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-10 td/osd-rep-recov-eio.sh 0 2024-09-18T15:47:57.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:47:57.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:47:57.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:47:57.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:47:57.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:47:57.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:47:57.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-10 2024-09-18T15:47:57.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:47:57.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:47:57.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:47:57.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T15:47:57.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:47:57.571 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-10 2024-09-18T15:47:57.571 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:47:57.571 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-10 2024-09-18T15:47:57.571 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-10 2024-09-18T15:47:57.571 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:47:57.941 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:47:57.941 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:47:57.941 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:47:57.941 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:47:57.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:47:57.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:47:57.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-18T15:47:57.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:47:57.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:47:57.942 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-18T15:47:57.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:47:57.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-18T15:47:57.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:47:57.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:47:57.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:47:57.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:47:57.944 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:47:57.944 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:47:57.944 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:47:57.944 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:47:57.944 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:47:57.944 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:47:57.945 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:47:57.945 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:47:57.945 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:47:58.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:47:58.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:47:58.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:47:58.027 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-18T15:47:58.028 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:47:58.028 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:47:58.028 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:47:58.028 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:47:58.028 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:47:58.029 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:47:58.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:47:58.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.3.asok injectdataerr pool-rep obj-warn-10 2024-09-18T15:47:58.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-10 2024-09-18T15:47:58.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:47:58.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-18T15:47:58.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-10 2024-09-18T15:47:58.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-18T15:47:58.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-18T15:47:58.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-18T15:47:58.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-10 td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:47:58.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:47:58.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:47:58.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-18T15:47:58.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-11 2024-09-18T15:47:58.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:47:58.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-18T15:47:58.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-11 2024-09-18T15:47:58.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:58.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-18T15:47:58.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:58.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-18T15:47:58.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:58.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-18T15:47:58.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:47:58.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-18T15:47:58.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-11 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:47:58.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-11 td/osd-rep-recov-eio.sh 0 2024-09-18T15:47:58.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:47:58.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:47:58.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:47:58.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:47:58.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:47:58.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:47:58.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-11 2024-09-18T15:47:58.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:47:58.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:47:58.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:47:58.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T15:47:58.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:47:58.290 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-11 2024-09-18T15:47:58.290 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:47:58.290 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-11 2024-09-18T15:47:58.291 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-11 2024-09-18T15:47:58.291 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:47:58.656 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:47:58.656 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:47:58.657 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:47:58.657 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:47:58.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:47:58.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:47:58.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-18T15:47:58.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:47:58.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:47:58.657 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-18T15:47:58.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:47:58.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-18T15:47:58.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:47:58.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:47:58.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:47:58.660 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:47:58.660 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:47:58.660 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:47:58.660 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:47:58.660 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:47:58.660 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:47:58.660 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:47:58.660 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:47:58.660 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:47:58.660 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:47:58.741 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:47:58.741 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:47:58.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:47:58.743 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-18T15:47:58.743 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:47:58.743 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:47:58.743 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:47:58.743 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:47:58.743 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:47:58.743 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:47:58.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:47:58.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.3.asok injectdataerr pool-rep obj-warn-11 2024-09-18T15:47:58.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-11 2024-09-18T15:47:58.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:47:58.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-18T15:47:58.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-11 2024-09-18T15:47:58.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-18T15:47:58.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-18T15:47:58.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-18T15:47:58.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-11 td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:47:58.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:47:58.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:47:58.855 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:200: TEST_rados_repair_warning: get_pg pool-rep obj-warn-1 2024-09-18T15:47:58.855 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=pool-rep 2024-09-18T15:47:58.855 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj-warn-1 2024-09-18T15:47:58.855 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map pool-rep obj-warn-1 2024-09-18T15:47:58.855 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-18T15:47:59.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:200: TEST_rados_repair_warning: local pgid=2.0 2024-09-18T15:47:59.262 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:202: TEST_rados_repair_warning: get_osds pool-rep obj-warn-1 2024-09-18T15:47:59.262 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:47:59.262 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-1 2024-09-18T15:47:59.263 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-1 2024-09-18T15:47:59.263 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:47:59.631 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:47:59.631 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:47:59.631 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:47:59.631 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:47:59.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:202: TEST_rados_repair_warning: object_osds=('3' '1' '0') 2024-09-18T15:47:59.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:202: TEST_rados_repair_warning: local object_osds 2024-09-18T15:47:59.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:203: TEST_rados_repair_warning: local primary=3 2024-09-18T15:47:59.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:204: TEST_rados_repair_warning: local bad_peer=1 2024-09-18T15:47:59.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:206: TEST_rados_repair_warning: wait_for_clean 2024-09-18T15:47:59.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:47:59.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:47:59.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:47:59.632 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:47:59.632 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:47:59.632 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:47:59.632 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:47:59.632 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:47:59.632 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:47:59.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:47:59.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:47:59.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:47:59.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:47:59.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:47:59.721 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:48:00.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:48:00.146 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:48:00.146 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T15:48:00.146 INFO:tasks.workunit.client.0.smithi047.stderr:3' 2024-09-18T15:48:00.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:48:00.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:48:00.146 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:48:00.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803813 2024-09-18T15:48:00.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803813 2024-09-18T15:48:00.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803813' 2024-09-18T15:48:00.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:48:00.252 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:48:00.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574878 2024-09-18T15:48:00.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574878 2024-09-18T15:48:00.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803813 1-55834574878' 2024-09-18T15:48:00.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:48:00.397 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:48:00.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345943 2024-09-18T15:48:00.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345943 2024-09-18T15:48:00.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803813 1-55834574878 2-85899345943' 2024-09-18T15:48:00.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:48:00.503 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:48:00.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=137438953488 2024-09-18T15:48:00.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 137438953488 2024-09-18T15:48:00.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803813 1-55834574878 2-85899345943 3-137438953488' 2024-09-18T15:48:00.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:48:00.609 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803813 2024-09-18T15:48:00.609 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:48:00.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:48:00.610 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803813 2024-09-18T15:48:00.610 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:48:00.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803813 2024-09-18T15:48:00.611 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 25769803813 2024-09-18T15:48:00.612 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803813' 2024-09-18T15:48:00.612 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:48:01.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803811 -lt 25769803813 2024-09-18T15:48:01.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:48:02.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:48:02.017 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:48:02.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803813 -lt 25769803813 2024-09-18T15:48:02.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:48:02.451 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574878 2024-09-18T15:48:02.451 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:48:02.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:48:02.453 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574878 2024-09-18T15:48:02.453 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:48:02.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574878 2024-09-18T15:48:02.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574878' 2024-09-18T15:48:02.454 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 55834574878 2024-09-18T15:48:02.454 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:48:02.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574878 -lt 55834574878 2024-09-18T15:48:02.860 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:48:02.860 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345943 2024-09-18T15:48:02.860 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:48:02.861 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:48:02.861 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345943 2024-09-18T15:48:02.861 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:48:02.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345943 2024-09-18T15:48:02.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345943' 2024-09-18T15:48:02.863 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 85899345943 2024-09-18T15:48:02.864 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:48:03.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345943 -lt 85899345943 2024-09-18T15:48:03.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:48:03.226 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-137438953488 2024-09-18T15:48:03.227 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:48:03.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:48:03.228 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-137438953488 2024-09-18T15:48:03.228 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:48:03.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=137438953488 2024-09-18T15:48:03.229 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 137438953488 2024-09-18T15:48:03.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 137438953488' 2024-09-18T15:48:03.229 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:48:03.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 137438953488 -lt 137438953488 2024-09-18T15:48:03.593 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:48:03.593 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:48:03.593 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:48:04.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-18T15:48:04.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:48:04.086 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:48:04.086 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:48:04.086 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:48:04.086 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:48:04.087 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:48:04.087 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:48:04.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-09-18T15:48:04.441 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:48:04.441 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:48:04.441 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:48:04.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-09-18T15:48:04.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:48:04.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:48:04.930 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:207: TEST_rados_repair_warning: ceph pg 2.0 query 2024-09-18T15:48:04.930 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:207: TEST_rados_repair_warning: jq .info.stats.stat_sum.num_objects_repaired 2024-09-18T15:48:05.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:207: TEST_rados_repair_warning: COUNT=11 2024-09-18T15:48:05.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:208: TEST_rados_repair_warning: test 11 = 11 2024-09-18T15:48:05.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:209: TEST_rados_repair_warning: flush_pg_stats 2024-09-18T15:48:05.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:48:05.042 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:48:05.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:48:05.468 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:48:05.468 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T15:48:05.468 INFO:tasks.workunit.client.0.smithi047.stderr:3' 2024-09-18T15:48:05.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:48:05.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:48:05.468 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:48:05.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803816 2024-09-18T15:48:05.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803816 2024-09-18T15:48:05.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803816' 2024-09-18T15:48:05.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:48:05.575 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:48:05.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574881 2024-09-18T15:48:05.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574881 2024-09-18T15:48:05.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803816 1-55834574881' 2024-09-18T15:48:05.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:48:05.681 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:48:05.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345946 2024-09-18T15:48:05.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345946 2024-09-18T15:48:05.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803816 1-55834574881 2-85899345946' 2024-09-18T15:48:05.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:48:05.799 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:48:05.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=137438953491 2024-09-18T15:48:05.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 137438953491 2024-09-18T15:48:05.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803816 1-55834574881 2-85899345946 3-137438953491' 2024-09-18T15:48:05.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:48:05.905 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803816 2024-09-18T15:48:05.905 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:48:05.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:48:05.906 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803816 2024-09-18T15:48:05.906 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:48:05.907 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803816 2024-09-18T15:48:05.907 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 25769803816 2024-09-18T15:48:05.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803816' 2024-09-18T15:48:05.908 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:48:06.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803816 -lt 25769803816 2024-09-18T15:48:06.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:48:06.303 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574881 2024-09-18T15:48:06.303 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:48:06.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:48:06.305 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574881 2024-09-18T15:48:06.305 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:48:06.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574881 2024-09-18T15:48:06.306 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 55834574881 2024-09-18T15:48:06.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574881' 2024-09-18T15:48:06.306 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:48:06.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574881 -lt 55834574881 2024-09-18T15:48:06.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:48:06.679 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345946 2024-09-18T15:48:06.679 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:48:06.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:48:06.680 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345946 2024-09-18T15:48:06.680 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:48:06.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345946 2024-09-18T15:48:06.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345946' 2024-09-18T15:48:06.682 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 85899345946 2024-09-18T15:48:06.682 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:48:07.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345944 -lt 85899345946 2024-09-18T15:48:07.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:48:08.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:48:08.091 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:48:08.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345946 -lt 85899345946 2024-09-18T15:48:08.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:48:08.541 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-137438953491 2024-09-18T15:48:08.541 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:48:08.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:48:08.542 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-137438953491 2024-09-18T15:48:08.542 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:48:08.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=137438953491 2024-09-18T15:48:08.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 137438953491' 2024-09-18T15:48:08.543 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 137438953491 2024-09-18T15:48:08.544 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:48:08.949 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 137438953491 -lt 137438953491 2024-09-18T15:48:08.950 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:210: TEST_rados_repair_warning: ceph pg dump --format=json-pretty 2024-09-18T15:48:08.950 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:210: TEST_rados_repair_warning: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-09-18T15:48:09.301 INFO:tasks.workunit.client.0.smithi047.stderr:dumped all 2024-09-18T15:48:09.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:210: TEST_rados_repair_warning: COUNT=11 2024-09-18T15:48:09.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:211: TEST_rados_repair_warning: test 11 = 11 2024-09-18T15:48:09.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:213: TEST_rados_repair_warning: ceph health 2024-09-18T15:48:09.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:213: TEST_rados_repair_warning: grep -q 'Too many repaired reads on 1 OSDs' 2024-09-18T15:48:09.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:214: TEST_rados_repair_warning: ceph health detail 2024-09-18T15:48:09.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:214: TEST_rados_repair_warning: grep -q 'osd.3 had 11 reads repaired' 2024-09-18T15:48:10.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:216: TEST_rados_repair_warning: ceph health mute OSD_TOO_MANY_REPAIRS 2024-09-18T15:48:11.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:217: TEST_rados_repair_warning: set -o pipefail 2024-09-18T15:48:11.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:219: TEST_rados_repair_warning: ceph health 2024-09-18T15:48:11.013 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:219: TEST_rados_repair_warning: grep -q 'Too many repaired reads on 1 OSDs' 2024-09-18T15:48:11.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:220: TEST_rados_repair_warning: set +o pipefail 2024-09-18T15:48:11.457 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: seq 1 11 2024-09-18T15:48:11.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-18T15:48:11.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-1 td/osd-rep-recov-eio.sh 0 2024-09-18T15:48:11.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:48:11.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:48:11.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:48:11.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:48:11.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:48:11.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:48:11.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-1 2024-09-18T15:48:11.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:48:11.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:11.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:48:11.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T15:48:11.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:48:11.460 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-1 2024-09-18T15:48:11.460 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:48:11.460 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-1 2024-09-18T15:48:11.460 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-1 2024-09-18T15:48:11.460 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:48:11.831 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:48:11.832 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:48:11.832 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:48:11.832 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:48:11.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:48:11.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:48:11.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-18T15:48:11.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:48:11.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:48:11.832 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-18T15:48:11.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:48:11.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-18T15:48:11.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:48:11.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:48:11.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:48:11.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:48:11.834 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:48:11.834 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:48:11.835 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:48:11.835 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:48:11.835 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:11.835 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:11.835 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:11.835 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:48:11.835 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:48:11.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:48:11.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:48:11.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:48:11.918 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-18T15:48:11.918 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:48:11.918 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:48:11.918 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:11.918 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:11.918 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:11.918 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:48:11.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:48:11.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.3.asok injectdataerr pool-rep obj-warn-1 2024-09-18T15:48:12.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-1 td/osd-rep-recov-eio.sh 1 2024-09-18T15:48:12.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:48:12.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:48:12.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:48:12.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:48:12.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:48:12.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:48:12.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-1 2024-09-18T15:48:12.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:48:12.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:12.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:48:12.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-18T15:48:12.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:48:12.003 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-1 2024-09-18T15:48:12.003 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:48:12.003 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-1 2024-09-18T15:48:12.004 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-1 2024-09-18T15:48:12.004 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:48:12.393 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:48:12.393 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:48:12.393 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:48:12.393 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:48:12.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:48:12.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:48:12.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-18T15:48:12.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:48:12.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:48:12.394 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-18T15:48:12.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:48:12.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-18T15:48:12.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:48:12.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-18T15:48:12.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:48:12.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:48:12.396 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:48:12.396 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-18T15:48:12.396 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:48:12.396 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:48:12.396 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:12.396 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:12.396 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:12.396 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.1.asok 2024-09-18T15:48:12.397 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:48:12.477 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:48:12.477 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:48:12.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:48:12.479 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-18T15:48:12.479 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:48:12.479 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:48:12.479 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:12.479 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:12.479 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:12.479 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.1.asok 2024-09-18T15:48:12.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:48:12.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.1.asok injectdataerr pool-rep obj-warn-1 2024-09-18T15:48:12.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-09-18T15:48:12.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:48:12.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:48:12.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-18T15:48:12.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-18T15:48:12.563 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:48:12.564 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:48:12.564 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:48:12.564 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:48:12.564 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:12.564 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:12.564 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:12.564 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:48:12.564 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-18T15:48:12.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:48:12.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-1 2024-09-18T15:48:12.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:12.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-18T15:48:12.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-1 2024-09-18T15:48:12.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-18T15:48:12.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-18T15:48:12.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-18T15:48:12.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-1 td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:48:12.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:48:12.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:48:12.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-18T15:48:12.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-2 td/osd-rep-recov-eio.sh 0 2024-09-18T15:48:12.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:48:12.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:48:12.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:48:12.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:48:12.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:48:12.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:48:12.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-2 2024-09-18T15:48:12.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:48:12.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:12.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:48:12.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T15:48:12.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:48:12.718 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-2 2024-09-18T15:48:12.719 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:48:12.719 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-2 2024-09-18T15:48:12.719 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-2 2024-09-18T15:48:12.719 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:48:13.121 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:48:13.121 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:48:13.122 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:48:13.122 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:48:13.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:48:13.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:48:13.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-18T15:48:13.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:48:13.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:48:13.122 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-18T15:48:13.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:48:13.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-18T15:48:13.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:48:13.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:48:13.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:48:13.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:48:13.124 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:48:13.124 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:48:13.125 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:48:13.125 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:48:13.125 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:13.125 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:13.125 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:13.125 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:48:13.125 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:48:13.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:48:13.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:48:13.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:48:13.207 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-18T15:48:13.207 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:48:13.207 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:48:13.207 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:13.207 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:13.208 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:13.208 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:48:13.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:48:13.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.3.asok injectdataerr pool-rep obj-warn-2 2024-09-18T15:48:13.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-2 td/osd-rep-recov-eio.sh 1 2024-09-18T15:48:13.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:48:13.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:48:13.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:48:13.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:48:13.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:48:13.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:48:13.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-2 2024-09-18T15:48:13.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:48:13.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:13.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:48:13.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-18T15:48:13.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:48:13.293 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-2 2024-09-18T15:48:13.293 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:48:13.293 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-2 2024-09-18T15:48:13.293 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-2 2024-09-18T15:48:13.293 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:48:13.710 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:48:13.710 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:48:13.711 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:48:13.711 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:48:13.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:48:13.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:48:13.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-18T15:48:13.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:48:13.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:48:13.711 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-18T15:48:13.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:48:13.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-18T15:48:13.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:48:13.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-18T15:48:13.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:48:13.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:48:13.713 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:48:13.713 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-18T15:48:13.713 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:48:13.713 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:48:13.713 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:13.713 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:13.713 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:13.714 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.1.asok 2024-09-18T15:48:13.714 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:48:13.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:48:13.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:48:13.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:48:13.797 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-18T15:48:13.797 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:48:13.797 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:48:13.797 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:13.797 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:13.797 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:13.797 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.1.asok 2024-09-18T15:48:13.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:48:13.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.1.asok injectdataerr pool-rep obj-warn-2 2024-09-18T15:48:13.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-09-18T15:48:13.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:48:13.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:48:13.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-18T15:48:13.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-18T15:48:13.882 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:48:13.882 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:48:13.882 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:48:13.882 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:48:13.882 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:13.882 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:13.882 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:13.882 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:48:13.882 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-18T15:48:13.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:48:13.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-2 2024-09-18T15:48:13.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:13.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-18T15:48:13.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-2 2024-09-18T15:48:13.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-18T15:48:13.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-18T15:48:13.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-18T15:48:13.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-2 td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:48:14.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:48:14.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:48:14.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-18T15:48:14.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-3 td/osd-rep-recov-eio.sh 0 2024-09-18T15:48:14.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:48:14.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:48:14.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:48:14.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:48:14.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:48:14.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:48:14.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-3 2024-09-18T15:48:14.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:48:14.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:14.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:48:14.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T15:48:14.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:48:14.017 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-3 2024-09-18T15:48:14.017 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:48:14.017 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-3 2024-09-18T15:48:14.017 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-3 2024-09-18T15:48:14.017 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:48:14.409 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:48:14.409 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:48:14.409 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:48:14.409 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:48:14.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:48:14.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:48:14.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-18T15:48:14.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:48:14.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:48:14.410 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-18T15:48:14.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:48:14.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-18T15:48:14.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:48:14.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:48:14.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:48:14.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:48:14.412 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:48:14.412 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:48:14.412 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:48:14.412 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:48:14.412 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:14.412 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:14.412 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:14.412 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:48:14.412 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:48:14.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:48:14.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:48:14.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:48:14.495 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-18T15:48:14.495 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:48:14.495 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:48:14.495 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:14.495 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:14.495 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:14.496 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:48:14.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:48:14.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.3.asok injectdataerr pool-rep obj-warn-3 2024-09-18T15:48:14.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-3 td/osd-rep-recov-eio.sh 1 2024-09-18T15:48:14.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:48:14.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:48:14.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:48:14.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:48:14.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:48:14.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:48:14.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-3 2024-09-18T15:48:14.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:48:14.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:14.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:48:14.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-18T15:48:14.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:48:14.580 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-3 2024-09-18T15:48:14.580 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:48:14.580 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-3 2024-09-18T15:48:14.580 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-3 2024-09-18T15:48:14.581 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:48:14.969 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:48:14.969 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:48:14.969 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:48:14.969 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:48:14.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:48:14.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:48:14.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-18T15:48:14.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:48:14.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:48:14.970 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-18T15:48:14.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:48:14.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-18T15:48:14.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:48:14.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-18T15:48:14.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:48:14.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:48:14.972 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:48:14.972 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-18T15:48:14.972 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:48:14.972 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:48:14.972 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:14.972 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:14.972 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:14.972 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.1.asok 2024-09-18T15:48:14.972 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:48:15.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:48:15.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:48:15.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:48:15.055 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-18T15:48:15.055 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:48:15.055 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:48:15.055 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:15.055 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:15.055 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:15.055 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.1.asok 2024-09-18T15:48:15.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:48:15.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.1.asok injectdataerr pool-rep obj-warn-3 2024-09-18T15:48:15.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-09-18T15:48:15.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:48:15.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:48:15.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-18T15:48:15.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-18T15:48:15.139 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:48:15.139 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:48:15.139 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:48:15.140 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:48:15.140 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:15.140 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:15.140 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:15.141 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:48:15.141 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-18T15:48:15.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:48:15.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-3 2024-09-18T15:48:15.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:15.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-18T15:48:15.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-3 2024-09-18T15:48:15.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-18T15:48:15.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-18T15:48:15.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-18T15:48:15.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-3 td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:48:15.265 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:48:15.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:48:15.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-18T15:48:15.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-4 td/osd-rep-recov-eio.sh 0 2024-09-18T15:48:15.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:48:15.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:48:15.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:48:15.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:48:15.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:48:15.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:48:15.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-4 2024-09-18T15:48:15.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:48:15.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:15.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:48:15.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T15:48:15.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:48:15.269 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-4 2024-09-18T15:48:15.269 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:48:15.269 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-4 2024-09-18T15:48:15.269 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-4 2024-09-18T15:48:15.269 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:48:15.680 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:48:15.680 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:48:15.680 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:48:15.680 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:48:15.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:48:15.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:48:15.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-18T15:48:15.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:48:15.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:48:15.681 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-18T15:48:15.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:48:15.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-18T15:48:15.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:48:15.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:48:15.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:48:15.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:48:15.683 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:48:15.683 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:48:15.683 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:48:15.683 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:48:15.683 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:15.683 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:15.683 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:15.683 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:48:15.683 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:48:15.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:48:15.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:48:15.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:48:15.767 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-18T15:48:15.767 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:48:15.767 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:48:15.767 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:15.767 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:15.767 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:15.767 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:48:15.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:48:15.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.3.asok injectdataerr pool-rep obj-warn-4 2024-09-18T15:48:15.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-4 td/osd-rep-recov-eio.sh 1 2024-09-18T15:48:15.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:48:15.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:48:15.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:48:15.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:48:15.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:48:15.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:48:15.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-4 2024-09-18T15:48:15.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:48:15.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:15.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:48:15.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-18T15:48:15.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:48:15.852 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-4 2024-09-18T15:48:15.852 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:48:15.852 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-4 2024-09-18T15:48:15.852 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-4 2024-09-18T15:48:15.852 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:48:16.215 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:48:16.215 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:48:16.215 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:48:16.215 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:48:16.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:48:16.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:48:16.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-18T15:48:16.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:48:16.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:48:16.215 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-18T15:48:16.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:48:16.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-18T15:48:16.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:48:16.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-18T15:48:16.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:48:16.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:48:16.217 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:48:16.218 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-18T15:48:16.218 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:48:16.218 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:48:16.218 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:16.218 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:16.218 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:16.218 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.1.asok 2024-09-18T15:48:16.218 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:48:16.299 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:48:16.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:48:16.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:48:16.301 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-18T15:48:16.301 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:48:16.301 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:48:16.301 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:16.301 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:16.301 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:16.301 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.1.asok 2024-09-18T15:48:16.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:48:16.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.1.asok injectdataerr pool-rep obj-warn-4 2024-09-18T15:48:16.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-09-18T15:48:16.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:48:16.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:48:16.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-18T15:48:16.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-18T15:48:16.386 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:48:16.386 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:48:16.386 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:48:16.386 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:48:16.386 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:16.386 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:16.386 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:16.387 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:48:16.387 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-18T15:48:16.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:48:16.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-4 2024-09-18T15:48:16.469 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:16.469 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-18T15:48:16.469 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-4 2024-09-18T15:48:16.469 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-18T15:48:16.469 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-18T15:48:16.469 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-18T15:48:16.469 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-4 td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:48:16.514 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:48:16.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:48:16.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-18T15:48:16.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-5 td/osd-rep-recov-eio.sh 0 2024-09-18T15:48:16.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:48:16.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:48:16.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:48:16.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:48:16.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:48:16.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:48:16.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-5 2024-09-18T15:48:16.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:48:16.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:16.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:48:16.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T15:48:16.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:48:16.518 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-5 2024-09-18T15:48:16.518 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:48:16.518 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-5 2024-09-18T15:48:16.518 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-5 2024-09-18T15:48:16.518 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:48:16.918 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:48:16.918 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:48:16.918 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:48:16.918 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:48:16.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:48:16.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:48:16.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-18T15:48:16.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:48:16.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:48:16.918 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-18T15:48:16.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:48:16.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-18T15:48:16.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:48:16.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:48:16.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:48:16.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:48:16.921 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:48:16.921 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:48:16.921 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:48:16.921 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:48:16.921 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:16.921 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:16.921 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:16.921 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:48:16.921 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:48:17.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:48:17.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:48:17.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:48:17.004 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-18T15:48:17.004 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:48:17.004 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:48:17.004 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:17.004 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:17.004 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:17.004 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:48:17.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:48:17.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.3.asok injectdataerr pool-rep obj-warn-5 2024-09-18T15:48:17.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-5 td/osd-rep-recov-eio.sh 1 2024-09-18T15:48:17.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:48:17.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:48:17.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:48:17.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:48:17.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:48:17.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:48:17.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-5 2024-09-18T15:48:17.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:48:17.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:17.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:48:17.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-18T15:48:17.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:48:17.090 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-5 2024-09-18T15:48:17.090 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:48:17.090 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-5 2024-09-18T15:48:17.090 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-5 2024-09-18T15:48:17.090 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:48:17.455 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:48:17.456 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:48:17.456 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:48:17.456 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:48:17.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:48:17.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:48:17.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-18T15:48:17.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:48:17.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:48:17.456 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-18T15:48:17.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:48:17.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-18T15:48:17.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:48:17.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-18T15:48:17.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:48:17.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:48:17.458 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:48:17.458 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-18T15:48:17.458 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:48:17.459 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:48:17.459 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:17.459 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:17.459 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:17.459 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.1.asok 2024-09-18T15:48:17.459 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:48:17.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:48:17.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:48:17.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:48:17.542 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-18T15:48:17.542 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:48:17.542 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:48:17.542 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:17.543 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:17.543 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:17.543 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.1.asok 2024-09-18T15:48:17.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:48:17.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.1.asok injectdataerr pool-rep obj-warn-5 2024-09-18T15:48:17.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-09-18T15:48:17.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:48:17.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:48:17.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-18T15:48:17.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-18T15:48:17.627 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:48:17.627 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:48:17.627 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:48:17.627 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:48:17.627 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:17.627 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:17.627 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:17.627 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:48:17.627 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-18T15:48:17.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:48:17.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-5 2024-09-18T15:48:17.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:17.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-18T15:48:17.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-5 2024-09-18T15:48:17.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-18T15:48:17.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-18T15:48:17.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-18T15:48:17.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-5 td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:48:17.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:48:17.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:48:17.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-18T15:48:17.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-6 td/osd-rep-recov-eio.sh 0 2024-09-18T15:48:17.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:48:17.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:48:17.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:48:17.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:48:17.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:48:17.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:48:17.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-6 2024-09-18T15:48:17.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:48:17.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:17.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:48:17.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T15:48:17.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:48:17.752 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-6 2024-09-18T15:48:17.752 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:48:17.752 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-6 2024-09-18T15:48:17.752 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-6 2024-09-18T15:48:17.752 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:48:18.163 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:48:18.163 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:48:18.163 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:48:18.163 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:48:18.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:48:18.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:48:18.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-18T15:48:18.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:48:18.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:48:18.164 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-18T15:48:18.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:48:18.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-18T15:48:18.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:48:18.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:48:18.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:48:18.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:48:18.166 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:48:18.166 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:48:18.166 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:48:18.166 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:48:18.166 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:18.166 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:18.166 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:18.166 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:48:18.166 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:48:18.248 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:48:18.248 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:48:18.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:48:18.249 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-18T15:48:18.249 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:48:18.250 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:48:18.250 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:18.250 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:18.250 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:18.250 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:48:18.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:48:18.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.3.asok injectdataerr pool-rep obj-warn-6 2024-09-18T15:48:18.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-6 td/osd-rep-recov-eio.sh 1 2024-09-18T15:48:18.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:48:18.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:48:18.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:48:18.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:48:18.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:48:18.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:48:18.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-6 2024-09-18T15:48:18.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:48:18.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:18.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:48:18.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-18T15:48:18.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:48:18.335 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-6 2024-09-18T15:48:18.335 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:48:18.336 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-6 2024-09-18T15:48:18.336 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-6 2024-09-18T15:48:18.336 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:48:18.742 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:48:18.743 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:48:18.743 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:48:18.743 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:48:18.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:48:18.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:48:18.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-18T15:48:18.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:48:18.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:48:18.743 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-18T15:48:18.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:48:18.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-18T15:48:18.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:48:18.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-18T15:48:18.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:48:18.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:48:18.745 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:48:18.745 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-18T15:48:18.746 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:48:18.746 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:48:18.746 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:18.746 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:18.746 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:18.746 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.1.asok 2024-09-18T15:48:18.746 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:48:18.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:48:18.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:48:18.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:48:18.829 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-18T15:48:18.829 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:48:18.829 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:48:18.829 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:18.829 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:18.829 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:18.829 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.1.asok 2024-09-18T15:48:18.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:48:18.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.1.asok injectdataerr pool-rep obj-warn-6 2024-09-18T15:48:18.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-09-18T15:48:18.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:48:18.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:48:18.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-18T15:48:18.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-18T15:48:18.914 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:48:18.914 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:48:18.914 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:48:18.914 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:48:18.915 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:18.915 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:18.915 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:18.915 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:48:18.915 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-18T15:48:18.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:48:18.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-6 2024-09-18T15:48:18.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:18.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-18T15:48:18.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-6 2024-09-18T15:48:18.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-18T15:48:18.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-18T15:48:18.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-18T15:48:18.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-6 td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:48:19.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:48:19.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:48:19.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-18T15:48:19.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-7 td/osd-rep-recov-eio.sh 0 2024-09-18T15:48:19.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:48:19.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:48:19.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:48:19.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:48:19.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:48:19.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:48:19.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-7 2024-09-18T15:48:19.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:48:19.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:19.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:48:19.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T15:48:19.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:48:19.036 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-7 2024-09-18T15:48:19.036 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:48:19.036 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-7 2024-09-18T15:48:19.036 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-7 2024-09-18T15:48:19.036 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:48:19.426 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:48:19.426 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:48:19.426 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:48:19.426 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:48:19.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:48:19.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:48:19.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-18T15:48:19.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:48:19.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:48:19.427 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-18T15:48:19.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:48:19.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-18T15:48:19.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:48:19.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:48:19.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:48:19.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:48:19.429 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:48:19.429 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:48:19.429 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:48:19.429 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:48:19.429 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:19.429 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:19.430 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:19.430 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:48:19.430 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:48:19.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:48:19.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:48:19.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:48:19.512 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-18T15:48:19.512 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:48:19.512 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:48:19.512 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:19.512 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:19.513 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:19.513 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:48:19.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:48:19.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.3.asok injectdataerr pool-rep obj-warn-7 2024-09-18T15:48:19.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-7 td/osd-rep-recov-eio.sh 1 2024-09-18T15:48:19.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:48:19.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:48:19.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:48:19.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:48:19.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:48:19.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:48:19.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-7 2024-09-18T15:48:19.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:48:19.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:19.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:48:19.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-18T15:48:19.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:48:19.598 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-7 2024-09-18T15:48:19.598 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:48:19.598 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-7 2024-09-18T15:48:19.598 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-7 2024-09-18T15:48:19.598 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:48:19.993 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:48:19.994 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:48:19.994 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:48:19.994 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:48:19.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:48:19.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:48:19.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-18T15:48:19.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:48:19.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:48:19.994 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-18T15:48:19.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:48:19.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-18T15:48:19.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:48:19.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-18T15:48:19.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:48:19.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:48:19.997 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:48:19.997 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-18T15:48:19.997 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:48:19.997 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:48:19.997 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:19.997 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:19.997 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:19.997 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.1.asok 2024-09-18T15:48:19.997 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:48:20.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:48:20.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:48:20.080 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:48:20.080 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-18T15:48:20.080 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:48:20.080 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:48:20.080 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:20.080 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:20.080 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:20.080 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.1.asok 2024-09-18T15:48:20.080 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:48:20.080 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.1.asok injectdataerr pool-rep obj-warn-7 2024-09-18T15:48:20.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-09-18T15:48:20.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:48:20.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:48:20.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-18T15:48:20.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-18T15:48:20.165 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:48:20.165 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:48:20.166 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:48:20.166 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:48:20.166 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:20.166 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:20.166 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:20.166 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:48:20.166 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-18T15:48:20.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:48:20.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-7 2024-09-18T15:48:20.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:20.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-18T15:48:20.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-7 2024-09-18T15:48:20.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-18T15:48:20.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-18T15:48:20.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-18T15:48:20.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-7 td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:48:20.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:48:20.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:48:20.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-18T15:48:20.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-8 td/osd-rep-recov-eio.sh 0 2024-09-18T15:48:20.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:48:20.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:48:20.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:48:20.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:48:20.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:48:20.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:48:20.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-8 2024-09-18T15:48:20.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:48:20.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:20.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:48:20.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T15:48:20.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:48:20.310 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-8 2024-09-18T15:48:20.310 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:48:20.310 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-8 2024-09-18T15:48:20.310 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-8 2024-09-18T15:48:20.311 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:48:20.723 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:48:20.723 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:48:20.723 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:48:20.723 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:48:20.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:48:20.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:48:20.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-18T15:48:20.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:48:20.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:48:20.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-18T15:48:20.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:48:20.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-18T15:48:20.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:48:20.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:48:20.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:48:20.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:48:20.726 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:48:20.726 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:48:20.726 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:48:20.727 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:48:20.727 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:20.727 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:20.727 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:20.727 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:48:20.727 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:48:20.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:48:20.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:48:20.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:48:20.809 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-18T15:48:20.809 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:48:20.809 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:48:20.809 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:20.809 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:20.809 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:20.809 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:48:20.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:48:20.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.3.asok injectdataerr pool-rep obj-warn-8 2024-09-18T15:48:20.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-8 td/osd-rep-recov-eio.sh 1 2024-09-18T15:48:20.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:48:20.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:48:20.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:48:20.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:48:20.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:48:20.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:48:20.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-8 2024-09-18T15:48:20.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:48:20.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:20.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:48:20.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-18T15:48:20.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:48:20.894 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-8 2024-09-18T15:48:20.894 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:48:20.894 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-8 2024-09-18T15:48:20.894 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-8 2024-09-18T15:48:20.894 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:48:21.298 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:48:21.298 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:48:21.298 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:48:21.298 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:48:21.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:48:21.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:48:21.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-18T15:48:21.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:48:21.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:48:21.298 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-18T15:48:21.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:48:21.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-18T15:48:21.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:48:21.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-18T15:48:21.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:48:21.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:48:21.300 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:48:21.301 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-18T15:48:21.301 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:48:21.301 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:48:21.301 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:21.301 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:21.301 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:21.301 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.1.asok 2024-09-18T15:48:21.301 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:48:21.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:48:21.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:48:21.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:48:21.384 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-18T15:48:21.384 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:48:21.384 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:48:21.384 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:21.384 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:21.384 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:21.384 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.1.asok 2024-09-18T15:48:21.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:48:21.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.1.asok injectdataerr pool-rep obj-warn-8 2024-09-18T15:48:21.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-09-18T15:48:21.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:48:21.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:48:21.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-18T15:48:21.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-18T15:48:21.469 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:48:21.469 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:48:21.469 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:48:21.469 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:48:21.469 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:21.469 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:21.469 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:21.469 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:48:21.469 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-18T15:48:21.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:48:21.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-8 2024-09-18T15:48:21.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:21.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-18T15:48:21.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-8 2024-09-18T15:48:21.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-18T15:48:21.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-18T15:48:21.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-18T15:48:21.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-8 td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:48:21.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:48:21.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:48:21.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-18T15:48:21.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-9 td/osd-rep-recov-eio.sh 0 2024-09-18T15:48:21.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:48:21.592 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:48:21.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:48:21.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:48:21.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:48:21.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:48:21.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-9 2024-09-18T15:48:21.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:48:21.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:21.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:48:21.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T15:48:21.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:48:21.593 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-9 2024-09-18T15:48:21.593 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:48:21.593 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-9 2024-09-18T15:48:21.593 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-9 2024-09-18T15:48:21.594 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:48:21.967 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:48:21.967 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:48:21.967 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:48:21.967 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:48:21.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:48:21.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:48:21.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-18T15:48:21.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:48:21.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:48:21.968 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-18T15:48:21.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:48:21.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-18T15:48:21.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:48:21.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:48:21.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:48:21.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:48:21.970 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:48:21.970 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:48:21.970 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:48:21.970 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:48:21.970 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:21.970 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:21.970 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:21.970 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:48:21.971 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:48:22.052 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:48:22.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:48:22.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:48:22.054 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-18T15:48:22.054 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:48:22.054 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:48:22.054 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:22.054 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:22.054 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:22.054 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:48:22.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:48:22.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.3.asok injectdataerr pool-rep obj-warn-9 2024-09-18T15:48:22.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-9 td/osd-rep-recov-eio.sh 1 2024-09-18T15:48:22.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:48:22.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:48:22.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:48:22.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:48:22.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:48:22.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:48:22.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-9 2024-09-18T15:48:22.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:48:22.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:22.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:48:22.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-18T15:48:22.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:48:22.139 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-9 2024-09-18T15:48:22.139 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:48:22.140 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-9 2024-09-18T15:48:22.140 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-9 2024-09-18T15:48:22.140 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:48:22.579 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:48:22.579 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:48:22.579 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:48:22.579 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:48:22.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:48:22.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:48:22.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-18T15:48:22.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:48:22.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:48:22.579 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-18T15:48:22.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:48:22.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-18T15:48:22.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:48:22.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-18T15:48:22.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:48:22.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:48:22.582 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:48:22.582 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-18T15:48:22.582 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:48:22.582 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:48:22.582 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:22.582 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:22.582 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:22.582 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.1.asok 2024-09-18T15:48:22.582 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:48:22.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:48:22.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:48:22.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:48:22.665 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-18T15:48:22.666 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:48:22.666 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:48:22.666 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:22.666 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:22.666 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:22.666 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.1.asok 2024-09-18T15:48:22.666 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:48:22.666 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.1.asok injectdataerr pool-rep obj-warn-9 2024-09-18T15:48:22.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-09-18T15:48:22.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:48:22.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:48:22.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-18T15:48:22.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-18T15:48:22.750 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:48:22.750 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:48:22.750 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:48:22.750 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:48:22.751 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:22.751 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:22.751 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:22.751 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:48:22.751 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-18T15:48:22.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:48:22.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-9 2024-09-18T15:48:22.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:22.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-18T15:48:22.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-9 2024-09-18T15:48:22.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-18T15:48:22.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-18T15:48:22.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-18T15:48:22.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-9 td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:48:22.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:48:22.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:48:22.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-18T15:48:22.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-10 td/osd-rep-recov-eio.sh 0 2024-09-18T15:48:22.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:48:22.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:48:22.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:48:22.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:48:22.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:48:22.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:48:22.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-10 2024-09-18T15:48:22.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:48:22.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:22.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:48:22.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T15:48:22.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:48:22.868 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-10 2024-09-18T15:48:22.868 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:48:22.868 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-10 2024-09-18T15:48:22.868 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-10 2024-09-18T15:48:22.868 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:48:23.280 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:48:23.280 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:48:23.280 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:48:23.281 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:48:23.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:48:23.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:48:23.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-18T15:48:23.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:48:23.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:48:23.281 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-18T15:48:23.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:48:23.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-18T15:48:23.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:48:23.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:48:23.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:48:23.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:48:23.282 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:48:23.283 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:48:23.283 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:48:23.283 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:48:23.284 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:23.284 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:23.284 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:23.284 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:48:23.284 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:48:23.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:48:23.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:48:23.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:48:23.367 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-18T15:48:23.367 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:48:23.367 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:48:23.367 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:23.367 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:23.367 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:23.367 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:48:23.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:48:23.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.3.asok injectdataerr pool-rep obj-warn-10 2024-09-18T15:48:23.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-10 td/osd-rep-recov-eio.sh 1 2024-09-18T15:48:23.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:48:23.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:48:23.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:48:23.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:48:23.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:48:23.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:48:23.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-10 2024-09-18T15:48:23.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:48:23.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:23.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:48:23.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-18T15:48:23.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:48:23.452 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-10 2024-09-18T15:48:23.452 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:48:23.452 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-10 2024-09-18T15:48:23.453 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-10 2024-09-18T15:48:23.453 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:48:23.830 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:48:23.830 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:48:23.830 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:48:23.831 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:48:23.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:48:23.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:48:23.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-18T15:48:23.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:48:23.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:48:23.831 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-18T15:48:23.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:48:23.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-18T15:48:23.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:48:23.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-18T15:48:23.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:48:23.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:48:23.833 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:48:23.833 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-18T15:48:23.833 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:48:23.833 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:48:23.833 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:23.833 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:23.833 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:23.834 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.1.asok 2024-09-18T15:48:23.834 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:48:23.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:48:23.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:48:23.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:48:23.917 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-18T15:48:23.917 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:48:23.917 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:48:23.917 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:23.917 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:23.917 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:23.917 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.1.asok 2024-09-18T15:48:23.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:48:23.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.1.asok injectdataerr pool-rep obj-warn-10 2024-09-18T15:48:24.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-09-18T15:48:24.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:48:24.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:48:24.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-18T15:48:24.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-18T15:48:24.002 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:48:24.002 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:48:24.002 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:48:24.002 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:48:24.003 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:24.003 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:24.003 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:24.003 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:48:24.003 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-18T15:48:24.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:48:24.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-10 2024-09-18T15:48:24.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:24.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-18T15:48:24.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-10 2024-09-18T15:48:24.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-18T15:48:24.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-18T15:48:24.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-18T15:48:24.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-10 td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:48:24.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:48:24.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:48:24.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-18T15:48:24.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-11 td/osd-rep-recov-eio.sh 0 2024-09-18T15:48:24.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:48:24.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:48:24.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:48:24.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:48:24.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:48:24.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:48:24.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-11 2024-09-18T15:48:24.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:48:24.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:24.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:48:24.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T15:48:24.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:48:24.160 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-11 2024-09-18T15:48:24.160 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:48:24.160 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-11 2024-09-18T15:48:24.160 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-11 2024-09-18T15:48:24.160 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:48:24.619 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:48:24.619 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:48:24.620 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:48:24.620 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:48:24.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:48:24.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:48:24.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-18T15:48:24.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:48:24.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:48:24.620 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-18T15:48:24.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:48:24.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-18T15:48:24.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:48:24.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:48:24.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:48:24.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:48:24.622 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:48:24.622 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:48:24.622 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:48:24.622 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:48:24.623 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:24.623 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:24.623 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:24.623 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:48:24.623 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:48:24.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:48:24.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:48:24.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:48:24.706 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-18T15:48:24.706 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:48:24.706 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:48:24.706 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:24.706 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:24.706 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:24.706 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:48:24.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:48:24.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.3.asok injectdataerr pool-rep obj-warn-11 2024-09-18T15:48:24.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-11 td/osd-rep-recov-eio.sh 1 2024-09-18T15:48:24.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:48:24.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:48:24.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:48:24.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:48:24.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-18T15:48:24.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:48:24.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-11 2024-09-18T15:48:24.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:48:24.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:24.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:48:24.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-18T15:48:24.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:48:24.791 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-11 2024-09-18T15:48:24.791 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-18T15:48:24.791 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-11 2024-09-18T15:48:24.792 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-11 2024-09-18T15:48:24.792 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:48:25.196 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T15:48:25.196 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:48:25.196 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:48:25.196 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T15:48:25.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-18T15:48:25.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:48:25.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-18T15:48:25.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:48:25.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:48:25.197 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-18T15:48:25.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:48:25.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-18T15:48:25.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:48:25.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-18T15:48:25.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:48:25.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:48:25.199 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:48:25.199 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-18T15:48:25.199 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:48:25.199 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:48:25.199 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:25.199 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:25.199 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:25.199 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.1.asok 2024-09-18T15:48:25.200 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:48:25.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:48:25.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:48:25.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:48:25.281 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-18T15:48:25.282 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:48:25.282 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:48:25.283 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:25.283 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:25.283 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:25.283 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.1.asok 2024-09-18T15:48:25.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:48:25.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.1.asok injectdataerr pool-rep obj-warn-11 2024-09-18T15:48:25.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-09-18T15:48:25.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:48:25.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T15:48:25.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-18T15:48:25.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-18T15:48:25.367 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:48:25.367 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T15:48:25.368 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T15:48:25.368 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T15:48:25.368 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:25.368 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:25.368 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:25.368 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.3.asok 2024-09-18T15:48:25.368 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-18T15:48:25.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:48:25.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-11 2024-09-18T15:48:25.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:25.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-18T15:48:25.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-11 2024-09-18T15:48:25.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-18T15:48:25.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-18T15:48:25.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-18T15:48:25.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-11 td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:48:25.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:48:25.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-18T15:48:25.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:231: TEST_rados_repair_warning: wait_for_clean 2024-09-18T15:48:25.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:48:25.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:48:25.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:48:25.496 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:48:25.497 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:48:25.497 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:48:25.497 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:48:25.497 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:48:25.497 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:48:25.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:48:25.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:48:25.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:48:25.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:48:25.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:48:25.587 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:48:25.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:48:25.961 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:48:25.961 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T15:48:25.961 INFO:tasks.workunit.client.0.smithi047.stderr:3' 2024-09-18T15:48:25.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:48:25.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:48:25.961 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:48:26.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803822 2024-09-18T15:48:26.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803822 2024-09-18T15:48:26.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803822' 2024-09-18T15:48:26.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:48:26.067 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:48:26.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574887 2024-09-18T15:48:26.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574887 2024-09-18T15:48:26.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803822 1-55834574887' 2024-09-18T15:48:26.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:48:26.222 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:48:26.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345952 2024-09-18T15:48:26.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345952 2024-09-18T15:48:26.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803822 1-55834574887 2-85899345952' 2024-09-18T15:48:26.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:48:26.327 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:48:26.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=137438953497 2024-09-18T15:48:26.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 137438953497 2024-09-18T15:48:26.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803822 1-55834574887 2-85899345952 3-137438953497' 2024-09-18T15:48:26.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:48:26.439 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803822 2024-09-18T15:48:26.439 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:48:26.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:48:26.440 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803822 2024-09-18T15:48:26.440 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:48:26.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803822 2024-09-18T15:48:26.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803822' 2024-09-18T15:48:26.441 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 25769803822 2024-09-18T15:48:26.441 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:48:26.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803820 -lt 25769803822 2024-09-18T15:48:26.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:48:27.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:48:27.833 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:48:28.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803822 -lt 25769803822 2024-09-18T15:48:28.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:48:28.215 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574887 2024-09-18T15:48:28.216 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:48:28.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:48:28.217 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574887 2024-09-18T15:48:28.217 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:48:28.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574887 2024-09-18T15:48:28.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574887' 2024-09-18T15:48:28.218 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 55834574887 2024-09-18T15:48:28.218 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:48:28.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574887 -lt 55834574887 2024-09-18T15:48:28.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:48:28.619 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345952 2024-09-18T15:48:28.619 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:48:28.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:48:28.621 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345952 2024-09-18T15:48:28.621 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:48:28.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345952 2024-09-18T15:48:28.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345952' 2024-09-18T15:48:28.622 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 85899345952 2024-09-18T15:48:28.622 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:48:28.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345952 -lt 85899345952 2024-09-18T15:48:28.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:48:28.984 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-137438953497 2024-09-18T15:48:28.984 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:48:28.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:48:28.985 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-137438953497 2024-09-18T15:48:28.985 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:48:28.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=137438953497 2024-09-18T15:48:28.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 137438953497' 2024-09-18T15:48:28.986 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 137438953497 2024-09-18T15:48:28.987 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:48:29.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 137438953497 -lt 137438953497 2024-09-18T15:48:29.349 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:48:29.349 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:48:29.349 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:48:29.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-18T15:48:29.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:48:29.847 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:48:29.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:48:29.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:48:29.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:48:29.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:48:29.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:48:30.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-09-18T15:48:30.241 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:48:30.242 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:48:30.242 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:48:30.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-09-18T15:48:30.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:48:30.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:48:30.744 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:232: TEST_rados_repair_warning: ceph pg 2.0 query 2024-09-18T15:48:30.744 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:232: TEST_rados_repair_warning: jq .info.stats.stat_sum.num_objects_repaired 2024-09-18T15:48:30.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:232: TEST_rados_repair_warning: COUNT=22 2024-09-18T15:48:30.893 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:233: TEST_rados_repair_warning: expr 11 '*' 2 2024-09-18T15:48:30.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:233: TEST_rados_repair_warning: test 22 = 22 2024-09-18T15:48:30.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:234: TEST_rados_repair_warning: flush_pg_stats 2024-09-18T15:48:30.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:48:30.896 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:48:31.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:48:31.262 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:48:31.262 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T15:48:31.262 INFO:tasks.workunit.client.0.smithi047.stderr:3' 2024-09-18T15:48:31.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:48:31.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:48:31.262 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:48:31.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803825 2024-09-18T15:48:31.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803825 2024-09-18T15:48:31.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803825' 2024-09-18T15:48:31.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:48:31.383 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:48:31.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574890 2024-09-18T15:48:31.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574890 2024-09-18T15:48:31.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803825 1-55834574890' 2024-09-18T15:48:31.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:48:31.488 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:48:31.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345955 2024-09-18T15:48:31.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345955 2024-09-18T15:48:31.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803825 1-55834574890 2-85899345955' 2024-09-18T15:48:31.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:48:31.595 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T15:48:31.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=137438953500 2024-09-18T15:48:31.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 137438953500 2024-09-18T15:48:31.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803825 1-55834574890 2-85899345955 3-137438953500' 2024-09-18T15:48:31.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:48:31.701 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803825 2024-09-18T15:48:31.702 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:48:31.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:48:31.702 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803825 2024-09-18T15:48:31.703 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:48:31.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803825 2024-09-18T15:48:31.704 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 25769803825 2024-09-18T15:48:31.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803825' 2024-09-18T15:48:31.704 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:48:32.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803825 -lt 25769803825 2024-09-18T15:48:32.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:48:32.111 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574890 2024-09-18T15:48:32.111 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:48:32.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:48:32.112 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574890 2024-09-18T15:48:32.112 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:48:32.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574890 2024-09-18T15:48:32.113 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 55834574890 2024-09-18T15:48:32.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574890' 2024-09-18T15:48:32.114 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:48:32.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574890 -lt 55834574890 2024-09-18T15:48:32.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:48:32.521 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345955 2024-09-18T15:48:32.521 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:48:32.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:48:32.522 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345955 2024-09-18T15:48:32.522 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:48:32.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345955 2024-09-18T15:48:32.523 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 85899345955 2024-09-18T15:48:32.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345955' 2024-09-18T15:48:32.523 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:48:32.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345955 -lt 85899345955 2024-09-18T15:48:32.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:48:32.886 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-137438953500 2024-09-18T15:48:32.886 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:48:32.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T15:48:32.887 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-137438953500 2024-09-18T15:48:32.887 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:48:32.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=137438953500 2024-09-18T15:48:32.888 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 137438953500 2024-09-18T15:48:32.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 137438953500' 2024-09-18T15:48:32.889 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:48:33.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 137438953499 -lt 137438953500 2024-09-18T15:48:33.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:48:34.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:48:34.299 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T15:48:34.699 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 137438953500 -lt 137438953500 2024-09-18T15:48:34.700 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:235: TEST_rados_repair_warning: ceph pg dump --format=json-pretty 2024-09-18T15:48:34.700 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:235: TEST_rados_repair_warning: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-09-18T15:48:35.054 INFO:tasks.workunit.client.0.smithi047.stderr:dumped all 2024-09-18T15:48:35.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:235: TEST_rados_repair_warning: COUNT=33 2024-09-18T15:48:35.065 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:236: TEST_rados_repair_warning: expr 11 '*' 3 2024-09-18T15:48:35.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:236: TEST_rados_repair_warning: test 33 = 33 2024-09-18T15:48:35.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:240: TEST_rados_repair_warning: CHECKTIME=10 2024-09-18T15:48:35.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:241: TEST_rados_repair_warning: LOOPS=0 2024-09-18T15:48:35.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:242: TEST_rados_repair_warning: true 2024-09-18T15:48:35.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:244: TEST_rados_repair_warning: sleep 1 2024-09-18T15:48:36.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:245: TEST_rados_repair_warning: ceph health 2024-09-18T15:48:36.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:245: TEST_rados_repair_warning: grep -q 'Too many repaired reads on 2 OSDs' 2024-09-18T15:48:36.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:247: TEST_rados_repair_warning: break 2024-09-18T15:48:36.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:256: TEST_rados_repair_warning: ceph health detail 2024-09-18T15:48:36.529 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:256: TEST_rados_repair_warning: expr 11 '*' 2 2024-09-18T15:48:36.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:256: TEST_rados_repair_warning: grep -q 'osd.3 had 22 reads repaired' 2024-09-18T15:48:37.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:257: TEST_rados_repair_warning: ceph health detail 2024-09-18T15:48:37.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:257: TEST_rados_repair_warning: grep -q 'osd.1 had 11 reads repaired' 2024-09-18T15:48:37.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:259: TEST_rados_repair_warning: delete_pool pool-rep 2024-09-18T15:48:37.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-rep 2024-09-18T15:48:37.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-rep pool-rep --yes-i-really-really-mean-it 2024-09-18T15:48:38.124 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'pool-rep' does not exist 2024-09-18T15:48:38.135 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:43: run: teardown td/osd-rep-recov-eio.sh 2024-09-18T15:48:38.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:38.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:48:38.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-09-18T15:48:38.136 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:48:38.136 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:48:38.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:48:38.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:48:38.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:48:38.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:48:38.245 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:48:38.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:48:38.246 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:48:38.248 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:48:38.248 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:48:38.248 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:48:38.248 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:48:38.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:48:38.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:48:38.249 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:48:38.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:48:38.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:48:38.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-09-18T15:48:38.275 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:48:38.275 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:38.275 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:38.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.397939 2024-09-18T15:48:38.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:48:38.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:48:38.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:35: run: for func in $funcs 2024-09-18T15:48:38.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:36: run: setup td/osd-rep-recov-eio.sh 2024-09-18T15:48:38.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:38.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-rep-recov-eio.sh 2024-09-18T15:48:38.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:38.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:48:38.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-09-18T15:48:38.277 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:48:38.278 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:48:38.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:48:38.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:48:38.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:48:38.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:48:38.279 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:48:38.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:48:38.280 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:48:38.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:48:38.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:48:38.281 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:48:38.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:48:38.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:48:38.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:48:38.282 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:48:38.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:48:38.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:48:38.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-09-18T15:48:38.285 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:48:38.285 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:38.285 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:38.285 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.397939 2024-09-18T15:48:38.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:48:38.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:48:38.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-rep-recov-eio.sh 2024-09-18T15:48:38.287 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T15:48:38.288 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:38.288 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:38.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.397939 2024-09-18T15:48:38.290 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T15:48:38.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T15:48:38.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T15:48:38.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-rep-recov-eio.sh 1' TERM HUP INT 2024-09-18T15:48:38.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:38: run: run_mon td/osd-rep-recov-eio.sh a --mon_osd_warn_num_repaired=10 2024-09-18T15:48:38.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:38.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T15:48:38.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T15:48:38.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T15:48:38.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-rep-recov-eio.sh/a 2024-09-18T15:48:38.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-rep-recov-eio.sh/a --run-dir=td/osd-rep-recov-eio.sh --mon_osd_warn_num_repaired=10 2024-09-18T15:48:38.637 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T15:48:38.637 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:48:38.638 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:48:38.638 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:48:38.638 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:38.638 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:38.638 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:48:38.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-rep-recov-eio.sh/a '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --mon-cluster-log-file=td/osd-rep-recov-eio.sh/log --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_osd_warn_num_repaired=10 2024-09-18T15:48:38.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T15:48:38.881 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T15:48:38.881 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:48:38.881 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:48:38.881 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T15:48:38.881 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T15:48:38.882 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:48:38.883 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:48:38.883 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:48:38.883 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:38.883 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:38.883 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:38.883 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-mon.a.asok 2024-09-18T15:48:38.883 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:48:38.883 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.397939/ceph-mon.a.asok config get fsid 2024-09-18T15:48:38.962 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T15:48:38.962 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:48:38.962 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:48:38.962 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T15:48:38.962 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T15:48:38.962 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:48:38.962 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:48:38.962 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:48:38.962 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:48:38.963 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:38.963 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:38.963 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-mon.a.asok 2024-09-18T15:48:38.963 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:48:38.963 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.397939/ceph-mon.a.asok config get mon_host 2024-09-18T15:48:39.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:39: run: run_mgr td/osd-rep-recov-eio.sh x 2024-09-18T15:48:39.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:39.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T15:48:39.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T15:48:39.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T15:48:39.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-rep-recov-eio.sh/x 2024-09-18T15:48:39.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T15:48:39.886 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T15:48:39.886 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:48:39.886 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:48:39.886 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:48:39.886 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:39.886 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:39.886 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:48:39.887 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:48:39.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-rep-recov-eio.sh/x '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:48:39.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:40: run: ceph osd pool create foo 8 2024-09-18T15:48:40.167 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'foo' already exists 2024-09-18T15:48:40.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:42: run: TEST_rep_backfill_unfound td/osd-rep-recov-eio.sh 2024-09-18T15:48:40.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:264: TEST_rep_backfill_unfound: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:40.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:265: TEST_rep_backfill_unfound: local objname=myobject 2024-09-18T15:48:40.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:266: TEST_rep_backfill_unfound: local lastobj=300 2024-09-18T15:48:40.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:268: TEST_rep_backfill_unfound: local testobj=obj250 2024-09-18T15:48:40.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:270: TEST_rep_backfill_unfound: export CEPH_ARGS 2024-09-18T15:48:40.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:271: TEST_rep_backfill_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-18T15:48:40.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:272: TEST_rep_backfill_unfound: setup_osds 3 2024-09-18T15:48:40.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:48: setup_osds: local count=3 2024-09-18T15:48:40.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:49: setup_osds: shift 2024-09-18T15:48:40.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:50: setup_osds: local type= 2024-09-18T15:48:40.178 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: expr 3 - 1 2024-09-18T15:48:40.178 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: seq 0 2 2024-09-18T15:48:40.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T15:48:40.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 0 2024-09-18T15:48:40.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:48:40.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:48:40.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T15:48:40.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:48:40.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-09-18T15:48:40.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-18T15:48:40.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:48:40.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:48:40.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:48:40.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-09-18T15:48:40.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-09-18T15:48:40.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:48:40.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:48:40.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-18T15:48:40.183 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:48:40.183 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:48:40.183 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:48:40.183 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:48:40.184 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:48:40.184 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:48:40.184 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:48:40.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:48:40.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:48:40.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:48:40.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:48:40.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-18T15:48:40.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-18T15:48:40.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:48:40.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:48:40.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:48:40.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:48:40.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:48:40.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:48:40.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-09-18T15:48:40.185 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:48:40.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=947c566d-835e-4d49-a5f5-82959e55891b 2024-09-18T15:48:40.185 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 947c566d-835e-4d49-a5f5-82959e55891b 2024-09-18T15:48:40.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 947c566d-835e-4d49-a5f5-82959e55891b' 2024-09-18T15:48:40.186 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:48:40.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBY9upmLDDNCxAA5gbnhhGrb5R6H00dRvwvAA== 2024-09-18T15:48:40.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBY9upmLDDNCxAA5gbnhhGrb5R6H00dRvwvAA=="}' 2024-09-18T15:48:40.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 947c566d-835e-4d49-a5f5-82959e55891b -i td/osd-rep-recov-eio.sh/0/new.json 2024-09-18T15:48:40.505 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:48:40.514 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/0/new.json 2024-09-18T15:48:40.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBY9upmLDDNCxAA5gbnhhGrb5R6H00dRvwvAA== --osd-uuid 947c566d-835e-4d49-a5f5-82959e55891b 2024-09-18T15:48:40.542 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:48:40.542+0000 7ff913267440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:48:40.549 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:48:40.549+0000 7ff913267440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:48:40.550 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:48:40.550+0000 7ff913267440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:48:40.550 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:48:40.550+0000 7ff913267440 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-18T15:48:40.550 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:48:40.550+0000 7ff913267440 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-18T15:48:40.551 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:48:40.550+0000 7ff913267440 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-18T15:48:40.589 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:48:40.589+0000 7ff913267440 -1 bluestore(td/osd-rep-recov-eio.sh/0) _read_fsid unparsable uuid 2024-09-18T15:48:45.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/0/keyring 2024-09-18T15:48:45.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:48:45.911 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T15:48:45.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T15:48:45.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:48:46.424 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T15:48:46.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T15:48:46.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:48:46.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:48:46.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:48:46.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:48:46.449 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:48:46.448+0000 7f3ce460d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:48:46.471 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:48:46.470+0000 7f3ce460d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:48:46.472 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:48:46.471+0000 7f3ce460d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:48:46.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T15:48:46.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:48:46.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:48:46.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:48:46.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:48:46.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:48:46.876 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:48:46.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:48:46.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:48:46.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:48:47.007 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:48:47.007+0000 7f3ce460d440 -1 Falling back to public interface 2024-09-18T15:48:47.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:48:48.247 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:48:48.248 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:48:48.248 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:48:48.248 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:48:48.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:48:48.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:48:48.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:48:49.532 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:48:49.531+0000 7f3ce460d440 -1 osd.0 0 log_to_monitors true 2024-09-18T15:48:49.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:48:49.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:48:49.679 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:48:49.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:48:49.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:48:49.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:48:50.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:48:51.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:48:51.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:48:51.122 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:48:51.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:48:51.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:48:51.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:48:51.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:48:52.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:48:52.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:48:52.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:48:52.565 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:48:52.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:48:52.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:48:52.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:48:53.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:48:53.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:48:53.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:48:53.944 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:48:53.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:48:53.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:48:54.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:48:55.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:48:55.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:48:55.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:48:55.342 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:48:55.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:48:55.343 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:48:55.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:48:56.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:48:56.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:48:56.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:48:56.730 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:48:56.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:48:56.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:48:57.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:48:58.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:48:58.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:48:58.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:48:58.241 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:48:58.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:48:58.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:48:58.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:48:59.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:48:59.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:48:59.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:48:59.620 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:48:59.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:48:59.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:49:00.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:01.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:01.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:01.007 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:49:01.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:49:01.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:01.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:49:01.741 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:02.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:02.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:02.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:49:02.743 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:49:02.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:02.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:49:03.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:04.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:04.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:04.456 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:49:04.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:49:04.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:04.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:49:05.242 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:06.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:06.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:06.243 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:49:06.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:49:06.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:06.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:49:07.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:08.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:08.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:08.334 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:49:08.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:49:08.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:08.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:49:08.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:09.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:09.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:09.837 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:49:09.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:49:09.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:09.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:49:10.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:11.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:11.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:11.503 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:49:11.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:49:11.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:11.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:49:11.886 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/422795,v1:127.0.0.1:6803/422795] [v2:127.0.0.1:6804/422795,v1:127.0.0.1:6805/422795] exists,up 947c566d-835e-4d49-a5f5-82959e55891b 2024-09-18T15:49:11.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:49:11.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:49:11.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:49:11.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T15:49:11.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 1 2024-09-18T15:49:11.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:49:11.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:49:11.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T15:49:11.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:49:11.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2024-09-18T15:49:11.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-18T15:49:11.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:49:11.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:49:11.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:49:11.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2024-09-18T15:49:11.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2024-09-18T15:49:11.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:49:11.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:49:11.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-18T15:49:11.892 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:49:11.892 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:49:11.892 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:49:11.892 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:49:11.892 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:49:11.892 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:49:11.892 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:49:11.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:49:11.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:49:11.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:49:11.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:49:11.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-18T15:49:11.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-18T15:49:11.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:49:11.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:49:11.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:49:11.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:49:11.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:49:11.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:49:11.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2024-09-18T15:49:11.893 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:49:11.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=125e775e-7c27-4b2b-b949-4e269fd3c549 2024-09-18T15:49:11.893 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 125e775e-7c27-4b2b-b949-4e269fd3c549 2024-09-18T15:49:11.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 125e775e-7c27-4b2b-b949-4e269fd3c549' 2024-09-18T15:49:11.894 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:49:11.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB39upm+a/8NRAAz1eY+n9R1W/WlrGmHuvy6Q== 2024-09-18T15:49:11.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB39upm+a/8NRAAz1eY+n9R1W/WlrGmHuvy6Q=="}' 2024-09-18T15:49:11.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 125e775e-7c27-4b2b-b949-4e269fd3c549 -i td/osd-rep-recov-eio.sh/1/new.json 2024-09-18T15:49:12.838 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:49:12.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/1/new.json 2024-09-18T15:49:12.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB39upm+a/8NRAAz1eY+n9R1W/WlrGmHuvy6Q== --osd-uuid 125e775e-7c27-4b2b-b949-4e269fd3c549 2024-09-18T15:49:12.877 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:49:12.876+0000 7fc29e52b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:49:12.896 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:49:12.895+0000 7fc29e52b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:49:12.897 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:49:12.896+0000 7fc29e52b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:49:12.898 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:49:12.897+0000 7fc29e52b440 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-18T15:49:12.898 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:49:12.897+0000 7fc29e52b440 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-18T15:49:12.898 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:49:12.897+0000 7fc29e52b440 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-18T15:49:12.967 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:49:12.967+0000 7fc29e52b440 -1 bluestore(td/osd-rep-recov-eio.sh/1) _read_fsid unparsable uuid 2024-09-18T15:49:18.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/1/keyring 2024-09-18T15:49:18.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:49:18.944 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T15:49:18.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T15:49:18.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:49:19.464 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T15:49:19.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T15:49:19.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:49:19.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:49:19.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:49:19.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:49:19.489 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:49:19.488+0000 7fdf09f7f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:49:19.511 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:49:19.510+0000 7fdf09f7f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:49:19.512 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:49:19.511+0000 7fdf09f7f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:49:19.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T15:49:19.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:49:19.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:49:19.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:49:19.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:49:19.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:19.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:49:19.892 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:49:19.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:19.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:49:20.043 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:49:20.042+0000 7fdf09f7f440 -1 Falling back to public interface 2024-09-18T15:49:20.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:21.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:21.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:21.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:49:21.261 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:49:21.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:21.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:49:21.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:22.591 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:49:22.590+0000 7fdf09f7f440 -1 osd.1 0 log_to_monitors true 2024-09-18T15:49:22.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:22.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:22.710 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:49:22.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:49:22.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:22.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:49:23.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:24.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:24.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:24.269 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:49:24.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:49:24.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:24.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:49:24.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:25.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:25.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:25.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:49:25.681 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:49:25.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:25.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:49:26.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:27.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:27.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:27.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:49:27.063 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:49:27.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:27.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:49:27.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:28.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:28.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:28.532 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:49:28.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:49:28.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:28.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:49:28.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:29.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:29.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:29.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:49:29.913 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:49:29.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:29.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:49:30.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:31.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:31.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:31.374 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:49:31.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:49:31.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:31.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:49:31.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:32.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:32.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:32.787 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:49:32.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:49:32.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:32.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:49:33.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:34.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:34.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:34.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:49:34.170 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:49:34.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:34.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:49:34.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:35.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:35.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:35.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:49:35.848 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:49:35.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:35.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:49:36.599 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:37.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:37.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:37.600 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:49:37.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:49:37.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:37.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:49:38.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:39.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:39.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:39.073 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:49:39.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:49:39.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:39.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:49:39.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:40.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:40.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:40.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:49:40.856 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:49:40.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:40.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:49:41.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:42.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:42.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:42.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:49:42.622 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:49:42.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:42.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:49:43.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:44.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:44.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:44.382 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:49:44.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:49:44.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:44.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:49:45.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:46.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:46.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:46.058 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T15:49:46.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:49:46.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:46.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:49:46.435 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/424524,v1:127.0.0.1:6811/424524] [v2:127.0.0.1:6812/424524,v1:127.0.0.1:6813/424524] exists,up 125e775e-7c27-4b2b-b949-4e269fd3c549 2024-09-18T15:49:46.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:49:46.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:49:46.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:49:46.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T15:49:46.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 2 2024-09-18T15:49:46.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:49:46.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:49:46.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T15:49:46.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:49:46.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/2 2024-09-18T15:49:46.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-18T15:49:46.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:49:46.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:49:46.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:49:46.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2024-09-18T15:49:46.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2024-09-18T15:49:46.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:49:46.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:49:46.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-18T15:49:46.441 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:49:46.441 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:49:46.442 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:49:46.442 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:49:46.442 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:49:46.442 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:49:46.442 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:49:46.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:49:46.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:49:46.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:49:46.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:49:46.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-18T15:49:46.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-18T15:49:46.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:49:46.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:49:46.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:49:46.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:49:46.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:49:46.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:49:46.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/2 2024-09-18T15:49:46.443 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:49:46.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=280995ca-621e-4fd2-b65d-477315941c06 2024-09-18T15:49:46.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 280995ca-621e-4fd2-b65d-477315941c06' 2024-09-18T15:49:46.444 INFO:tasks.workunit.client.0.smithi047.stdout:add osd2 280995ca-621e-4fd2-b65d-477315941c06 2024-09-18T15:49:46.444 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:49:46.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCa9upm0+IPGxAAMfe+xk9QMjG+QkRDa9K89w== 2024-09-18T15:49:46.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCa9upm0+IPGxAAMfe+xk9QMjG+QkRDa9K89w=="}' 2024-09-18T15:49:46.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 280995ca-621e-4fd2-b65d-477315941c06 -i td/osd-rep-recov-eio.sh/2/new.json 2024-09-18T15:49:46.963 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:49:46.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/2/new.json 2024-09-18T15:49:46.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCa9upm0+IPGxAAMfe+xk9QMjG+QkRDa9K89w== --osd-uuid 280995ca-621e-4fd2-b65d-477315941c06 2024-09-18T15:49:47.001 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:49:47.000+0000 7f5325b21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:49:47.003 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:49:47.002+0000 7f5325b21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:49:47.004 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:49:47.003+0000 7f5325b21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:49:47.004 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:49:47.003+0000 7f5325b21440 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-18T15:49:47.004 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:49:47.003+0000 7f5325b21440 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-18T15:49:47.004 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:49:47.003+0000 7f5325b21440 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-18T15:49:47.061 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:49:47.061+0000 7f5325b21440 -1 bluestore(td/osd-rep-recov-eio.sh/2) _read_fsid unparsable uuid 2024-09-18T15:49:53.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/2/keyring 2024-09-18T15:49:53.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:49:53.160 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd2 key to auth repository 2024-09-18T15:49:53.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T15:49:53.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:49:53.678 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T15:49:53.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T15:49:53.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:49:53.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:49:53.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:49:53.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:49:53.703 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:49:53.702+0000 7ff78ae22440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:49:53.727 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:49:53.726+0000 7ff78ae22440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:49:53.727 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:49:53.727+0000 7ff78ae22440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:49:54.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T15:49:54.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:49:54.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:49:54.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:49:54.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:49:54.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:54.106 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:49:54.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:49:54.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:54.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:49:54.256 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:49:54.255+0000 7ff78ae22440 -1 Falling back to public interface 2024-09-18T15:49:54.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:55.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:55.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:55.543 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:49:55.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:49:55.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:55.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:49:55.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:56.741 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:49:56.740+0000 7ff78ae22440 -1 osd.2 0 log_to_monitors true 2024-09-18T15:49:56.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:56.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:56.970 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:49:56.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:49:56.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:56.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:49:57.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:58.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:58.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:58.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:49:58.414 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:49:58.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:58.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:49:58.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:49:59.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:49:59.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:49:59.853 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:49:59.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:49:59.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:49:59.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:50:00.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:01.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:01.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:01.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:50:01.354 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:50:01.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:01.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:50:01.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:02.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:02.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:02.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:50:02.807 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:50:02.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:02.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:50:03.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:04.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:04.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:04.192 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:50:04.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:50:04.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:04.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:50:04.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:05.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:05.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:05.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:50:05.653 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:50:05.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:05.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:50:06.052 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:07.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:07.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:07.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:50:07.054 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:50:07.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:07.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:50:07.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:08.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:08.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:08.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:50:08.508 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:50:08.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:08.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:50:09.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:10.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:10.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:10.118 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:50:10.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:50:10.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:10.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:50:11.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:12.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:12.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:12.017 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:50:12.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:50:12.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:12.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:50:12.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:13.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:13.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:13.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:50:13.845 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:50:13.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:13.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:50:14.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:15.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:15.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:15.224 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:50:15.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:50:15.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:15.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:50:15.938 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:16.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:16.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:16.940 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:50:16.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:50:16.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:16.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:50:17.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:18.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:18.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:18.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:50:18.803 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:50:18.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:18.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:50:19.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:50:20.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:50:20.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:50:20.456 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T15:50:20.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:50:20.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:50:20.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:50:20.974 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/426278,v1:127.0.0.1:6819/426278] [v2:127.0.0.1:6820/426278,v1:127.0.0.1:6821/426278] exists,up 280995ca-621e-4fd2-b65d-477315941c06 2024-09-18T15:50:20.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:50:20.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:50:20.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:50:20.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:55: setup_osds: wait_for_clean 2024-09-18T15:50:20.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:50:20.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:50:20.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:50:20.976 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:50:20.976 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:50:20.976 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:50:20.976 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:50:20.976 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:50:20.976 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:50:21.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:50:21.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:50:21.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:50:21.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:50:21.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:50:21.066 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:50:21.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:50:21.468 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:50:21.468 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T15:50:21.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:50:21.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:50:21.468 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:50:21.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803793 2024-09-18T15:50:21.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803793 2024-09-18T15:50:21.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803793' 2024-09-18T15:50:21.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:50:21.579 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:50:21.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=51539607574 2024-09-18T15:50:21.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 51539607574 2024-09-18T15:50:21.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803793 1-51539607574' 2024-09-18T15:50:21.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:50:21.690 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:50:21.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378629 2024-09-18T15:50:21.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378629 2024-09-18T15:50:21.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803793 1-51539607574 2-81604378629' 2024-09-18T15:50:21.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:50:21.800 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803793 2024-09-18T15:50:21.800 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:50:21.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:50:21.801 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803793 2024-09-18T15:50:21.801 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:50:21.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803793 2024-09-18T15:50:21.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803793' 2024-09-18T15:50:21.802 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 25769803793 2024-09-18T15:50:21.803 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:50:22.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803791 -lt 25769803793 2024-09-18T15:50:22.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:50:23.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:50:23.199 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:50:23.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803791 -lt 25769803793 2024-09-18T15:50:23.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:50:24.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-18T15:50:24.623 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:50:25.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803794 -lt 25769803793 2024-09-18T15:50:25.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:50:25.051 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-51539607574 2024-09-18T15:50:25.051 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:50:25.052 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:50:25.052 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-51539607574 2024-09-18T15:50:25.052 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:50:25.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=51539607574 2024-09-18T15:50:25.054 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 51539607574 2024-09-18T15:50:25.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 51539607574' 2024-09-18T15:50:25.054 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:50:25.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 51539607574 -lt 51539607574 2024-09-18T15:50:25.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:50:25.461 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378629 2024-09-18T15:50:25.461 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:50:25.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:50:25.463 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378629 2024-09-18T15:50:25.463 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:50:25.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378629 2024-09-18T15:50:25.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378629' 2024-09-18T15:50:25.464 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 81604378629 2024-09-18T15:50:25.464 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:50:25.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378630 -lt 81604378629 2024-09-18T15:50:25.870 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:50:25.870 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:50:25.870 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:50:26.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-18T15:50:26.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:50:26.368 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:50:26.368 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:50:26.368 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:50:26.368 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:50:26.368 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:50:26.368 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:50:26.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-18T15:50:26.760 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:50:26.760 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:50:26.760 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:50:27.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-18T15:50:27.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:50:27.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:50:27.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:274: TEST_rep_backfill_unfound: local poolname=test-pool 2024-09-18T15:50:27.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:275: TEST_rep_backfill_unfound: create_pool test-pool 1 1 2024-09-18T15:50:27.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test-pool 1 1 2024-09-18T15:50:27.962 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test-pool' already exists 2024-09-18T15:50:27.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:50:28.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:276: TEST_rep_backfill_unfound: wait_for_clean 2024-09-18T15:50:28.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:50:28.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:50:28.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:50:28.974 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:50:28.975 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:50:28.975 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:50:28.975 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:50:28.975 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:50:28.975 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:50:29.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:50:29.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:50:29.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:50:29.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:50:29.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:50:29.078 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:50:29.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:50:29.511 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:50:29.512 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T15:50:29.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:50:29.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:50:29.512 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:50:29.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803797 2024-09-18T15:50:29.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803797 2024-09-18T15:50:29.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803797' 2024-09-18T15:50:29.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:50:29.622 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:50:29.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=51539607577 2024-09-18T15:50:29.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 51539607577 2024-09-18T15:50:29.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803797 1-51539607577' 2024-09-18T15:50:29.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:50:29.733 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:50:29.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378632 2024-09-18T15:50:29.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378632 2024-09-18T15:50:29.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803797 1-51539607577 2-81604378632' 2024-09-18T15:50:29.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:50:29.847 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803797 2024-09-18T15:50:29.847 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:50:29.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:50:29.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803797 2024-09-18T15:50:29.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:50:29.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803797 2024-09-18T15:50:29.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803797' 2024-09-18T15:50:29.850 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 25769803797 2024-09-18T15:50:29.850 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:50:30.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803795 -lt 25769803797 2024-09-18T15:50:30.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:50:31.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:50:31.268 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:50:31.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803795 -lt 25769803797 2024-09-18T15:50:31.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:50:32.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-18T15:50:32.657 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:50:33.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803797 -lt 25769803797 2024-09-18T15:50:33.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:50:33.077 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-51539607577 2024-09-18T15:50:33.077 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:50:33.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:50:33.079 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-51539607577 2024-09-18T15:50:33.079 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:50:33.080 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=51539607577 2024-09-18T15:50:33.080 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 51539607577 2024-09-18T15:50:33.080 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 51539607577' 2024-09-18T15:50:33.080 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:50:33.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 51539607578 -lt 51539607577 2024-09-18T15:50:33.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:50:33.446 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378632 2024-09-18T15:50:33.446 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:50:33.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:50:33.447 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378632 2024-09-18T15:50:33.447 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:50:33.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378632 2024-09-18T15:50:33.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378632' 2024-09-18T15:50:33.449 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 81604378632 2024-09-18T15:50:33.449 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:50:33.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378633 -lt 81604378632 2024-09-18T15:50:33.816 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:50:33.817 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:50:33.817 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:50:34.343 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-18T15:50:34.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:50:34.344 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:50:34.344 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:50:34.344 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:50:34.344 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:50:34.344 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:50:34.344 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:50:34.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-09-18T15:50:34.700 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:50:34.700 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:50:34.701 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:50:35.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-09-18T15:50:35.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:50:35.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:50:35.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:278: TEST_rep_backfill_unfound: ceph pg dump pgs 2024-09-18T15:50:35.589 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T15:50:35.590 INFO:tasks.workunit.client.0.smithi047.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:50:20.569312+0000 0'0 23:27 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T15:48:40.058280+0000 0'0 2024-09-18T15:48:40.058280+0000 0 0 periodic scrub scheduled @ 2024-09-19T23:14:27.079138+0000 0 0 2024-09-18T15:50:35.590 INFO:tasks.workunit.client.0.smithi047.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:50:20.570465+0000 0'0 23:27 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T15:48:40.058280+0000 0'0 2024-09-18T15:48:40.058280+0000 0 0 periodic scrub scheduled @ 2024-09-20T03:47:40.458262+0000 0 0 2024-09-18T15:50:35.590 INFO:tasks.workunit.client.0.smithi047.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:50:20.569420+0000 0'0 23:14 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T15:48:40.058280+0000 0'0 2024-09-18T15:48:40.058280+0000 0 0 periodic scrub scheduled @ 2024-09-19T22:23:28.343746+0000 0 0 2024-09-18T15:50:35.590 INFO:tasks.workunit.client.0.smithi047.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:50:20.569926+0000 0'0 23:27 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T15:48:40.058280+0000 0'0 2024-09-18T15:48:40.058280+0000 0 0 periodic scrub scheduled @ 2024-09-19T18:14:28.373014+0000 0 0 2024-09-18T15:50:35.590 INFO:tasks.workunit.client.0.smithi047.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:50:29.199924+0000 0'0 23:10 [2,1,0] 2 [2,1,0] 2 0'0 2024-09-18T15:50:27.637221+0000 0'0 2024-09-18T15:50:27.637221+0000 0 0 periodic scrub scheduled @ 2024-09-20T02:13:19.192771+0000 0 0 2024-09-18T15:50:35.590 INFO:tasks.workunit.client.0.smithi047.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:50:20.569392+0000 0'0 23:27 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T15:48:40.058280+0000 0'0 2024-09-18T15:48:40.058280+0000 0 0 periodic scrub scheduled @ 2024-09-19T22:12:43.604676+0000 0 0 2024-09-18T15:50:35.590 INFO:tasks.workunit.client.0.smithi047.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:50:21.036690+0000 0'0 23:39 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-18T15:48:40.058280+0000 0'0 2024-09-18T15:48:40.058280+0000 0 0 periodic scrub scheduled @ 2024-09-20T01:44:26.635599+0000 0 0 2024-09-18T15:50:35.590 INFO:tasks.workunit.client.0.smithi047.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:50:20.569339+0000 0'0 23:14 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T15:48:40.058280+0000 0'0 2024-09-18T15:48:40.058280+0000 0 0 periodic scrub scheduled @ 2024-09-19T19:41:06.406520+0000 0 0 2024-09-18T15:50:35.590 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:50:20.440635+0000 0'0 23:27 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T15:48:40.058280+0000 0'0 2024-09-18T15:48:40.058280+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:54:12.813414+0000 0 0 2024-09-18T15:50:35.590 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T15:50:35.590 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T15:50:35.591 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T15:50:35.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:280: TEST_rep_backfill_unfound: rados_put td/osd-rep-recov-eio.sh test-pool myobject 2024-09-18T15:50:35.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:50:35.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=test-pool 2024-09-18T15:50:35.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=myobject 2024-09-18T15:50:35.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:50:35.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-18T15:50:35.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:50:35.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-18T15:50:35.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:50:35.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-18T15:50:35.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:50:35.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-18T15:50:35.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool test-pool put myobject td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:35.767 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:282: TEST_rep_backfill_unfound: get_osds test-pool myobject 2024-09-18T15:50:35.767 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-09-18T15:50:35.767 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-18T15:50:35.767 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool myobject 2024-09-18T15:50:35.767 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:50:36.213 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-09-18T15:50:36.213 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:50:36.213 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:50:36.213 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-09-18T15:50:36.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:282: TEST_rep_backfill_unfound: initial_osds=('2' '1' '0') 2024-09-18T15:50:36.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:282: TEST_rep_backfill_unfound: local -a initial_osds 2024-09-18T15:50:36.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:283: TEST_rep_backfill_unfound: local last_osd=0 2024-09-18T15:50:36.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:284: TEST_rep_backfill_unfound: kill_daemons td/osd-rep-recov-eio.sh TERM osd.0 2024-09-18T15:50:36.214 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:50:36.214 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:50:36.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:50:36.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:50:36.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:50:36.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:50:36.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:285: TEST_rep_backfill_unfound: ceph osd down 0 2024-09-18T15:50:37.309 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already down. 2024-09-18T15:50:37.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:286: TEST_rep_backfill_unfound: ceph osd out 0 2024-09-18T15:50:38.206 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already out. 2024-09-18T15:50:38.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:288: TEST_rep_backfill_unfound: ceph pg dump pgs 2024-09-18T15:50:38.563 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T15:50:38.564 INFO:tasks.workunit.client.0.smithi047.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:50:20.569312+0000 0'0 23:27 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T15:48:40.058280+0000 0'0 2024-09-18T15:48:40.058280+0000 0 0 periodic scrub scheduled @ 2024-09-19T23:14:27.079138+0000 0 0 2024-09-18T15:50:38.564 INFO:tasks.workunit.client.0.smithi047.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:50:20.570465+0000 0'0 23:27 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T15:48:40.058280+0000 0'0 2024-09-18T15:48:40.058280+0000 0 0 periodic scrub scheduled @ 2024-09-20T03:47:40.458262+0000 0 0 2024-09-18T15:50:38.564 INFO:tasks.workunit.client.0.smithi047.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:50:20.569420+0000 0'0 23:14 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T15:48:40.058280+0000 0'0 2024-09-18T15:48:40.058280+0000 0 0 periodic scrub scheduled @ 2024-09-19T22:23:28.343746+0000 0 0 2024-09-18T15:50:38.564 INFO:tasks.workunit.client.0.smithi047.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:50:20.569926+0000 0'0 23:27 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T15:48:40.058280+0000 0'0 2024-09-18T15:48:40.058280+0000 0 0 periodic scrub scheduled @ 2024-09-19T18:14:28.373014+0000 0 0 2024-09-18T15:50:38.564 INFO:tasks.workunit.client.0.smithi047.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:50:29.199924+0000 0'0 23:10 [2,1,0] 2 [2,1,0] 2 0'0 2024-09-18T15:50:27.637221+0000 0'0 2024-09-18T15:50:27.637221+0000 0 0 periodic scrub scheduled @ 2024-09-20T02:13:19.192771+0000 0 0 2024-09-18T15:50:38.564 INFO:tasks.workunit.client.0.smithi047.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:50:20.569392+0000 0'0 23:27 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T15:48:40.058280+0000 0'0 2024-09-18T15:48:40.058280+0000 0 0 periodic scrub scheduled @ 2024-09-19T22:12:43.604676+0000 0 0 2024-09-18T15:50:38.564 INFO:tasks.workunit.client.0.smithi047.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 stale+active+clean 2024-09-18T15:50:21.036690+0000 0'0 23:39 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-18T15:48:40.058280+0000 0'0 2024-09-18T15:48:40.058280+0000 0 0 periodic scrub scheduled @ 2024-09-20T01:44:26.635599+0000 0 0 2024-09-18T15:50:38.564 INFO:tasks.workunit.client.0.smithi047.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:50:20.569339+0000 0'0 23:14 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T15:48:40.058280+0000 0'0 2024-09-18T15:48:40.058280+0000 0 0 periodic scrub scheduled @ 2024-09-19T19:41:06.406520+0000 0 0 2024-09-18T15:50:38.564 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:50:20.440635+0000 0'0 23:27 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T15:48:40.058280+0000 0'0 2024-09-18T15:48:40.058280+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:54:12.813414+0000 0 0 2024-09-18T15:50:38.564 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T15:50:38.564 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T15:50:38.565 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T15:50:38.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:290: TEST_rep_backfill_unfound: dd if=/dev/urandom of=td/osd-rep-recov-eio.sh/ORIGINAL bs=1024 count=4 2024-09-18T15:50:38.575 INFO:tasks.workunit.client.0.smithi047.stderr:4+0 records in 2024-09-18T15:50:38.575 INFO:tasks.workunit.client.0.smithi047.stderr:4+0 records out 2024-09-18T15:50:38.575 INFO:tasks.workunit.client.0.smithi047.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 9.3108e-05 s, 44.0 MB/s 2024-09-18T15:50:38.575 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: seq 1 300 2024-09-18T15:50:38.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:38.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj1 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:38.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:38.699 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj2 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:38.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:38.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj3 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:38.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:38.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj4 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:39.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:39.119 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj5 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:39.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:39.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj6 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:39.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:39.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj7 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:39.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:39.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj8 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:39.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:39.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj9 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:39.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:39.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj10 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:39.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:39.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj11 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:40.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:40.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj12 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:40.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:40.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj13 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:40.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:40.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj14 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:40.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:40.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj15 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:40.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:40.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj16 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:40.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:40.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj17 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:40.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:40.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj18 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:41.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:41.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj19 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:41.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:41.157 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj20 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:41.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:41.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj21 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:41.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:41.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj22 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:41.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:41.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj23 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:41.642 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:41.642 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj24 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:41.826 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:41.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj25 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:42.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:42.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj26 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:42.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:42.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj27 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:42.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:42.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj28 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:42.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:42.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj29 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:42.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:42.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj30 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:42.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:42.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj31 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:42.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:42.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj32 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:42.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:42.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj33 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:43.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:43.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj34 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:43.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:43.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj35 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:43.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:43.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj36 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:43.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:43.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj37 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:43.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:43.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj38 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:43.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:43.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj39 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:43.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:43.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj40 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:43.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:43.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj41 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:44.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:44.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj42 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:44.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:44.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj43 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:44.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:44.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj44 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:44.483 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:44.483 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj45 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:44.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:44.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj46 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:44.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:44.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj47 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:44.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:44.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj48 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:45.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:45.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj49 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:45.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:45.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj50 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:45.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:45.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj51 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:45.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:45.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj52 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:45.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:45.740 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj53 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:45.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:45.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj54 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:46.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:46.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj55 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:46.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:46.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj56 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:46.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:46.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj57 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:46.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:46.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj58 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:46.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:46.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj59 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:46.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:46.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj60 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:46.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:46.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj61 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:46.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:46.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj62 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:47.096 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:47.096 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj63 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:47.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:47.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj64 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:47.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:47.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj65 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:47.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:47.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj66 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:47.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:47.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj67 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:47.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:47.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj68 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:47.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:47.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj69 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:48.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:48.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj70 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:48.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:48.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj71 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:48.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:48.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj72 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:48.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:48.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj73 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:48.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:48.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj74 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:48.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:48.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj75 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:48.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:48.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj76 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:48.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:48.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj77 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:49.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:49.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj78 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:49.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:49.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj79 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:49.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:49.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj80 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:49.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:49.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj81 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:49.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:49.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj82 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:49.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:49.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj83 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:49.862 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:49.862 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj84 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:49.988 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:49.988 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj85 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:50.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:50.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj86 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:50.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:50.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj87 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:50.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:50.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj88 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:50.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:50.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj89 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:50.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:50.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj90 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:50.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:50.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj91 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:50.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:50.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj92 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:51.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:51.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj93 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:51.194 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:51.194 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj94 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:51.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:51.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj95 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:51.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:51.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj96 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:51.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:51.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj97 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:51.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:51.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj98 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:51.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:51.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj99 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:52.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:52.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj100 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:52.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:52.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj101 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:52.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:52.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj102 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:52.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:52.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj103 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:52.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:52.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj104 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:52.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:52.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj105 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:52.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:52.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj106 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:52.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:52.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj107 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:53.095 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:53.096 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj108 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:53.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:53.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj109 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:53.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:53.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj110 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:53.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:53.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj111 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:53.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:53.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj112 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:53.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:53.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj113 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:53.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:53.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj114 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:54.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:54.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj115 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:54.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:54.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj116 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:54.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:54.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj117 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:54.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:54.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj118 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:54.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:54.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj119 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:54.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:54.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj120 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:54.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:54.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj121 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:55.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:55.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj122 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:55.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:55.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj123 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:55.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:55.299 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj124 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:55.432 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:55.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj125 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:55.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:55.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj126 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:55.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:55.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj127 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:55.826 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:55.826 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj128 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:55.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:55.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj129 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:56.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:56.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj130 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:56.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:56.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj131 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:56.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:56.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj132 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:56.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:56.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj133 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:56.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:56.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj134 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:56.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:56.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj135 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:56.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:56.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj136 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:57.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:57.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj137 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:57.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:57.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj138 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:57.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:57.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj139 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:57.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:57.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj140 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:57.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:57.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj141 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:57.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:57.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj142 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:57.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:57.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj143 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:57.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:57.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj144 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:58.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:58.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj145 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:58.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:58.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj146 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:58.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:58.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj147 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:58.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:58.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj148 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:58.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:58.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj149 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:58.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:58.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj150 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:58.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:58.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj151 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:59.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:59.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj152 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:59.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:59.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj153 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:59.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:59.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj154 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:59.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:59.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj155 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:59.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:59.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj156 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:59.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:59.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj157 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:59.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:59.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj158 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:50:59.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:50:59.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj159 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:00.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:00.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj160 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:00.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:00.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj161 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:00.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:00.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj162 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:00.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:00.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj163 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:00.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:00.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj164 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:00.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:00.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj165 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:00.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:00.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj166 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:00.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:00.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj167 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:01.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:01.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj168 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:01.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:01.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj169 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:01.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:01.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj170 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:01.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:01.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj171 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:01.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:01.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj172 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:01.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:01.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj173 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:01.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:01.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj174 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:01.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:01.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj175 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:02.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:02.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj176 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:02.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:02.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj177 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:02.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:02.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj178 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:02.491 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:02.491 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj179 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:02.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:02.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj180 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:02.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:02.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj181 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:02.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:02.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj182 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:02.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:02.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj183 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:03.119 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:03.119 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj184 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:03.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:03.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj185 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:03.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:03.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj186 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:03.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:03.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj187 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:03.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:03.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj188 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:03.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:03.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj189 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:03.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:03.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj190 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:04.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:04.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj191 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:04.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:04.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj192 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:04.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:04.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj193 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:04.476 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:04.476 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj194 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:04.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:04.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj195 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:04.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:04.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj196 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:04.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:04.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj197 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:04.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:04.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj198 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:05.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:05.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj199 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:05.247 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:05.247 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj200 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:05.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:05.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj201 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:05.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:05.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj202 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:05.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:05.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj203 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:05.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:05.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj204 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:05.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:05.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj205 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:06.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:06.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj206 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:06.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:06.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj207 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:06.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:06.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj208 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:06.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:06.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj209 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:06.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:06.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj210 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:06.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:06.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj211 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:06.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:06.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj212 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:06.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:06.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj213 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:07.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:07.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj214 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:07.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:07.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj215 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:07.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:07.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj216 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:07.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:07.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj217 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:07.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:07.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj218 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:07.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:07.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj219 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:07.826 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:07.826 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj220 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:07.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:07.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj221 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:08.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:08.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj222 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:08.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:08.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj223 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:08.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:08.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj224 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:08.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:08.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj225 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:08.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:08.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj226 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:08.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:08.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj227 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:08.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:08.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj228 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:09.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:09.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj229 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:09.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:09.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj230 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:09.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:09.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj231 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:09.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:09.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj232 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:09.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:09.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj233 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:09.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:09.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj234 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:09.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:09.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj235 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:09.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:09.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj236 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:10.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:10.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj237 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:10.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:10.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj238 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:10.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:10.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj239 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:10.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:10.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj240 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:10.599 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:10.599 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj241 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:10.741 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:10.741 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj242 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:10.875 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:10.875 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj243 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:11.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:11.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj244 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:11.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:11.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj245 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:11.285 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:11.285 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj246 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:11.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:11.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj247 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:11.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:11.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj248 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:11.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:11.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj249 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:11.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:11.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj250 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:12.022 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:12.022 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj251 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:12.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:12.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj252 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:12.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:12.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj253 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:12.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:12.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj254 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:12.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:12.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj255 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:12.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:12.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj256 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:12.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:12.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj257 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:13.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:13.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj258 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:13.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:13.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj259 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:13.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:13.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj260 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:13.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:13.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj261 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:13.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:13.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj262 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:13.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:13.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj263 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:13.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:13.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj264 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:13.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:13.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj265 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:14.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:14.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj266 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:14.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:14.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj267 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:14.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:14.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj268 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:14.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:14.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj269 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:14.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:14.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj270 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:14.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:14.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj271 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:14.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:14.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj272 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:15.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:15.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj273 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:15.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:15.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj274 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:15.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:15.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj275 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:15.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:15.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj276 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:15.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:15.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj277 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:15.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:15.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj278 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:15.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:15.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj279 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:15.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:15.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj280 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:16.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:16.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj281 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:16.242 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:16.242 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj282 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:16.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:16.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj283 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:16.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:16.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj284 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:16.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:16.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj285 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:17.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:17.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj286 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:17.212 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:17.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj287 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:17.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:17.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj288 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:17.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:17.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj289 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:17.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:17.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj290 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:17.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:17.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj291 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:17.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:17.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj292 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:18.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:18.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj293 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:18.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:18.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj294 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:18.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:18.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj295 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:18.437 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:18.437 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj296 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:18.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:18.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj297 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:18.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:18.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj298 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:18.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:18.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj299 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:18.948 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:18.948 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj300 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:51:19.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:296: TEST_rep_backfill_unfound: inject_eio rep data test-pool obj250 td/osd-rep-recov-eio.sh 0 2024-09-18T15:51:19.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:51:19.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:51:19.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:51:19.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:51:19.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=test-pool 2024-09-18T15:51:19.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:51:19.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj250 2024-09-18T15:51:19.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:51:19.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:51:19.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:51:19.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T15:51:19.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:51:19.076 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds test-pool obj250 2024-09-18T15:51:19.076 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-09-18T15:51:19.076 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj250 2024-09-18T15:51:19.076 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool obj250 2024-09-18T15:51:19.076 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:51:19.494 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-09-18T15:51:19.494 INFO:tasks.workunit.client.0.smithi047.stderr:1' 2024-09-18T15:51:19.494 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 2024-09-18T15:51:19.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('2' '1') 2024-09-18T15:51:19.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:51:19.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=2 2024-09-18T15:51:19.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:51:19.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:51:19.495 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/2/type 2024-09-18T15:51:19.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:51:19.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 2 bluestore_debug_inject_read_err true 2024-09-18T15:51:19.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:51:19.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=2 2024-09-18T15:51:19.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:51:19.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:51:19.497 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:51:19.497 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.2 2024-09-18T15:51:19.498 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T15:51:19.498 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T15:51:19.498 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:51:19.498 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:51:19.498 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:51:19.498 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.2.asok 2024-09-18T15:51:19.498 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.2.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:51:19.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:51:19.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:51:19.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:51:19.580 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.2 2024-09-18T15:51:19.581 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T15:51:19.581 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T15:51:19.581 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:51:19.581 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:51:19.581 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:51:19.581 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.2.asok 2024-09-18T15:51:19.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:51:19.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.2.asok injectdataerr test-pool obj250 2024-09-18T15:51:19.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:297: TEST_rep_backfill_unfound: inject_eio rep data test-pool obj250 td/osd-rep-recov-eio.sh 1 2024-09-18T15:51:19.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:51:19.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:51:19.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:51:19.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:51:19.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=test-pool 2024-09-18T15:51:19.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:51:19.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj250 2024-09-18T15:51:19.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:51:19.666 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:51:19.666 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:51:19.666 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-18T15:51:19.666 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:51:19.666 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds test-pool obj250 2024-09-18T15:51:19.666 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-09-18T15:51:19.666 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj250 2024-09-18T15:51:19.666 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool obj250 2024-09-18T15:51:19.666 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:51:20.034 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-09-18T15:51:20.035 INFO:tasks.workunit.client.0.smithi047.stderr:1' 2024-09-18T15:51:20.035 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 2024-09-18T15:51:20.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('2' '1') 2024-09-18T15:51:20.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:51:20.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-18T15:51:20.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:51:20.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:51:20.035 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-18T15:51:20.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:51:20.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-18T15:51:20.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:51:20.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-18T15:51:20.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:51:20.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:51:20.037 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:51:20.037 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-18T15:51:20.037 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:51:20.038 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:51:20.038 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:51:20.038 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:51:20.038 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:51:20.038 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.1.asok 2024-09-18T15:51:20.038 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:51:20.119 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:51:20.119 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:51:20.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:51:20.121 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-18T15:51:20.121 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:51:20.121 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:51:20.121 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:51:20.121 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:51:20.121 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:51:20.121 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.1.asok 2024-09-18T15:51:20.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:51:20.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.1.asok injectdataerr test-pool obj250 2024-09-18T15:51:20.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:299: TEST_rep_backfill_unfound: activate_osd td/osd-rep-recov-eio.sh 0 2024-09-18T15:51:20.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:51:20.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:51:20.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:51:20.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:51:20.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-09-18T15:51:20.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-18T15:51:20.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:51:20.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:51:20.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:51:20.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-09-18T15:51:20.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-09-18T15:51:20.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:51:20.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:51:20.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-18T15:51:20.208 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:51:20.208 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:51:20.208 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:51:20.208 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:51:20.208 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:51:20.208 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:51:20.208 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:51:20.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:51:20.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:51:20.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-18T15:51:20.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-18T15:51:20.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:51:20.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:51:20.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:51:20.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:51:20.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:51:20.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:51:20.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-09-18T15:51:20.209 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T15:51:20.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:51:20.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:51:20.210 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-rep-recov-eio.sh/0/whoami 2024-09-18T15:51:20.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:51:20.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:51:20.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:51:20.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:51:20.230 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:51:20.229+0000 7fe46701e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:51:20.232 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:51:20.231+0000 7fe46701e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:51:20.233 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:51:20.232+0000 7fe46701e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:51:20.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:51:20.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:51:20.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:51:20.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:51:20.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:51:20.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:51:20.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:51:20.623 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:51:20.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:51:20.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:51:20.755 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:51:20.754+0000 7fe46701e440 -1 Falling back to public interface 2024-09-18T15:51:21.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:51:22.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:51:22.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:51:22.032 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:51:22.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:51:22.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:51:22.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:51:22.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:51:23.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:51:23.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:51:23.411 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:51:23.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:51:23.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:51:23.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:51:23.588 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:51:23.587+0000 7fe46701e440 -1 osd.0 24 log_to_monitors true 2024-09-18T15:51:23.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:51:24.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:51:24.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:51:24.849 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:51:24.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:51:24.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:51:24.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:51:25.400 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up out weight 0 up_from 30 up_thru 19 down_at 25 last_clean_interval [6,24) [v2:127.0.0.1:6802/434169,v1:127.0.0.1:6803/434169] [v2:127.0.0.1:6804/434169,v1:127.0.0.1:6805/434169] exists,up 947c566d-835e-4d49-a5f5-82959e55891b 2024-09-18T15:51:25.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:51:25.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:51:25.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:51:25.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:300: TEST_rep_backfill_unfound: ceph osd in 0 2024-09-18T15:51:26.612 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already in. 2024-09-18T15:51:26.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:302: TEST_rep_backfill_unfound: sleep 15 2024-09-18T15:51:41.625 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: seq 1 360 2024-09-18T15:51:41.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-09-18T15:51:41.626 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-09-18T15:51:41.626 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-09-18T15:51:41.626 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-09-18T15:51:41.627 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-09-18T15:51:41.627 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-18T15:51:42.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-18T15:51:42.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-18T15:51:42.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-09-18T15:51:42.005 INFO:tasks.workunit.client.0.smithi047.stdout:active+undersized+degraded+remapped+backfilling 2024-09-18T15:51:42.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2024-09-18T15:51:42.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2024-09-18T15:51:42.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2024-09-18T15:51:43.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-09-18T15:51:43.007 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-09-18T15:51:43.007 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-09-18T15:51:43.007 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-09-18T15:51:43.007 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-09-18T15:51:43.007 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-18T15:51:43.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-18T15:51:43.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-18T15:51:43.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-09-18T15:51:43.379 INFO:tasks.workunit.client.0.smithi047.stdout:active+undersized+degraded+remapped+backfilling 2024-09-18T15:51:43.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2024-09-18T15:51:43.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2024-09-18T15:51:43.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2024-09-18T15:51:44.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-09-18T15:51:44.381 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-09-18T15:51:44.381 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-09-18T15:51:44.382 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-09-18T15:51:44.382 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-09-18T15:51:44.382 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-18T15:51:44.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-18T15:51:44.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-18T15:51:44.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-09-18T15:51:44.750 INFO:tasks.workunit.client.0.smithi047.stdout:active+undersized+degraded+remapped+backfilling 2024-09-18T15:51:44.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2024-09-18T15:51:44.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2024-09-18T15:51:44.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2024-09-18T15:51:45.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-09-18T15:51:45.752 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-09-18T15:51:45.753 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-09-18T15:51:45.753 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-09-18T15:51:45.753 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-09-18T15:51:45.753 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-18T15:51:46.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+backfill_unfound+undersized+degraded+remapped 2024-09-18T15:51:46.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+backfill_unfound+undersized+degraded+remapped 2024-09-18T15:51:46.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-09-18T15:51:46.125 INFO:tasks.workunit.client.0.smithi047.stdout:active+backfill_unfound+undersized+degraded+remapped 2024-09-18T15:51:46.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 0 = 0 ']' 2024-09-18T15:51:46.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:308: TEST_rep_backfill_unfound: break 2024-09-18T15:51:46.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:314: TEST_rep_backfill_unfound: ceph pg dump pgs 2024-09-18T15:51:46.475 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T15:51:46.475 INFO:tasks.workunit.client.0.smithi047.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:51:27.125443+0000 0'0 38:69 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T15:48:40.058280+0000 0'0 2024-09-18T15:48:40.058280+0000 0 0 periodic scrub scheduled @ 2024-09-20T01:56:54.503440+0000 0 0 2024-09-18T15:51:46.476 INFO:tasks.workunit.client.0.smithi047.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:51:28.182207+0000 0'0 39:71 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T15:48:40.058280+0000 0'0 2024-09-18T15:48:40.058280+0000 0 0 periodic scrub scheduled @ 2024-09-19T19:41:55.548558+0000 0 0 2024-09-18T15:51:46.476 INFO:tasks.workunit.client.0.smithi047.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:51:28.181405+0000 0'0 39:57 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T15:48:40.058280+0000 0'0 2024-09-18T15:48:40.058280+0000 0 0 periodic scrub scheduled @ 2024-09-19T22:05:22.365991+0000 0 0 2024-09-18T15:51:46.476 INFO:tasks.workunit.client.0.smithi047.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:51:28.148286+0000 0'0 39:71 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T15:48:40.058280+0000 0'0 2024-09-18T15:48:40.058280+0000 0 0 periodic scrub scheduled @ 2024-09-20T01:21:16.544405+0000 0 0 2024-09-18T15:51:46.476 INFO:tasks.workunit.client.0.smithi047.stdout:2.0 301 1 28 0 1 1232896 0 0 101 200 101 active+backfill_unfound+undersized+degraded+remapped 2024-09-18T15:51:42.577335+0000 28'301 39:902 [2,1,0] 2 [2,1] 2 0'0 2024-09-18T15:50:27.637221+0000 0'0 2024-09-18T15:50:27.637221+0000 0 0 periodic scrub scheduled @ 2024-09-20T01:55:15.523774+0000 0 0 2024-09-18T15:51:46.476 INFO:tasks.workunit.client.0.smithi047.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:51:27.235921+0000 0'0 38:69 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T15:48:40.058280+0000 0'0 2024-09-18T15:48:40.058280+0000 0 0 periodic scrub scheduled @ 2024-09-20T01:44:01.065108+0000 0 0 2024-09-18T15:51:46.476 INFO:tasks.workunit.client.0.smithi047.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:51:27.706847+0000 0'0 38:58 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-18T15:48:40.058280+0000 0'0 2024-09-18T15:48:40.058280+0000 0 0 periodic scrub scheduled @ 2024-09-20T03:05:09.206961+0000 0 0 2024-09-18T15:51:46.476 INFO:tasks.workunit.client.0.smithi047.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:51:28.181488+0000 0'0 39:57 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T15:48:40.058280+0000 0'0 2024-09-18T15:48:40.058280+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:20:42.033574+0000 0 0 2024-09-18T15:51:46.476 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:51:28.182470+0000 0'0 39:71 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T15:48:40.058280+0000 0'0 2024-09-18T15:48:40.058280+0000 0 0 periodic scrub scheduled @ 2024-09-20T03:36:11.100282+0000 0 0 2024-09-18T15:51:46.476 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T15:51:46.476 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T15:51:46.477 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T15:51:46.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:315: TEST_rep_backfill_unfound: ceph pg 2.0 list_unfound 2024-09-18T15:51:46.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:315: TEST_rep_backfill_unfound: grep -q obj250 2024-09-18T15:51:46.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:318: TEST_rep_backfill_unfound: timeout 5 rados -p test-pool get obj250 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:51.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:319: TEST_rep_backfill_unfound: test 124 = 124 2024-09-18T15:51:51.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:321: TEST_rep_backfill_unfound: ceph pg 2.0 mark_unfound_lost delete 2024-09-18T15:51:51.820 INFO:tasks.workunit.client.0.smithi047.stderr:pg has no unfound objects 2024-09-18T15:51:51.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:323: TEST_rep_backfill_unfound: wait_for_clean 2024-09-18T15:51:51.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:51:51.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:51:51.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:51:51.830 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:51:51.831 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:51:51.831 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:51:51.831 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:51:51.831 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:51:51.831 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:51:51.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:51:51.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:51:51.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:51:51.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:51:51.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:51:51.925 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:51:52.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:51:52.354 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:51:52.354 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T15:51:52.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:51:52.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:51:52.354 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:51:52.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018888 2024-09-18T15:51:52.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018888 2024-09-18T15:51:52.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-128849018888' 2024-09-18T15:51:52.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:51:52.498 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:51:52.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=51539607596 2024-09-18T15:51:52.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 51539607596 2024-09-18T15:51:52.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-128849018888 1-51539607596' 2024-09-18T15:51:52.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:51:52.649 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:51:52.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378651 2024-09-18T15:51:52.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378651 2024-09-18T15:51:52.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-128849018888 1-51539607596 2-81604378651' 2024-09-18T15:51:52.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:51:52.759 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-128849018888 2024-09-18T15:51:52.760 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:51:52.760 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:51:52.761 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-128849018888 2024-09-18T15:51:52.761 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:51:52.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018888 2024-09-18T15:51:52.762 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 128849018888 2024-09-18T15:51:52.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 128849018888' 2024-09-18T15:51:52.763 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:51:53.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018886 -lt 128849018888 2024-09-18T15:51:53.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:51:54.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:51:54.295 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:51:54.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018888 -lt 128849018888 2024-09-18T15:51:54.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:51:54.680 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-51539607596 2024-09-18T15:51:54.680 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:51:54.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:51:54.681 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-51539607596 2024-09-18T15:51:54.682 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:51:54.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=51539607596 2024-09-18T15:51:54.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 51539607596' 2024-09-18T15:51:54.683 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 51539607596 2024-09-18T15:51:54.683 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:51:55.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 51539607596 -lt 51539607596 2024-09-18T15:51:55.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:51:55.201 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378651 2024-09-18T15:51:55.201 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:51:55.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:51:55.202 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378651 2024-09-18T15:51:55.203 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:51:55.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378651 2024-09-18T15:51:55.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378651' 2024-09-18T15:51:55.204 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 81604378651 2024-09-18T15:51:55.204 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:51:55.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378651 -lt 81604378651 2024-09-18T15:51:55.616 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:51:55.616 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:51:55.616 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:51:56.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-18T15:51:56.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:51:56.149 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:51:56.149 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:51:56.149 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:51:56.149 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:51:56.149 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:51:56.149 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:51:56.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-09-18T15:51:56.504 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:51:56.505 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:51:56.505 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:51:57.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-09-18T15:51:57.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:51:57.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:51:57.005 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: seq 1 300 2024-09-18T15:51:57.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:57.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj1 = obj250 ']' 2024-09-18T15:51:57.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj1 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:57.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj2 = obj250 ']' 2024-09-18T15:51:57.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj2 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:57.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj3 = obj250 ']' 2024-09-18T15:51:57.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj3 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:57.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj4 = obj250 ']' 2024-09-18T15:51:57.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj4 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:57.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj5 = obj250 ']' 2024-09-18T15:51:57.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj5 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:57.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj6 = obj250 ']' 2024-09-18T15:51:57.212 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj6 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:57.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj7 = obj250 ']' 2024-09-18T15:51:57.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj7 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:57.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj8 = obj250 ']' 2024-09-18T15:51:57.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj8 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:57.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj9 = obj250 ']' 2024-09-18T15:51:57.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj9 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:57.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj10 = obj250 ']' 2024-09-18T15:51:57.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj10 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:57.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj11 = obj250 ']' 2024-09-18T15:51:57.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj11 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:57.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj12 = obj250 ']' 2024-09-18T15:51:57.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj12 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.406 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.406 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:57.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj13 = obj250 ']' 2024-09-18T15:51:57.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj13 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:57.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj14 = obj250 ']' 2024-09-18T15:51:57.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj14 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:57.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj15 = obj250 ']' 2024-09-18T15:51:57.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj15 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:57.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj16 = obj250 ']' 2024-09-18T15:51:57.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj16 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:57.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj17 = obj250 ']' 2024-09-18T15:51:57.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj17 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:57.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj18 = obj250 ']' 2024-09-18T15:51:57.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj18 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:57.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj19 = obj250 ']' 2024-09-18T15:51:57.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj19 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:57.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj20 = obj250 ']' 2024-09-18T15:51:57.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj20 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:57.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj21 = obj250 ']' 2024-09-18T15:51:57.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj21 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:57.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj22 = obj250 ']' 2024-09-18T15:51:57.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj22 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:57.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj23 = obj250 ']' 2024-09-18T15:51:57.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj23 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:57.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj24 = obj250 ']' 2024-09-18T15:51:57.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj24 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:57.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj25 = obj250 ']' 2024-09-18T15:51:57.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj25 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:57.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj26 = obj250 ']' 2024-09-18T15:51:57.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj26 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:57.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj27 = obj250 ']' 2024-09-18T15:51:57.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj27 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:57.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj28 = obj250 ']' 2024-09-18T15:51:57.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj28 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:57.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj29 = obj250 ']' 2024-09-18T15:51:57.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj29 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:57.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj30 = obj250 ']' 2024-09-18T15:51:57.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj30 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:57.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj31 = obj250 ']' 2024-09-18T15:51:57.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj31 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:57.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj32 = obj250 ']' 2024-09-18T15:51:57.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj32 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:57.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:57.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj33 = obj250 ']' 2024-09-18T15:51:57.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj33 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:58.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj34 = obj250 ']' 2024-09-18T15:51:58.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj34 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:58.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj35 = obj250 ']' 2024-09-18T15:51:58.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj35 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:58.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj36 = obj250 ']' 2024-09-18T15:51:58.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj36 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:58.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj37 = obj250 ']' 2024-09-18T15:51:58.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj37 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:58.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj38 = obj250 ']' 2024-09-18T15:51:58.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj38 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.194 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:58.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj39 = obj250 ']' 2024-09-18T15:51:58.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj39 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:58.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj40 = obj250 ']' 2024-09-18T15:51:58.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj40 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:58.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj41 = obj250 ']' 2024-09-18T15:51:58.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj41 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:58.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj42 = obj250 ']' 2024-09-18T15:51:58.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj42 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:58.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj43 = obj250 ']' 2024-09-18T15:51:58.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj43 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:58.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj44 = obj250 ']' 2024-09-18T15:51:58.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj44 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:58.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj45 = obj250 ']' 2024-09-18T15:51:58.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj45 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.390 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:58.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj46 = obj250 ']' 2024-09-18T15:51:58.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj46 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:58.419 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj47 = obj250 ']' 2024-09-18T15:51:58.419 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj47 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:58.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj48 = obj250 ']' 2024-09-18T15:51:58.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj48 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:58.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj49 = obj250 ']' 2024-09-18T15:51:58.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj49 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:58.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj50 = obj250 ']' 2024-09-18T15:51:58.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj50 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:58.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj51 = obj250 ']' 2024-09-18T15:51:58.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj51 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:58.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj52 = obj250 ']' 2024-09-18T15:51:58.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj52 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.585 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:58.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj53 = obj250 ']' 2024-09-18T15:51:58.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj53 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:58.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj54 = obj250 ']' 2024-09-18T15:51:58.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj54 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.642 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:58.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj55 = obj250 ']' 2024-09-18T15:51:58.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj55 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:58.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj56 = obj250 ']' 2024-09-18T15:51:58.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj56 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:58.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj57 = obj250 ']' 2024-09-18T15:51:58.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj57 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:58.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj58 = obj250 ']' 2024-09-18T15:51:58.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj58 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:58.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj59 = obj250 ']' 2024-09-18T15:51:58.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj59 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.780 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:58.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj60 = obj250 ']' 2024-09-18T15:51:58.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj60 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:58.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj61 = obj250 ']' 2024-09-18T15:51:58.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj61 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:58.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj62 = obj250 ']' 2024-09-18T15:51:58.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj62 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:58.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj63 = obj250 ']' 2024-09-18T15:51:58.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj63 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:58.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj64 = obj250 ']' 2024-09-18T15:51:58.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj64 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:58.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj65 = obj250 ']' 2024-09-18T15:51:58.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj65 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.948 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.949 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:58.949 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj66 = obj250 ']' 2024-09-18T15:51:58.949 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj66 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:58.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:58.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj67 = obj250 ']' 2024-09-18T15:51:58.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj67 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:59.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj68 = obj250 ']' 2024-09-18T15:51:59.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj68 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:59.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj69 = obj250 ']' 2024-09-18T15:51:59.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj69 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:59.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj70 = obj250 ']' 2024-09-18T15:51:59.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj70 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.091 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:59.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj71 = obj250 ']' 2024-09-18T15:51:59.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj71 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:59.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj72 = obj250 ']' 2024-09-18T15:51:59.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj72 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:59.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj73 = obj250 ']' 2024-09-18T15:51:59.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj73 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:59.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj74 = obj250 ']' 2024-09-18T15:51:59.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj74 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:59.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj75 = obj250 ']' 2024-09-18T15:51:59.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj75 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:59.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj76 = obj250 ']' 2024-09-18T15:51:59.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj76 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:59.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj77 = obj250 ']' 2024-09-18T15:51:59.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj77 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:59.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj78 = obj250 ']' 2024-09-18T15:51:59.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj78 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:59.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj79 = obj250 ']' 2024-09-18T15:51:59.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj79 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:59.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj80 = obj250 ']' 2024-09-18T15:51:59.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj80 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:59.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj81 = obj250 ']' 2024-09-18T15:51:59.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj81 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:59.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj82 = obj250 ']' 2024-09-18T15:51:59.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj82 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:59.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj83 = obj250 ']' 2024-09-18T15:51:59.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj83 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.477 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:59.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj84 = obj250 ']' 2024-09-18T15:51:59.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj84 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:59.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj85 = obj250 ']' 2024-09-18T15:51:59.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj85 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:59.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj86 = obj250 ']' 2024-09-18T15:51:59.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj86 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:59.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj87 = obj250 ']' 2024-09-18T15:51:59.563 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj87 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:59.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj88 = obj250 ']' 2024-09-18T15:51:59.591 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj88 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:59.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj89 = obj250 ']' 2024-09-18T15:51:59.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj89 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.666 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:59.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj90 = obj250 ']' 2024-09-18T15:51:59.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj90 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:59.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj91 = obj250 ']' 2024-09-18T15:51:59.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj91 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:59.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj92 = obj250 ']' 2024-09-18T15:51:59.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj92 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:59.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj93 = obj250 ']' 2024-09-18T15:51:59.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj93 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:59.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj94 = obj250 ']' 2024-09-18T15:51:59.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj94 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:59.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj95 = obj250 ']' 2024-09-18T15:51:59.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj95 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:59.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj96 = obj250 ']' 2024-09-18T15:51:59.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj96 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:59.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj97 = obj250 ']' 2024-09-18T15:51:59.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj97 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.947 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.948 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:59.949 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj98 = obj250 ']' 2024-09-18T15:51:59.949 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj98 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:51:59.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:51:59.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj99 = obj250 ']' 2024-09-18T15:51:59.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj99 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:00.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj100 = obj250 ']' 2024-09-18T15:52:00.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj100 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:00.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj101 = obj250 ']' 2024-09-18T15:52:00.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj101 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:00.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj102 = obj250 ']' 2024-09-18T15:52:00.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj102 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:00.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj103 = obj250 ']' 2024-09-18T15:52:00.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj103 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.155 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:00.155 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj104 = obj250 ']' 2024-09-18T15:52:00.155 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj104 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:00.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj105 = obj250 ']' 2024-09-18T15:52:00.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj105 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:00.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj106 = obj250 ']' 2024-09-18T15:52:00.211 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj106 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:00.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj107 = obj250 ']' 2024-09-18T15:52:00.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj107 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.265 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:00.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj108 = obj250 ']' 2024-09-18T15:52:00.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj108 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:00.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj109 = obj250 ']' 2024-09-18T15:52:00.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj109 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:00.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj110 = obj250 ']' 2024-09-18T15:52:00.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj110 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:00.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj111 = obj250 ']' 2024-09-18T15:52:00.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj111 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:00.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj112 = obj250 ']' 2024-09-18T15:52:00.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj112 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.406 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:00.406 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj113 = obj250 ']' 2024-09-18T15:52:00.406 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj113 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:00.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj114 = obj250 ']' 2024-09-18T15:52:00.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj114 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:00.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj115 = obj250 ']' 2024-09-18T15:52:00.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj115 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:00.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj116 = obj250 ']' 2024-09-18T15:52:00.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj116 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:00.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj117 = obj250 ']' 2024-09-18T15:52:00.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj117 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:00.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj118 = obj250 ']' 2024-09-18T15:52:00.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj118 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:00.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj119 = obj250 ']' 2024-09-18T15:52:00.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj119 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:00.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj120 = obj250 ']' 2024-09-18T15:52:00.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj120 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:00.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj121 = obj250 ']' 2024-09-18T15:52:00.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj121 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:00.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj122 = obj250 ']' 2024-09-18T15:52:00.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj122 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:00.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj123 = obj250 ']' 2024-09-18T15:52:00.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj123 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:00.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj124 = obj250 ']' 2024-09-18T15:52:00.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj124 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.740 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.741 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:00.741 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj125 = obj250 ']' 2024-09-18T15:52:00.741 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj125 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:00.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj126 = obj250 ']' 2024-09-18T15:52:00.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj126 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:00.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj127 = obj250 ']' 2024-09-18T15:52:00.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj127 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:00.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj128 = obj250 ']' 2024-09-18T15:52:00.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj128 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:00.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj129 = obj250 ']' 2024-09-18T15:52:00.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj129 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:00.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj130 = obj250 ']' 2024-09-18T15:52:00.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj130 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.907 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:00.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj131 = obj250 ']' 2024-09-18T15:52:00.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj131 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:00.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj132 = obj250 ']' 2024-09-18T15:52:00.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj132 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:00.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj133 = obj250 ']' 2024-09-18T15:52:00.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj133 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.991 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:00.991 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:00.992 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj134 = obj250 ']' 2024-09-18T15:52:00.992 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj134 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:01.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj135 = obj250 ']' 2024-09-18T15:52:01.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj135 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:01.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj136 = obj250 ']' 2024-09-18T15:52:01.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj136 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:01.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj137 = obj250 ']' 2024-09-18T15:52:01.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj137 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:01.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj138 = obj250 ']' 2024-09-18T15:52:01.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj138 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:01.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj139 = obj250 ']' 2024-09-18T15:52:01.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj139 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:01.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj140 = obj250 ']' 2024-09-18T15:52:01.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj140 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:01.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj141 = obj250 ']' 2024-09-18T15:52:01.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj141 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:01.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj142 = obj250 ']' 2024-09-18T15:52:01.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj142 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.242 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:01.242 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj143 = obj250 ']' 2024-09-18T15:52:01.242 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj143 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:01.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj144 = obj250 ']' 2024-09-18T15:52:01.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj144 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:01.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj145 = obj250 ']' 2024-09-18T15:52:01.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj145 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:01.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj146 = obj250 ']' 2024-09-18T15:52:01.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj146 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:01.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj147 = obj250 ']' 2024-09-18T15:52:01.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj147 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:01.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj148 = obj250 ']' 2024-09-18T15:52:01.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj148 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:01.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj149 = obj250 ']' 2024-09-18T15:52:01.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj149 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.437 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:01.437 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj150 = obj250 ']' 2024-09-18T15:52:01.437 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj150 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:01.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj151 = obj250 ']' 2024-09-18T15:52:01.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj151 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.492 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:01.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj152 = obj250 ']' 2024-09-18T15:52:01.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj152 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:01.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj153 = obj250 ']' 2024-09-18T15:52:01.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj153 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.549 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:01.549 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj154 = obj250 ']' 2024-09-18T15:52:01.549 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj154 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:01.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj155 = obj250 ']' 2024-09-18T15:52:01.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj155 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:01.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj156 = obj250 ']' 2024-09-18T15:52:01.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj156 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.660 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:01.660 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj157 = obj250 ']' 2024-09-18T15:52:01.660 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj157 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:01.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj158 = obj250 ']' 2024-09-18T15:52:01.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj158 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:01.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj159 = obj250 ']' 2024-09-18T15:52:01.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj159 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:01.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj160 = obj250 ']' 2024-09-18T15:52:01.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj160 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:01.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj161 = obj250 ']' 2024-09-18T15:52:01.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj161 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:01.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj162 = obj250 ']' 2024-09-18T15:52:01.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj162 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:01.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj163 = obj250 ']' 2024-09-18T15:52:01.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj163 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:01.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj164 = obj250 ']' 2024-09-18T15:52:01.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj164 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:01.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj165 = obj250 ']' 2024-09-18T15:52:01.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj165 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:01.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj166 = obj250 ']' 2024-09-18T15:52:01.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj166 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.938 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:01.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj167 = obj250 ']' 2024-09-18T15:52:01.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj167 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:01.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj168 = obj250 ']' 2024-09-18T15:52:01.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj168 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:01.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:01.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj169 = obj250 ']' 2024-09-18T15:52:01.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj169 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:02.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj170 = obj250 ']' 2024-09-18T15:52:02.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj170 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:02.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj171 = obj250 ']' 2024-09-18T15:52:02.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj171 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:02.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj172 = obj250 ']' 2024-09-18T15:52:02.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj172 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:02.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj173 = obj250 ']' 2024-09-18T15:52:02.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj173 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:02.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj174 = obj250 ']' 2024-09-18T15:52:02.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj174 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:02.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj175 = obj250 ']' 2024-09-18T15:52:02.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj175 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:02.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj176 = obj250 ']' 2024-09-18T15:52:02.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj176 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.242 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:02.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj177 = obj250 ']' 2024-09-18T15:52:02.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj177 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:02.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj178 = obj250 ']' 2024-09-18T15:52:02.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj178 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.299 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:02.299 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj179 = obj250 ']' 2024-09-18T15:52:02.299 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj179 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:02.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj180 = obj250 ']' 2024-09-18T15:52:02.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj180 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:02.355 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj181 = obj250 ']' 2024-09-18T15:52:02.355 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj181 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:02.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj182 = obj250 ']' 2024-09-18T15:52:02.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj182 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:02.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj183 = obj250 ']' 2024-09-18T15:52:02.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj183 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.437 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:02.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj184 = obj250 ']' 2024-09-18T15:52:02.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj184 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:02.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj185 = obj250 ']' 2024-09-18T15:52:02.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj185 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:02.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj186 = obj250 ']' 2024-09-18T15:52:02.494 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj186 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:02.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj187 = obj250 ']' 2024-09-18T15:52:02.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj187 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:02.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj188 = obj250 ']' 2024-09-18T15:52:02.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj188 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:02.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj189 = obj250 ']' 2024-09-18T15:52:02.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj189 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:02.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj190 = obj250 ']' 2024-09-18T15:52:02.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj190 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:02.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj191 = obj250 ']' 2024-09-18T15:52:02.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj191 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:02.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj192 = obj250 ']' 2024-09-18T15:52:02.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj192 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:02.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj193 = obj250 ']' 2024-09-18T15:52:02.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj193 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:02.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj194 = obj250 ']' 2024-09-18T15:52:02.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj194 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:02.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj195 = obj250 ']' 2024-09-18T15:52:02.746 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj195 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:02.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj196 = obj250 ']' 2024-09-18T15:52:02.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj196 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:02.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj197 = obj250 ']' 2024-09-18T15:52:02.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj197 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:02.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj198 = obj250 ']' 2024-09-18T15:52:02.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj198 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:02.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj199 = obj250 ']' 2024-09-18T15:52:02.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj199 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:02.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj200 = obj250 ']' 2024-09-18T15:52:02.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj200 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:02.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj201 = obj250 ']' 2024-09-18T15:52:02.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj201 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:02.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj202 = obj250 ']' 2024-09-18T15:52:02.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj202 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:02.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj203 = obj250 ']' 2024-09-18T15:52:02.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj203 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:02.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:02.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj204 = obj250 ']' 2024-09-18T15:52:02.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj204 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:03.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj205 = obj250 ']' 2024-09-18T15:52:03.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj205 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.052 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:03.052 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj206 = obj250 ']' 2024-09-18T15:52:03.052 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj206 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.079 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.080 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:03.080 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj207 = obj250 ']' 2024-09-18T15:52:03.080 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj207 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:03.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj208 = obj250 ']' 2024-09-18T15:52:03.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj208 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.135 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:03.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj209 = obj250 ']' 2024-09-18T15:52:03.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj209 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:03.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj210 = obj250 ']' 2024-09-18T15:52:03.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj210 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:03.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj211 = obj250 ']' 2024-09-18T15:52:03.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj211 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:03.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj212 = obj250 ']' 2024-09-18T15:52:03.220 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj212 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.247 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:03.248 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj213 = obj250 ']' 2024-09-18T15:52:03.248 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj213 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:03.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj214 = obj250 ']' 2024-09-18T15:52:03.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj214 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:03.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj215 = obj250 ']' 2024-09-18T15:52:03.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj215 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:03.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj216 = obj250 ']' 2024-09-18T15:52:03.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj216 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:03.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj217 = obj250 ']' 2024-09-18T15:52:03.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj217 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:03.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj218 = obj250 ']' 2024-09-18T15:52:03.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj218 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:03.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj219 = obj250 ']' 2024-09-18T15:52:03.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj219 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:03.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj220 = obj250 ']' 2024-09-18T15:52:03.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj220 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.470 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:03.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj221 = obj250 ']' 2024-09-18T15:52:03.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj221 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:03.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj222 = obj250 ']' 2024-09-18T15:52:03.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj222 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:03.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj223 = obj250 ']' 2024-09-18T15:52:03.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj223 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:03.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj224 = obj250 ']' 2024-09-18T15:52:03.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj224 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:03.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj225 = obj250 ']' 2024-09-18T15:52:03.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj225 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.612 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:03.612 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj226 = obj250 ']' 2024-09-18T15:52:03.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj226 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.660 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:03.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj227 = obj250 ']' 2024-09-18T15:52:03.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj227 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:03.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj228 = obj250 ']' 2024-09-18T15:52:03.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj228 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:03.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj229 = obj250 ']' 2024-09-18T15:52:03.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj229 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:03.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj230 = obj250 ']' 2024-09-18T15:52:03.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj230 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:03.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj231 = obj250 ']' 2024-09-18T15:52:03.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj231 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:03.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj232 = obj250 ']' 2024-09-18T15:52:03.801 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj232 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:03.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj233 = obj250 ']' 2024-09-18T15:52:03.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj233 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:03.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj234 = obj250 ']' 2024-09-18T15:52:03.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj234 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:03.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj235 = obj250 ']' 2024-09-18T15:52:03.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj235 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:03.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj236 = obj250 ']' 2024-09-18T15:52:03.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj236 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:03.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj237 = obj250 ']' 2024-09-18T15:52:03.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj237 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:03.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj238 = obj250 ']' 2024-09-18T15:52:03.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj238 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:03.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:03.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj239 = obj250 ']' 2024-09-18T15:52:03.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj239 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:04.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj240 = obj250 ']' 2024-09-18T15:52:04.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj240 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:04.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj241 = obj250 ']' 2024-09-18T15:52:04.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj241 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:04.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj242 = obj250 ']' 2024-09-18T15:52:04.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj242 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:04.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj243 = obj250 ']' 2024-09-18T15:52:04.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj243 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:04.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj244 = obj250 ']' 2024-09-18T15:52:04.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj244 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:04.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj245 = obj250 ']' 2024-09-18T15:52:04.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj245 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:04.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj246 = obj250 ']' 2024-09-18T15:52:04.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj246 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:04.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj247 = obj250 ']' 2024-09-18T15:52:04.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj247 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:04.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj248 = obj250 ']' 2024-09-18T15:52:04.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj248 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:04.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj249 = obj250 ']' 2024-09-18T15:52:04.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj249 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:04.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj250 = obj250 ']' 2024-09-18T15:52:04.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:329: TEST_rep_backfill_unfound: rados -p test-pool get obj250 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.370 INFO:tasks.workunit.client.0.smithi047.stderr:error getting test-pool/obj250: (2) No such file or directory 2024-09-18T15:52:04.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:04.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj251 = obj250 ']' 2024-09-18T15:52:04.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj251 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:04.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj252 = obj250 ']' 2024-09-18T15:52:04.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj252 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:04.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj253 = obj250 ']' 2024-09-18T15:52:04.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj253 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:04.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj254 = obj250 ']' 2024-09-18T15:52:04.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj254 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.483 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:04.483 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj255 = obj250 ']' 2024-09-18T15:52:04.483 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj255 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.510 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:04.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj256 = obj250 ']' 2024-09-18T15:52:04.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj256 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:04.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj257 = obj250 ']' 2024-09-18T15:52:04.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj257 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:04.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj258 = obj250 ']' 2024-09-18T15:52:04.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj258 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:04.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj259 = obj250 ']' 2024-09-18T15:52:04.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj259 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.622 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:04.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj260 = obj250 ']' 2024-09-18T15:52:04.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj260 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:04.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj261 = obj250 ']' 2024-09-18T15:52:04.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj261 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:04.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj262 = obj250 ']' 2024-09-18T15:52:04.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj262 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:04.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj263 = obj250 ']' 2024-09-18T15:52:04.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj263 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:04.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj264 = obj250 ']' 2024-09-18T15:52:04.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj264 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:04.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj265 = obj250 ']' 2024-09-18T15:52:04.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj265 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:04.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj266 = obj250 ']' 2024-09-18T15:52:04.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj266 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:04.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj267 = obj250 ']' 2024-09-18T15:52:04.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj267 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:04.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj268 = obj250 ']' 2024-09-18T15:52:04.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj268 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.875 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:04.875 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj269 = obj250 ']' 2024-09-18T15:52:04.875 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj269 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.902 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:04.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj270 = obj250 ']' 2024-09-18T15:52:04.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj270 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:04.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj271 = obj250 ']' 2024-09-18T15:52:04.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj271 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:04.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj272 = obj250 ']' 2024-09-18T15:52:04.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj272 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.985 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:04.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:04.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj273 = obj250 ']' 2024-09-18T15:52:04.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj273 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:05.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj274 = obj250 ']' 2024-09-18T15:52:05.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj274 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:05.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj275 = obj250 ']' 2024-09-18T15:52:05.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj275 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:05.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj276 = obj250 ']' 2024-09-18T15:52:05.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj276 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.096 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:05.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj277 = obj250 ']' 2024-09-18T15:52:05.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj277 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:05.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj278 = obj250 ']' 2024-09-18T15:52:05.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj278 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:05.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj279 = obj250 ']' 2024-09-18T15:52:05.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj279 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:05.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj280 = obj250 ']' 2024-09-18T15:52:05.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj280 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:05.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj281 = obj250 ']' 2024-09-18T15:52:05.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj281 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:05.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj282 = obj250 ']' 2024-09-18T15:52:05.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj282 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.265 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.265 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:05.265 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj283 = obj250 ']' 2024-09-18T15:52:05.265 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj283 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:05.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj284 = obj250 ']' 2024-09-18T15:52:05.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj284 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:05.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj285 = obj250 ']' 2024-09-18T15:52:05.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj285 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:05.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj286 = obj250 ']' 2024-09-18T15:52:05.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj286 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:05.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj287 = obj250 ']' 2024-09-18T15:52:05.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj287 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:05.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj288 = obj250 ']' 2024-09-18T15:52:05.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj288 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.432 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:05.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj289 = obj250 ']' 2024-09-18T15:52:05.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj289 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:05.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj290 = obj250 ']' 2024-09-18T15:52:05.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj290 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:05.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj291 = obj250 ']' 2024-09-18T15:52:05.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj291 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:05.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj292 = obj250 ']' 2024-09-18T15:52:05.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj292 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:05.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj293 = obj250 ']' 2024-09-18T15:52:05.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj293 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:05.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj294 = obj250 ']' 2024-09-18T15:52:05.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj294 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.599 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:05.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj295 = obj250 ']' 2024-09-18T15:52:05.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj295 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:05.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj296 = obj250 ']' 2024-09-18T15:52:05.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj296 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:05.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj297 = obj250 ']' 2024-09-18T15:52:05.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj297 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.699 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:05.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj298 = obj250 ']' 2024-09-18T15:52:05.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj298 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:05.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj299 = obj250 ']' 2024-09-18T15:52:05.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj299 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:52:05.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj300 = obj250 ']' 2024-09-18T15:52:05.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj300 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:336: TEST_rep_backfill_unfound: rm -f td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:52:05.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:338: TEST_rep_backfill_unfound: delete_pool test-pool 2024-09-18T15:52:05.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test-pool 2024-09-18T15:52:05.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test-pool test-pool --yes-i-really-really-mean-it 2024-09-18T15:52:06.268 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test-pool' does not exist 2024-09-18T15:52:06.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:43: run: teardown td/osd-rep-recov-eio.sh 2024-09-18T15:52:06.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:52:06.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:52:06.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-09-18T15:52:06.280 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:52:06.280 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:52:06.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:52:06.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:52:06.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:52:06.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:52:06.387 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:52:06.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:52:06.388 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:52:06.389 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:52:06.389 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:52:06.390 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:52:06.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:52:06.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:52:06.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:52:06.391 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:52:06.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:52:06.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:52:06.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-09-18T15:52:06.426 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:52:06.426 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:52:06.426 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:52:06.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.397939 2024-09-18T15:52:06.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:52:06.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:52:06.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:35: run: for func in $funcs 2024-09-18T15:52:06.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:36: run: setup td/osd-rep-recov-eio.sh 2024-09-18T15:52:06.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:52:06.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-rep-recov-eio.sh 2024-09-18T15:52:06.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:52:06.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:52:06.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-09-18T15:52:06.429 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:52:06.429 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:52:06.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:52:06.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:52:06.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:52:06.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:52:06.430 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:52:06.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:52:06.431 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:52:06.432 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:52:06.432 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:52:06.432 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:52:06.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:52:06.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:52:06.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:52:06.433 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:52:06.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:52:06.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:52:06.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-09-18T15:52:06.436 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:52:06.436 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:52:06.436 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:52:06.437 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.397939 2024-09-18T15:52:06.437 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:52:06.437 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:52:06.437 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-rep-recov-eio.sh 2024-09-18T15:52:06.439 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T15:52:06.439 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:52:06.439 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:52:06.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.397939 2024-09-18T15:52:06.441 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T15:52:06.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T15:52:06.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T15:52:06.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-rep-recov-eio.sh 1' TERM HUP INT 2024-09-18T15:52:06.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:38: run: run_mon td/osd-rep-recov-eio.sh a --mon_osd_warn_num_repaired=10 2024-09-18T15:52:06.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:52:06.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T15:52:06.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T15:52:06.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T15:52:06.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-rep-recov-eio.sh/a 2024-09-18T15:52:06.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-rep-recov-eio.sh/a --run-dir=td/osd-rep-recov-eio.sh --mon_osd_warn_num_repaired=10 2024-09-18T15:52:06.790 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T15:52:06.790 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:52:06.790 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:52:06.790 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:52:06.790 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:52:06.790 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:52:06.790 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:52:06.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-rep-recov-eio.sh/a '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --mon-cluster-log-file=td/osd-rep-recov-eio.sh/log --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_osd_warn_num_repaired=10 2024-09-18T15:52:07.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T15:52:07.050 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T15:52:07.050 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:52:07.050 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:52:07.050 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T15:52:07.050 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T15:52:07.050 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:52:07.050 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:52:07.050 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:52:07.051 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:52:07.051 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:52:07.051 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:52:07.051 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-mon.a.asok 2024-09-18T15:52:07.051 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:52:07.051 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.397939/ceph-mon.a.asok config get fsid 2024-09-18T15:52:07.130 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T15:52:07.130 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:52:07.130 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:52:07.130 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T15:52:07.130 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T15:52:07.131 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:52:07.131 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:52:07.131 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:52:07.131 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:52:07.132 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:52:07.132 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:52:07.132 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-mon.a.asok 2024-09-18T15:52:07.132 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:52:07.132 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.397939/ceph-mon.a.asok config get mon_host 2024-09-18T15:52:07.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:39: run: run_mgr td/osd-rep-recov-eio.sh x 2024-09-18T15:52:07.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:52:07.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T15:52:07.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T15:52:07.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T15:52:07.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-rep-recov-eio.sh/x 2024-09-18T15:52:07.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T15:52:08.056 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T15:52:08.056 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:52:08.056 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:52:08.056 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:52:08.056 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:52:08.056 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:52:08.056 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:52:08.057 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:52:08.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-rep-recov-eio.sh/x '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:52:08.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:40: run: ceph osd pool create foo 8 2024-09-18T15:52:08.377 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'foo' already exists 2024-09-18T15:52:08.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:42: run: TEST_rep_read_unfound td/osd-rep-recov-eio.sh 2024-09-18T15:52:08.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:420: TEST_rep_read_unfound: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:52:08.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:421: TEST_rep_read_unfound: local objname=myobject 2024-09-18T15:52:08.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:423: TEST_rep_read_unfound: setup_osds 3 _filestore 2024-09-18T15:52:08.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:48: setup_osds: local count=3 2024-09-18T15:52:08.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:49: setup_osds: shift 2024-09-18T15:52:08.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:50: setup_osds: local type=_filestore 2024-09-18T15:52:08.388 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: expr 3 - 1 2024-09-18T15:52:08.388 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: seq 0 2 2024-09-18T15:52:08.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T15:52:08.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd_filestore td/osd-rep-recov-eio.sh 0 2024-09-18T15:52:08.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:52:08.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-18T15:52:08.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=0 2024-09-18T15:52:08.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-18T15:52:08.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-rep-recov-eio.sh/0 2024-09-18T15:52:08.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-18T15:52:08.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:52:08.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:52:08.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:52:08.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-09-18T15:52:08.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-09-18T15:52:08.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-18T15:52:08.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-18T15:52:08.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-18T15:52:08.393 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-18T15:52:08.393 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:52:08.393 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:52:08.394 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:52:08.394 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:52:08.394 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:52:08.394 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:52:08.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:52:08.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-18T15:52:08.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-18T15:52:08.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-18T15:52:08.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-18T15:52:08.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-18T15:52:08.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:52:08.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:52:08.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:52:08.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-18T15:52:08.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-18T15:52:08.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-rep-recov-eio.sh/0 2024-09-18T15:52:08.395 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-18T15:52:08.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=247c49c4-b63b-413e-968c-6470461a19ef 2024-09-18T15:52:08.395 INFO:tasks.workunit.client.0.smithi047.stdout:add osd2 247c49c4-b63b-413e-968c-6470461a19ef 2024-09-18T15:52:08.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd2 247c49c4-b63b-413e-968c-6470461a19ef' 2024-09-18T15:52:08.396 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-18T15:52:08.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQAo9+pmRVdXGBAAqtuosYfMMV2dptue69qJLA== 2024-09-18T15:52:08.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQAo9+pmRVdXGBAAqtuosYfMMV2dptue69qJLA=="}' 2024-09-18T15:52:08.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new 247c49c4-b63b-413e-968c-6470461a19ef -i td/osd-rep-recov-eio.sh/0/new.json 2024-09-18T15:52:08.706 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:52:08.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-rep-recov-eio.sh/0/new.json 2024-09-18T15:52:08.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 0 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAo9+pmRVdXGBAAqtuosYfMMV2dptue69qJLA== --osd-uuid 247c49c4-b63b-413e-968c-6470461a19ef --osd-objectstore=filestore 2024-09-18T15:52:08.743 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:08.742+0000 7f30fcb21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:08.751 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:08.750+0000 7f30fcb21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:08.752 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:08.751+0000 7f30fcb21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:09.169 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:09.168+0000 7f30fcb21440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:52:09.648 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:09.647+0000 7f30fcb21440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:52:09.648 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:09.647+0000 7f30fcb21440 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:52:09.648 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:09.647+0000 7f30fcb21440 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:52:10.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-rep-recov-eio.sh/0/keyring 2024-09-18T15:52:10.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-18T15:52:10.194 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T15:52:10.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd0 key to auth repository 2024-09-18T15:52:10.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-rep-recov-eio.sh/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:52:10.344 INFO:tasks.workunit.client.0.smithi047.stderr:Error EINVAL: key for osd.0 not found in provided keyring 2024-09-18T15:52:10.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.0 2024-09-18T15:52:10.347 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T15:52:10.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 0 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:52:10.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-18T15:52:10.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-18T15:52:10.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-18T15:52:10.372 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:10.371+0000 7f5ddb526440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:10.375 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:10.374+0000 7f5ddb526440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:10.376 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:10.375+0000 7f5ddb526440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:10.377 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:10.377+0000 7f5ddb526440 -1 Falling back to public interface 2024-09-18T15:52:10.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 0 2024-09-18T15:52:10.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:52:10.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:52:10.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:52:10.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:52:10.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:10.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:52:10.663 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:52:10.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:10.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:52:10.838 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:10.837+0000 7f5ddb526440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:52:10.846 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:10.845+0000 7f5ddb526440 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:52:10.846 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:10.845+0000 7f5ddb526440 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:52:10.850 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:10.849+0000 7f5ddb526440 -1 osd.0 0 log_to_monitors true 2024-09-18T15:52:10.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:52:11.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:52:11.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:11.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:52:11.985 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:52:11.985 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:11.985 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:52:12.326 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/441968,v1:127.0.0.1:6801/441968] [v2:127.0.0.1:6802/441968,v1:127.0.0.1:6803/441968] exists,up 247c49c4-b63b-413e-968c-6470461a19ef 2024-09-18T15:52:12.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:52:12.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:52:12.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:52:12.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T15:52:12.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd_filestore td/osd-rep-recov-eio.sh 1 2024-09-18T15:52:12.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:52:12.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-18T15:52:12.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=1 2024-09-18T15:52:12.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-18T15:52:12.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-rep-recov-eio.sh/1 2024-09-18T15:52:12.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-18T15:52:12.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:52:12.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:52:12.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:52:12.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2024-09-18T15:52:12.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2024-09-18T15:52:12.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-18T15:52:12.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-18T15:52:12.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-18T15:52:12.331 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-18T15:52:12.331 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:52:12.331 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:52:12.331 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:52:12.331 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:52:12.332 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:52:12.332 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:52:12.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:52:12.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-18T15:52:12.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-18T15:52:12.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-18T15:52:12.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-18T15:52:12.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-18T15:52:12.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:52:12.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:52:12.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:52:12.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-18T15:52:12.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-18T15:52:12.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-rep-recov-eio.sh/1 2024-09-18T15:52:12.333 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-18T15:52:12.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=dadb0228-872b-47b3-855d-8193f4acf2a5 2024-09-18T15:52:12.333 INFO:tasks.workunit.client.0.smithi047.stdout:add osd2 dadb0228-872b-47b3-855d-8193f4acf2a5 2024-09-18T15:52:12.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd2 dadb0228-872b-47b3-855d-8193f4acf2a5' 2024-09-18T15:52:12.334 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-18T15:52:12.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQAs9+pmM0yaFBAAsD5FmHbxbcLK62oXsZH8TQ== 2024-09-18T15:52:12.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQAs9+pmM0yaFBAAsD5FmHbxbcLK62oXsZH8TQ=="}' 2024-09-18T15:52:12.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new dadb0228-872b-47b3-855d-8193f4acf2a5 -i td/osd-rep-recov-eio.sh/1/new.json 2024-09-18T15:52:13.086 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:52:13.096 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-rep-recov-eio.sh/1/new.json 2024-09-18T15:52:13.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 1 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAs9+pmM0yaFBAAsD5FmHbxbcLK62oXsZH8TQ== --osd-uuid dadb0228-872b-47b3-855d-8193f4acf2a5 --osd-objectstore=filestore 2024-09-18T15:52:13.124 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:13.123+0000 7f3145c31440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:13.152 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:13.151+0000 7f3145c31440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:13.153 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:13.152+0000 7f3145c31440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:13.824 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:13.823+0000 7f3145c31440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:52:14.596 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:14.595+0000 7f3145c31440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:52:14.596 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:14.595+0000 7f3145c31440 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:52:14.596 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:14.595+0000 7f3145c31440 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:52:15.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-rep-recov-eio.sh/1/keyring 2024-09-18T15:52:15.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-18T15:52:15.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd1 key to auth repository 2024-09-18T15:52:15.144 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T15:52:15.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-rep-recov-eio.sh/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:52:15.481 INFO:tasks.workunit.client.0.smithi047.stderr:Error EINVAL: key for osd.1 not found in provided keyring 2024-09-18T15:52:15.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.1 2024-09-18T15:52:15.485 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T15:52:15.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 1 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:52:15.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-18T15:52:15.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-18T15:52:15.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-18T15:52:15.509 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:15.509+0000 7fd626c0f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:15.512 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:15.511+0000 7fd626c0f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:15.513 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:15.512+0000 7fd626c0f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:15.514 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:15.513+0000 7fd626c0f440 -1 Falling back to public interface 2024-09-18T15:52:15.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 1 2024-09-18T15:52:15.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:52:15.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:52:15.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:52:15.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:52:15.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:15.953 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:52:15.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:52:15.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:15.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:52:16.103 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:16.102+0000 7fd626c0f440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:52:16.111 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:16.110+0000 7fd626c0f440 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:52:16.111 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:16.110+0000 7fd626c0f440 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:52:16.115 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:16.114+0000 7fd626c0f440 -1 osd.1 0 log_to_monitors true 2024-09-18T15:52:16.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:52:17.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:52:17.419 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:17.419 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:52:17.419 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:52:17.419 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:17.419 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:52:17.836 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/442329,v1:127.0.0.1:6811/442329] [v2:127.0.0.1:6812/442329,v1:127.0.0.1:6813/442329] exists,up dadb0228-872b-47b3-855d-8193f4acf2a5 2024-09-18T15:52:17.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:52:17.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:52:17.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:52:17.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T15:52:17.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd_filestore td/osd-rep-recov-eio.sh 2 2024-09-18T15:52:17.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:52:17.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-18T15:52:17.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=2 2024-09-18T15:52:17.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-18T15:52:17.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-rep-recov-eio.sh/2 2024-09-18T15:52:17.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-18T15:52:17.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:52:17.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:52:17.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:52:17.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2024-09-18T15:52:17.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2024-09-18T15:52:17.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-18T15:52:17.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-18T15:52:17.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-18T15:52:17.841 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-18T15:52:17.841 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:52:17.841 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:52:17.841 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:52:17.841 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:52:17.841 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:52:17.841 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:52:17.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:52:17.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-18T15:52:17.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-18T15:52:17.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-18T15:52:17.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-18T15:52:17.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-18T15:52:17.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:52:17.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:52:17.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:52:17.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-18T15:52:17.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-18T15:52:17.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-rep-recov-eio.sh/2 2024-09-18T15:52:17.842 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-18T15:52:17.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=a5592e3d-fc56-4a8a-840e-fd50a64c0b63 2024-09-18T15:52:17.843 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd2 a5592e3d-fc56-4a8a-840e-fd50a64c0b63' 2024-09-18T15:52:17.843 INFO:tasks.workunit.client.0.smithi047.stdout:add osd2 a5592e3d-fc56-4a8a-840e-fd50a64c0b63 2024-09-18T15:52:17.843 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-18T15:52:17.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQAx9+pmPr3+MhAA0mARd9cdSiu+w1a0GAtEXg== 2024-09-18T15:52:17.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQAx9+pmPr3+MhAA0mARd9cdSiu+w1a0GAtEXg=="}' 2024-09-18T15:52:17.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new a5592e3d-fc56-4a8a-840e-fd50a64c0b63 -i td/osd-rep-recov-eio.sh/2/new.json 2024-09-18T15:52:18.536 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:52:18.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-rep-recov-eio.sh/2/new.json 2024-09-18T15:52:18.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 2 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAx9+pmPr3+MhAA0mARd9cdSiu+w1a0GAtEXg== --osd-uuid a5592e3d-fc56-4a8a-840e-fd50a64c0b63 --osd-objectstore=filestore 2024-09-18T15:52:18.574 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:18.573+0000 7f435ac0e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:18.576 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:18.575+0000 7f435ac0e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:18.577 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:18.576+0000 7f435ac0e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:19.232 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:19.231+0000 7f435ac0e440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:52:19.829 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:19.827+0000 7f435ac0e440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:52:19.829 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:19.828+0000 7f435ac0e440 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:52:19.829 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:19.828+0000 7f435ac0e440 -1 journal do_read_entry(4096): bad header magic 2024-09-18T15:52:20.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-rep-recov-eio.sh/2/keyring 2024-09-18T15:52:20.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-18T15:52:20.609 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd2 key to auth repository 2024-09-18T15:52:20.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd2 key to auth repository 2024-09-18T15:52:20.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-rep-recov-eio.sh/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:52:21.217 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T15:52:21.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.2 2024-09-18T15:52:21.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 2 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:52:21.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-18T15:52:21.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-18T15:52:21.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-18T15:52:21.242 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:21.241+0000 7f6e9231d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:21.244 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:21.243+0000 7f6e9231d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:21.245 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:21.244+0000 7f6e9231d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:21.246 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:21.245+0000 7f6e9231d440 -1 Falling back to public interface 2024-09-18T15:52:21.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 2 2024-09-18T15:52:21.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:52:21.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:52:21.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:52:21.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:52:21.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:21.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:52:21.651 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:52:21.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:21.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:52:21.653 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:21.652+0000 7f6e9231d440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:52:21.661 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:21.660+0000 7f6e9231d440 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:52:21.661 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:21.660+0000 7f6e9231d440 -1 journal do_read_entry(8192): bad header magic 2024-09-18T15:52:21.665 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:21.664+0000 7f6e9231d440 -1 osd.2 0 log_to_monitors true 2024-09-18T15:52:22.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:52:23.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:52:23.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:23.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:52:23.112 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:52:23.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:23.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:52:23.537 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/442613,v1:127.0.0.1:6819/442613] [v2:127.0.0.1:6820/442613,v1:127.0.0.1:6821/442613] exists,up a5592e3d-fc56-4a8a-840e-fd50a64c0b63 2024-09-18T15:52:23.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:52:23.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:52:23.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:52:23.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:55: setup_osds: wait_for_clean 2024-09-18T15:52:23.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:52:23.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:52:23.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:52:23.539 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:52:23.539 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:52:23.539 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:52:23.540 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:52:23.540 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:52:23.540 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:52:23.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:52:23.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:52:23.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:52:23.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:52:23.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:52:23.630 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:52:24.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:52:24.003 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:52:24.003 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T15:52:24.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:52:24.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:52:24.003 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:52:24.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803781 2024-09-18T15:52:24.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803781 2024-09-18T15:52:24.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803781' 2024-09-18T15:52:24.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:52:24.113 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:52:24.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574852 2024-09-18T15:52:24.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574852 2024-09-18T15:52:24.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803781 1-55834574852' 2024-09-18T15:52:24.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:52:24.347 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:52:24.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378627 2024-09-18T15:52:24.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378627 2024-09-18T15:52:24.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803781 1-55834574852 2-81604378627' 2024-09-18T15:52:24.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:52:24.455 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803781 2024-09-18T15:52:24.455 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:52:24.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:52:24.456 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803781 2024-09-18T15:52:24.457 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:52:24.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803781 2024-09-18T15:52:24.458 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 25769803781 2024-09-18T15:52:24.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803781' 2024-09-18T15:52:24.458 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:52:24.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803779 -lt 25769803781 2024-09-18T15:52:24.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:52:25.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:52:25.846 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:52:26.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803781 -lt 25769803781 2024-09-18T15:52:26.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:52:26.229 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574852 2024-09-18T15:52:26.229 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:52:26.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:52:26.230 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574852 2024-09-18T15:52:26.230 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:52:26.231 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574852 2024-09-18T15:52:26.231 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 55834574852 2024-09-18T15:52:26.231 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574852' 2024-09-18T15:52:26.231 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:52:26.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574852 -lt 55834574852 2024-09-18T15:52:26.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:52:26.754 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378627 2024-09-18T15:52:26.755 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:52:26.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:52:26.756 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378627 2024-09-18T15:52:26.756 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:52:26.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378627 2024-09-18T15:52:26.757 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 81604378627 2024-09-18T15:52:26.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378627' 2024-09-18T15:52:26.758 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:52:27.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378627 -lt 81604378627 2024-09-18T15:52:27.224 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:52:27.224 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:52:27.224 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:52:27.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-18T15:52:27.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:52:27.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:52:27.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:52:27.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:52:27.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:52:27.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:52:27.724 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:52:28.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-18T15:52:28.086 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:52:28.087 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:52:28.087 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:52:28.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-18T15:52:28.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:52:28.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:52:28.586 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:425: TEST_rep_read_unfound: ceph osd pool delete foo foo --yes-i-really-really-mean-it 2024-09-18T15:52:29.185 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'foo' does not exist 2024-09-18T15:52:29.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:426: TEST_rep_read_unfound: local poolname=test-pool 2024-09-18T15:52:29.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:427: TEST_rep_read_unfound: create_pool test-pool 1 1 2024-09-18T15:52:29.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test-pool 1 1 2024-09-18T15:52:29.781 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test-pool' already exists 2024-09-18T15:52:29.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:52:30.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:428: TEST_rep_read_unfound: ceph osd pool set test-pool size 2 2024-09-18T15:52:31.998 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 2 size to 2 2024-09-18T15:52:32.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:429: TEST_rep_read_unfound: wait_for_clean 2024-09-18T15:52:32.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:52:32.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:52:32.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:52:32.012 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:52:32.013 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:52:32.013 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:52:32.013 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:52:32.013 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:52:32.013 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:52:32.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:52:32.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:52:32.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:52:32.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:52:32.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:52:32.111 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:52:32.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:52:32.596 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:52:32.596 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T15:52:32.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:52:32.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:52:32.596 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:52:32.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803784 2024-09-18T15:52:32.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803784 2024-09-18T15:52:32.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803784' 2024-09-18T15:52:32.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:52:32.705 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:52:32.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574855 2024-09-18T15:52:32.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574855 2024-09-18T15:52:32.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803784 1-55834574855' 2024-09-18T15:52:32.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:52:32.814 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:52:32.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378630 2024-09-18T15:52:32.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378630 2024-09-18T15:52:32.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803784 1-55834574855 2-81604378630' 2024-09-18T15:52:32.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:52:32.922 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803784 2024-09-18T15:52:32.922 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:52:32.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:52:32.923 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803784 2024-09-18T15:52:32.923 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:52:32.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803784 2024-09-18T15:52:32.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803784' 2024-09-18T15:52:32.925 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 25769803784 2024-09-18T15:52:32.925 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:52:33.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803782 -lt 25769803784 2024-09-18T15:52:33.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:52:34.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:52:34.322 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:52:34.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803785 -lt 25769803784 2024-09-18T15:52:34.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:52:34.749 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574855 2024-09-18T15:52:34.749 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:52:34.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:52:34.750 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574855 2024-09-18T15:52:34.750 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:52:34.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574855 2024-09-18T15:52:34.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574855' 2024-09-18T15:52:34.752 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 55834574855 2024-09-18T15:52:34.752 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:52:35.157 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574856 -lt 55834574855 2024-09-18T15:52:35.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:52:35.158 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378630 2024-09-18T15:52:35.158 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:52:35.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:52:35.159 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378630 2024-09-18T15:52:35.160 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:52:35.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378630 2024-09-18T15:52:35.161 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 81604378630 2024-09-18T15:52:35.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378630' 2024-09-18T15:52:35.161 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:52:35.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378631 -lt 81604378630 2024-09-18T15:52:35.531 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:52:35.531 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:52:35.532 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:52:36.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:52:36.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:52:36.029 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:52:36.029 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:52:36.030 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:52:36.030 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:52:36.030 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:52:36.030 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:52:36.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:52:36.455 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:52:36.455 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:52:36.455 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:52:36.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:52:36.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:52:36.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:52:36.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:431: TEST_rep_read_unfound: ceph pg dump pgs 2024-09-18T15:52:37.324 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T15:52:37.324 INFO:tasks.workunit.client.0.smithi047.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:52:32.155574+0000 0'0 27:22 [2,1] 2 [2,1] 2 0'0 2024-09-18T15:52:29.509511+0000 0'0 2024-09-18T15:52:29.509511+0000 0 0 periodic scrub scheduled @ 2024-09-19T18:57:45.723673+0000 0 0 2024-09-18T15:52:37.324 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T15:52:37.324 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T15:52:37.324 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T15:52:37.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:433: TEST_rep_read_unfound: dd if=/dev/urandom bs=8k count=1 of=td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:52:37.335 INFO:tasks.workunit.client.0.smithi047.stderr:1+0 records in 2024-09-18T15:52:37.335 INFO:tasks.workunit.client.0.smithi047.stderr:1+0 records out 2024-09-18T15:52:37.335 INFO:tasks.workunit.client.0.smithi047.stderr:8192 bytes (8.2 kB, 8.0 KiB) copied, 6.8699e-05 s, 119 MB/s 2024-09-18T15:52:37.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:434: TEST_rep_read_unfound: rados -p test-pool put myobject td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:52:37.505 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:436: TEST_rep_read_unfound: get_primary test-pool myobject 2024-09-18T15:52:37.505 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test-pool 2024-09-18T15:52:37.505 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=myobject 2024-09-18T15:52:37.505 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test-pool myobject 2024-09-18T15:52:37.505 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:52:37.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:436: TEST_rep_read_unfound: local primary=2 2024-09-18T15:52:37.881 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:437: TEST_rep_read_unfound: get_not_primary test-pool myobject 2024-09-18T15:52:37.881 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test-pool 2024-09-18T15:52:37.882 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=myobject 2024-09-18T15:52:37.882 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test-pool myobject 2024-09-18T15:52:37.882 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test-pool 2024-09-18T15:52:37.882 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=myobject 2024-09-18T15:52:37.882 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test-pool myobject 2024-09-18T15:52:37.882 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T15:52:38.276 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-09-18T15:52:38.276 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test-pool myobject 2024-09-18T15:52:38.276 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-09-18T15:52:38.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:437: TEST_rep_read_unfound: local other=1 2024-09-18T15:52:38.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:439: TEST_rep_read_unfound: dd if=/dev/urandom bs=8k count=1 of=td/osd-rep-recov-eio.sh/CORRUPT 2024-09-18T15:52:38.658 INFO:tasks.workunit.client.0.smithi047.stderr:1+0 records in 2024-09-18T15:52:38.658 INFO:tasks.workunit.client.0.smithi047.stderr:1+0 records out 2024-09-18T15:52:38.659 INFO:tasks.workunit.client.0.smithi047.stderr:8192 bytes (8.2 kB, 8.0 KiB) copied, 7.0742e-05 s, 116 MB/s 2024-09-18T15:52:38.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:440: TEST_rep_read_unfound: objectstore_tool td/osd-rep-recov-eio.sh 2 myobject set-bytes td/osd-rep-recov-eio.sh/CORRUPT 2024-09-18T15:52:38.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:52:38.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:52:38.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-09-18T15:52:38.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:52:38.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-rep-recov-eio.sh 2 myobject set-bytes td/osd-rep-recov-eio.sh/CORRUPT 2024-09-18T15:52:38.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:52:38.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:52:38.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-09-18T15:52:38.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:52:38.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-rep-recov-eio.sh TERM osd.2 2024-09-18T15:52:38.659 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:52:38.659 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:52:38.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:52:38.660 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:52:38.660 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:52:38.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:52:38.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-rep-recov-eio.sh 2 myobject set-bytes td/osd-rep-recov-eio.sh/CORRUPT 2024-09-18T15:52:38.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:52:38.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:52:38.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-09-18T15:52:38.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:52:38.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-rep-recov-eio.sh/2 2024-09-18T15:52:38.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-rep-recov-eio.sh/2 myobject set-bytes td/osd-rep-recov-eio.sh/CORRUPT 2024-09-18T15:52:39.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-rep-recov-eio.sh 2 2024-09-18T15:52:39.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:52:39.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:52:39.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T15:52:39.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:52:39.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-rep-recov-eio.sh/2 2024-09-18T15:52:39.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-18T15:52:39.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:52:39.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:52:39.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:52:39.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2024-09-18T15:52:39.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2024-09-18T15:52:39.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:52:39.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:52:39.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-18T15:52:39.975 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:52:39.975 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:52:39.975 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:52:39.975 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:52:39.975 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:52:39.975 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:52:39.975 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:52:39.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:52:39.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:52:39.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-18T15:52:39.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-18T15:52:39.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:52:39.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:52:39.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:52:39.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:52:39.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:52:39.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:52:39.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-rep-recov-eio.sh/2 2024-09-18T15:52:39.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T15:52:39.977 INFO:tasks.workunit.client.0.smithi047.stderr:start osd.2 2024-09-18T15:52:39.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:52:39.977 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-rep-recov-eio.sh/2/whoami 2024-09-18T15:52:39.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T15:52:39.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:52:39.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:52:39.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:52:40.002 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:40.001+0000 7fdf4f535440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:40.005 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:40.004+0000 7fdf4f535440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:40.006 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:40.005+0000 7fdf4f535440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:40.007 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:40.007+0000 7fdf4f535440 -1 Falling back to public interface 2024-09-18T15:52:40.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T15:52:40.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:52:40.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:52:40.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:52:40.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:52:40.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:40.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:52:40.439 INFO:tasks.workunit.client.0.smithi047.stderr:0 2024-09-18T15:52:40.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:40.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:52:40.526 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:40.526+0000 7fdf4f535440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:52:40.534 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:40.533+0000 7fdf4f535440 -1 journal do_read_entry(454656): bad header magic 2024-09-18T15:52:40.534 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:40.533+0000 7fdf4f535440 -1 journal do_read_entry(454656): bad header magic 2024-09-18T15:52:40.539 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:40.538+0000 7fdf4f535440 -1 osd.2 28 log_to_monitors true 2024-09-18T15:52:40.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:52:41.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:52:41.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:41.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:52:41.887 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:52:41.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:41.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:52:42.272 INFO:tasks.workunit.client.0.smithi047.stderr:osd.2 up in weight 1 up_from 32 up_thru 32 down_at 29 last_clean_interval [19,28) [v2:127.0.0.1:6818/443690,v1:127.0.0.1:6819/443690] [v2:127.0.0.1:6820/443690,v1:127.0.0.1:6821/443690] exists,up a5592e3d-fc56-4a8a-840e-fd50a64c0b63 2024-09-18T15:52:42.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:52:42.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:52:42.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:52:42.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:52:42.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:52:42.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:52:42.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:52:42.273 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:52:42.273 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:52:42.273 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:52:42.273 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:52:42.273 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:52:42.273 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:52:42.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:52:42.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:52:42.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:52:42.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:52:42.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:52:42.364 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:52:42.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:52:42.845 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:52:42.845 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T15:52:42.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:52:42.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:52:42.846 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:52:42.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803788 2024-09-18T15:52:42.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803788 2024-09-18T15:52:42.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803788' 2024-09-18T15:52:42.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:52:42.955 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:52:43.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574859 2024-09-18T15:52:43.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574859 2024-09-18T15:52:43.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803788 1-55834574859' 2024-09-18T15:52:43.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:52:43.089 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:52:43.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=137438953475 2024-09-18T15:52:43.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 137438953475 2024-09-18T15:52:43.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803788 1-55834574859 2-137438953475' 2024-09-18T15:52:43.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:52:43.197 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803788 2024-09-18T15:52:43.197 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:52:43.197 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:52:43.198 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803788 2024-09-18T15:52:43.198 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:52:43.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803788 2024-09-18T15:52:43.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803788' 2024-09-18T15:52:43.199 INFO:tasks.workunit.client.0.smithi047.stderr:waiting osd.0 seq 25769803788 2024-09-18T15:52:43.199 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:52:43.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803786 -lt 25769803788 2024-09-18T15:52:43.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:52:44.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:52:44.607 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:52:44.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803789 -lt 25769803788 2024-09-18T15:52:44.985 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:52:44.985 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574859 2024-09-18T15:52:44.985 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:52:44.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:52:44.986 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574859 2024-09-18T15:52:44.987 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:52:44.988 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574859 2024-09-18T15:52:44.988 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574859' 2024-09-18T15:52:44.988 INFO:tasks.workunit.client.0.smithi047.stderr:waiting osd.1 seq 55834574859 2024-09-18T15:52:44.988 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:52:45.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574860 -lt 55834574859 2024-09-18T15:52:45.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:52:45.396 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-137438953475 2024-09-18T15:52:45.396 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:52:45.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:52:45.398 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-137438953475 2024-09-18T15:52:45.398 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:52:45.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=137438953475 2024-09-18T15:52:45.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 137438953475' 2024-09-18T15:52:45.399 INFO:tasks.workunit.client.0.smithi047.stderr:waiting osd.2 seq 137438953475 2024-09-18T15:52:45.399 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:52:45.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 137438953475 -lt 137438953475 2024-09-18T15:52:45.766 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:52:45.766 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:52:45.766 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:52:46.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:52:46.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:52:46.300 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:52:46.300 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:52:46.300 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:52:46.300 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:52:46.300 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:52:46.300 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:52:46.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:52:46.665 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:52:46.666 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:52:46.666 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:52:47.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:52:47.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:52:47.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:52:47.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:441: TEST_rep_read_unfound: objectstore_tool td/osd-rep-recov-eio.sh 1 myobject set-bytes td/osd-rep-recov-eio.sh/CORRUPT 2024-09-18T15:52:47.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:52:47.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:52:47.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:52:47.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:52:47.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-rep-recov-eio.sh 1 myobject set-bytes td/osd-rep-recov-eio.sh/CORRUPT 2024-09-18T15:52:47.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:52:47.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:52:47.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:52:47.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:52:47.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-rep-recov-eio.sh TERM osd.1 2024-09-18T15:52:47.165 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:52:47.165 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:52:47.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:52:47.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:52:47.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:52:47.470 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:52:47.470 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-rep-recov-eio.sh 1 myobject set-bytes td/osd-rep-recov-eio.sh/CORRUPT 2024-09-18T15:52:47.470 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:52:47.470 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:52:47.470 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:52:47.470 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:52:47.470 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-rep-recov-eio.sh/1 2024-09-18T15:52:47.470 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-rep-recov-eio.sh/1 myobject set-bytes td/osd-rep-recov-eio.sh/CORRUPT 2024-09-18T15:52:48.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-rep-recov-eio.sh 1 2024-09-18T15:52:48.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:52:48.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:52:48.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:52:48.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:52:48.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2024-09-18T15:52:48.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-18T15:52:48.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:52:48.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:52:48.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:52:48.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2024-09-18T15:52:48.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2024-09-18T15:52:48.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:52:48.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:52:48.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-18T15:52:48.733 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:52:48.733 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:52:48.734 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:52:48.734 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:52:48.734 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:52:48.734 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:52:48.734 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:52:48.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:52:48.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:52:48.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-18T15:52:48.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-18T15:52:48.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:52:48.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:52:48.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:52:48.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:52:48.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:52:48.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:52:48.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2024-09-18T15:52:48.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:52:48.735 INFO:tasks.workunit.client.0.smithi047.stderr:start osd.1 2024-09-18T15:52:48.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:52:48.737 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-rep-recov-eio.sh/1/whoami 2024-09-18T15:52:48.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:52:48.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:52:48.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:52:48.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:52:48.760 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:48.759+0000 7f54c800a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:48.763 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:48.762+0000 7f54c800a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:48.764 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:48.763+0000 7f54c800a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:52:48.766 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:48.765+0000 7f54c800a440 -1 Falling back to public interface 2024-09-18T15:52:49.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:52:49.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:52:49.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:52:49.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:52:49.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:52:49.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:52:49.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:52:49.112 INFO:tasks.workunit.client.0.smithi047.stderr:0 2024-09-18T15:52:49.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:52:49.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:52:49.162 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:49.161+0000 7f54c800a440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:52:49.169 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:49.168+0000 7f54c800a440 -1 journal do_read_entry(724992): bad header magic 2024-09-18T15:52:49.169 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:49.168+0000 7f54c800a440 -1 journal do_read_entry(724992): bad header magic 2024-09-18T15:52:49.174 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:52:49.173+0000 7f54c800a440 -1 osd.1 33 log_to_monitors true 2024-09-18T15:52:49.569 INFO:tasks.workunit.client.0.smithi047.stderr:osd.1 up in weight 1 up_from 37 up_thru 29 down_at 34 last_clean_interval [13,33) [v2:127.0.0.1:6810/444244,v1:127.0.0.1:6811/444244] [v2:127.0.0.1:6812/444244,v1:127.0.0.1:6813/444244] exists,up dadb0228-872b-47b3-855d-8193f4acf2a5 2024-09-18T15:52:49.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:52:49.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:52:49.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:52:49.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:52:49.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:52:49.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:52:49.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:52:49.570 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:52:49.570 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:52:49.570 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:52:49.570 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:52:49.570 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:52:49.570 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:52:49.660 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:52:49.660 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:52:49.660 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:52:49.660 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:52:49.660 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:52:49.660 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:52:50.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:52:50.071 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:52:50.071 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T15:52:50.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:52:50.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:52:50.072 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:52:50.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803792 2024-09-18T15:52:50.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803792 2024-09-18T15:52:50.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803792' 2024-09-18T15:52:50.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:52:50.181 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:52:50.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=158913789955 2024-09-18T15:52:50.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 158913789955 2024-09-18T15:52:50.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803792 1-158913789955' 2024-09-18T15:52:50.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:52:50.292 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:52:50.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=137438953478 2024-09-18T15:52:50.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 137438953478 2024-09-18T15:52:50.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803792 1-158913789955 2-137438953478' 2024-09-18T15:52:50.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:52:50.401 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803792 2024-09-18T15:52:50.401 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:52:50.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:52:50.402 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803792 2024-09-18T15:52:50.402 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:52:50.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803792 2024-09-18T15:52:50.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803792' 2024-09-18T15:52:50.403 INFO:tasks.workunit.client.0.smithi047.stderr:waiting osd.0 seq 25769803792 2024-09-18T15:52:50.403 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:52:50.844 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803790 -lt 25769803792 2024-09-18T15:52:50.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:52:51.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:52:51.846 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:52:52.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803792 -lt 25769803792 2024-09-18T15:52:52.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:52:52.232 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-158913789955 2024-09-18T15:52:52.233 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:52:52.233 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:52:52.234 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-158913789955 2024-09-18T15:52:52.234 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:52:52.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=158913789955 2024-09-18T15:52:52.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 158913789955' 2024-09-18T15:52:52.235 INFO:tasks.workunit.client.0.smithi047.stderr:waiting osd.1 seq 158913789955 2024-09-18T15:52:52.235 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:52:52.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 158913789955 -lt 158913789955 2024-09-18T15:52:52.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:52:52.608 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-137438953478 2024-09-18T15:52:52.609 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:52:52.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:52:52.610 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-137438953478 2024-09-18T15:52:52.610 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:52:52.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=137438953478 2024-09-18T15:52:52.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 137438953478' 2024-09-18T15:52:52.611 INFO:tasks.workunit.client.0.smithi047.stderr:waiting osd.2 seq 137438953478 2024-09-18T15:52:52.611 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:52:52.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 137438953479 -lt 137438953478 2024-09-18T15:52:52.979 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:52:52.979 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:52:52.979 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:52:53.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:52:53.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:52:53.504 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:52:53.504 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:52:53.504 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:52:53.504 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:52:53.504 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:52:53.504 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:52:53.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:52:53.894 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:52:53.895 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:52:53.895 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:52:54.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:52:54.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:52:54.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:52:54.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:445: TEST_rep_read_unfound: sleep 5 2024-09-18T15:52:54.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:443: TEST_rep_read_unfound: timeout 30 rados -p test-pool get myobject td/osd-rep-recov-eio.sh/tmp 2024-09-18T15:52:59.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:447: TEST_rep_read_unfound: flush_pg_stats 2024-09-18T15:52:59.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:52:59.610 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:53:00.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:53:00.046 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:53:00.046 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T15:53:00.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:53:00.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:53:00.046 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:53:00.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803796 2024-09-18T15:53:00.155 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803796 2024-09-18T15:53:00.155 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803796' 2024-09-18T15:53:00.155 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:53:00.155 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:53:00.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=158913789959 2024-09-18T15:53:00.264 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 158913789959 2024-09-18T15:53:00.264 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803796 1-158913789959' 2024-09-18T15:53:00.264 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:53:00.264 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:53:00.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=137438953482 2024-09-18T15:53:00.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 137438953482 2024-09-18T15:53:00.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803796 1-158913789959 2-137438953482' 2024-09-18T15:53:00.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:53:00.374 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803796 2024-09-18T15:53:00.374 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:53:00.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:53:00.375 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803796 2024-09-18T15:53:00.375 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:53:00.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803796 2024-09-18T15:53:00.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803796' 2024-09-18T15:53:00.376 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 25769803796 2024-09-18T15:53:00.377 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:53:00.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803794 -lt 25769803796 2024-09-18T15:53:00.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:53:01.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:53:01.782 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:53:02.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803794 -lt 25769803796 2024-09-18T15:53:02.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:53:03.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-18T15:53:03.161 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:53:03.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803796 -lt 25769803796 2024-09-18T15:53:03.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:53:03.539 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-158913789959 2024-09-18T15:53:03.539 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:53:03.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:53:03.540 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-158913789959 2024-09-18T15:53:03.540 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:53:03.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=158913789959 2024-09-18T15:53:03.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 158913789959' 2024-09-18T15:53:03.541 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 158913789959 2024-09-18T15:53:03.542 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:53:03.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 158913789959 -lt 158913789959 2024-09-18T15:53:03.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:53:03.935 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-137438953482 2024-09-18T15:53:03.935 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:53:03.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:53:03.936 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-137438953482 2024-09-18T15:53:03.937 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:53:03.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=137438953482 2024-09-18T15:53:03.938 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 137438953482' 2024-09-18T15:53:03.938 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 137438953482 2024-09-18T15:53:03.938 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:53:04.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 137438953483 -lt 137438953482 2024-09-18T15:53:04.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:448: TEST_rep_read_unfound: ceph --format=json pg dump pgs 2024-09-18T15:53:04.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:448: TEST_rep_read_unfound: jq . 2024-09-18T15:53:04.663 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T15:53:04.674 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:53:04.674 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_ready": true, 2024-09-18T15:53:04.674 INFO:tasks.workunit.client.0.smithi047.stdout: "pg_stats": [ 2024-09-18T15:53:04.674 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:53:04.674 INFO:tasks.workunit.client.0.smithi047.stdout: "pgid": "2.0", 2024-09-18T15:53:04.674 INFO:tasks.workunit.client.0.smithi047.stdout: "version": "28'1", 2024-09-18T15:53:04.674 INFO:tasks.workunit.client.0.smithi047.stdout: "reported_seq": 58, 2024-09-18T15:53:04.674 INFO:tasks.workunit.client.0.smithi047.stdout: "reported_epoch": 38, 2024-09-18T15:53:04.674 INFO:tasks.workunit.client.0.smithi047.stdout: "state": "active+recovery_unfound+degraded", 2024-09-18T15:53:04.674 INFO:tasks.workunit.client.0.smithi047.stdout: "last_fresh": "2024-09-18T15:52:54.658320+0000", 2024-09-18T15:53:04.675 INFO:tasks.workunit.client.0.smithi047.stdout: "last_change": "2024-09-18T15:52:54.859587+0000", 2024-09-18T15:53:04.675 INFO:tasks.workunit.client.0.smithi047.stdout: "last_active": "2024-09-18T15:52:54.658320+0000", 2024-09-18T15:53:04.675 INFO:tasks.workunit.client.0.smithi047.stdout: "last_peered": "2024-09-18T15:52:54.658320+0000", 2024-09-18T15:53:04.675 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean": "2024-09-18T15:52:50.853118+0000", 2024-09-18T15:53:04.675 INFO:tasks.workunit.client.0.smithi047.stdout: "last_became_active": "2024-09-18T15:52:50.744207+0000", 2024-09-18T15:53:04.675 INFO:tasks.workunit.client.0.smithi047.stdout: "last_became_peered": "2024-09-18T15:52:50.744207+0000", 2024-09-18T15:53:04.675 INFO:tasks.workunit.client.0.smithi047.stdout: "last_unstale": "2024-09-18T15:52:54.658320+0000", 2024-09-18T15:53:04.675 INFO:tasks.workunit.client.0.smithi047.stdout: "last_undegraded": "2024-09-18T15:52:54.657931+0000", 2024-09-18T15:53:04.675 INFO:tasks.workunit.client.0.smithi047.stdout: "last_fullsized": "2024-09-18T15:52:54.658320+0000", 2024-09-18T15:53:04.675 INFO:tasks.workunit.client.0.smithi047.stdout: "mapping_epoch": 37, 2024-09-18T15:53:04.675 INFO:tasks.workunit.client.0.smithi047.stdout: "log_start": "0'0", 2024-09-18T15:53:04.675 INFO:tasks.workunit.client.0.smithi047.stdout: "ondisk_log_start": "0'0", 2024-09-18T15:53:04.675 INFO:tasks.workunit.client.0.smithi047.stdout: "created": 23, 2024-09-18T15:53:04.675 INFO:tasks.workunit.client.0.smithi047.stdout: "last_epoch_clean": 38, 2024-09-18T15:53:04.676 INFO:tasks.workunit.client.0.smithi047.stdout: "parent": "0.0", 2024-09-18T15:53:04.676 INFO:tasks.workunit.client.0.smithi047.stdout: "parent_split_bits": 0, 2024-09-18T15:53:04.676 INFO:tasks.workunit.client.0.smithi047.stdout: "last_scrub": "0'0", 2024-09-18T15:53:04.676 INFO:tasks.workunit.client.0.smithi047.stdout: "last_scrub_stamp": "2024-09-18T15:52:29.509511+0000", 2024-09-18T15:53:04.676 INFO:tasks.workunit.client.0.smithi047.stdout: "last_deep_scrub": "0'0", 2024-09-18T15:53:04.676 INFO:tasks.workunit.client.0.smithi047.stdout: "last_deep_scrub_stamp": "2024-09-18T15:52:29.509511+0000", 2024-09-18T15:53:04.676 INFO:tasks.workunit.client.0.smithi047.stdout: "last_clean_scrub_stamp": "2024-09-18T15:52:29.509511+0000", 2024-09-18T15:53:04.676 INFO:tasks.workunit.client.0.smithi047.stdout: "objects_scrubbed": 0, 2024-09-18T15:53:04.676 INFO:tasks.workunit.client.0.smithi047.stdout: "log_size": 1, 2024-09-18T15:53:04.676 INFO:tasks.workunit.client.0.smithi047.stdout: "log_dups_size": 0, 2024-09-18T15:53:04.676 INFO:tasks.workunit.client.0.smithi047.stdout: "ondisk_log_size": 1, 2024-09-18T15:53:04.676 INFO:tasks.workunit.client.0.smithi047.stdout: "stats_invalid": false, 2024-09-18T15:53:04.676 INFO:tasks.workunit.client.0.smithi047.stdout: "dirty_stats_invalid": false, 2024-09-18T15:53:04.676 INFO:tasks.workunit.client.0.smithi047.stdout: "omap_stats_invalid": false, 2024-09-18T15:53:04.677 INFO:tasks.workunit.client.0.smithi047.stdout: "hitset_stats_invalid": false, 2024-09-18T15:53:04.677 INFO:tasks.workunit.client.0.smithi047.stdout: "hitset_bytes_stats_invalid": false, 2024-09-18T15:53:04.677 INFO:tasks.workunit.client.0.smithi047.stdout: "pin_stats_invalid": false, 2024-09-18T15:53:04.677 INFO:tasks.workunit.client.0.smithi047.stdout: "manifest_stats_invalid": false, 2024-09-18T15:53:04.677 INFO:tasks.workunit.client.0.smithi047.stdout: "snaptrimq_len": 0, 2024-09-18T15:53:04.677 INFO:tasks.workunit.client.0.smithi047.stdout: "last_scrub_duration": 0, 2024-09-18T15:53:04.677 INFO:tasks.workunit.client.0.smithi047.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-19T16:02:29.083205+0000", 2024-09-18T15:53:04.677 INFO:tasks.workunit.client.0.smithi047.stdout: "scrub_duration": 0, 2024-09-18T15:53:04.677 INFO:tasks.workunit.client.0.smithi047.stdout: "objects_trimmed": 0, 2024-09-18T15:53:04.677 INFO:tasks.workunit.client.0.smithi047.stdout: "snaptrim_duration": 0, 2024-09-18T15:53:04.677 INFO:tasks.workunit.client.0.smithi047.stdout: "stat_sum": { 2024-09-18T15:53:04.677 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes": 8192, 2024-09-18T15:53:04.677 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects": 1, 2024-09-18T15:53:04.677 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_clones": 0, 2024-09-18T15:53:04.678 INFO:tasks.workunit.client.0.smithi047.stdout: "num_object_copies": 2, 2024-09-18T15:53:04.678 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing_on_primary": 1, 2024-09-18T15:53:04.678 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_missing": 0, 2024-09-18T15:53:04.678 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_degraded": 2, 2024-09-18T15:53:04.678 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_misplaced": 0, 2024-09-18T15:53:04.678 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_unfound": 1, 2024-09-18T15:53:04.678 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_dirty": 1, 2024-09-18T15:53:04.678 INFO:tasks.workunit.client.0.smithi047.stdout: "num_whiteouts": 0, 2024-09-18T15:53:04.678 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read": 0, 2024-09-18T15:53:04.678 INFO:tasks.workunit.client.0.smithi047.stdout: "num_read_kb": 0, 2024-09-18T15:53:04.678 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write": 1, 2024-09-18T15:53:04.678 INFO:tasks.workunit.client.0.smithi047.stdout: "num_write_kb": 8, 2024-09-18T15:53:04.678 INFO:tasks.workunit.client.0.smithi047.stdout: "num_scrub_errors": 0, 2024-09-18T15:53:04.678 INFO:tasks.workunit.client.0.smithi047.stdout: "num_shallow_scrub_errors": 0, 2024-09-18T15:53:04.679 INFO:tasks.workunit.client.0.smithi047.stdout: "num_deep_scrub_errors": 0, 2024-09-18T15:53:04.679 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_recovered": 0, 2024-09-18T15:53:04.679 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_recovered": 0, 2024-09-18T15:53:04.679 INFO:tasks.workunit.client.0.smithi047.stdout: "num_keys_recovered": 0, 2024-09-18T15:53:04.679 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_omap": 0, 2024-09-18T15:53:04.679 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_hit_set_archive": 0, 2024-09-18T15:53:04.679 INFO:tasks.workunit.client.0.smithi047.stdout: "num_bytes_hit_set_archive": 0, 2024-09-18T15:53:04.679 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush": 0, 2024-09-18T15:53:04.679 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_kb": 0, 2024-09-18T15:53:04.679 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict": 0, 2024-09-18T15:53:04.679 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_kb": 0, 2024-09-18T15:53:04.679 INFO:tasks.workunit.client.0.smithi047.stdout: "num_promote": 0, 2024-09-18T15:53:04.679 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_high": 0, 2024-09-18T15:53:04.680 INFO:tasks.workunit.client.0.smithi047.stdout: "num_flush_mode_low": 0, 2024-09-18T15:53:04.680 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_some": 0, 2024-09-18T15:53:04.680 INFO:tasks.workunit.client.0.smithi047.stdout: "num_evict_mode_full": 0, 2024-09-18T15:53:04.680 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_pinned": 0, 2024-09-18T15:53:04.680 INFO:tasks.workunit.client.0.smithi047.stdout: "num_legacy_snapsets": 0, 2024-09-18T15:53:04.680 INFO:tasks.workunit.client.0.smithi047.stdout: "num_large_omap_objects": 0, 2024-09-18T15:53:04.680 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_manifest": 0, 2024-09-18T15:53:04.680 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_bytes": 0, 2024-09-18T15:53:04.680 INFO:tasks.workunit.client.0.smithi047.stdout: "num_omap_keys": 0, 2024-09-18T15:53:04.680 INFO:tasks.workunit.client.0.smithi047.stdout: "num_objects_repaired": 0 2024-09-18T15:53:04.680 INFO:tasks.workunit.client.0.smithi047.stdout: }, 2024-09-18T15:53:04.680 INFO:tasks.workunit.client.0.smithi047.stdout: "up": [ 2024-09-18T15:53:04.680 INFO:tasks.workunit.client.0.smithi047.stdout: 2, 2024-09-18T15:53:04.681 INFO:tasks.workunit.client.0.smithi047.stdout: 1 2024-09-18T15:53:04.681 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:53:04.681 INFO:tasks.workunit.client.0.smithi047.stdout: "acting": [ 2024-09-18T15:53:04.681 INFO:tasks.workunit.client.0.smithi047.stdout: 2, 2024-09-18T15:53:04.681 INFO:tasks.workunit.client.0.smithi047.stdout: 1 2024-09-18T15:53:04.681 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:53:04.681 INFO:tasks.workunit.client.0.smithi047.stdout: "avail_no_missing": [], 2024-09-18T15:53:04.681 INFO:tasks.workunit.client.0.smithi047.stdout: "object_location_counts": [ 2024-09-18T15:53:04.681 INFO:tasks.workunit.client.0.smithi047.stdout: { 2024-09-18T15:53:04.681 INFO:tasks.workunit.client.0.smithi047.stdout: "shards": "", 2024-09-18T15:53:04.681 INFO:tasks.workunit.client.0.smithi047.stdout: "objects": 1 2024-09-18T15:53:04.681 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:53:04.681 INFO:tasks.workunit.client.0.smithi047.stdout: ], 2024-09-18T15:53:04.681 INFO:tasks.workunit.client.0.smithi047.stdout: "blocked_by": [], 2024-09-18T15:53:04.682 INFO:tasks.workunit.client.0.smithi047.stdout: "up_primary": 2, 2024-09-18T15:53:04.682 INFO:tasks.workunit.client.0.smithi047.stdout: "acting_primary": 2, 2024-09-18T15:53:04.682 INFO:tasks.workunit.client.0.smithi047.stdout: "purged_snaps": [] 2024-09-18T15:53:04.682 INFO:tasks.workunit.client.0.smithi047.stdout: } 2024-09-18T15:53:04.682 INFO:tasks.workunit.client.0.smithi047.stdout: ] 2024-09-18T15:53:04.682 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:53:04.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:450: TEST_rep_read_unfound: ceph --format=json pg dump pgs 2024-09-18T15:53:04.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:450: TEST_rep_read_unfound: jq '.pg_stats | .[0].state' 2024-09-18T15:53:04.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:450: TEST_rep_read_unfound: grep -q recovery_unfound 2024-09-18T15:53:05.026 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T15:53:05.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:456: TEST_rep_read_unfound: objectstore_tool td/osd-rep-recov-eio.sh 1 myobject set-bytes td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:53:05.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:53:05.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T15:53:05.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T15:53:05.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T15:53:05.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-rep-recov-eio.sh 1 myobject set-bytes td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:53:05.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:53:05.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T15:53:05.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T15:53:05.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T15:53:05.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-rep-recov-eio.sh TERM osd.1 2024-09-18T15:53:05.039 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:53:05.039 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:53:05.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:53:05.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:53:05.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:53:05.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:53:05.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-rep-recov-eio.sh 1 myobject set-bytes td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:53:05.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:53:05.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T15:53:05.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T15:53:05.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T15:53:05.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-rep-recov-eio.sh/1 2024-09-18T15:53:05.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-rep-recov-eio.sh/1 myobject set-bytes td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:53:06.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-rep-recov-eio.sh 1 2024-09-18T15:53:06.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:53:06.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:53:06.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T15:53:06.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:53:06.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2024-09-18T15:53:06.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-18T15:53:06.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:53:06.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:53:06.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:53:06.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2024-09-18T15:53:06.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2024-09-18T15:53:06.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:53:06.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:53:06.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-18T15:53:06.482 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:53:06.482 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:53:06.482 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:53:06.482 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:53:06.482 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:53:06.482 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:53:06.482 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:53:06.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:53:06.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:53:06.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-18T15:53:06.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-18T15:53:06.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:53:06.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:53:06.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:53:06.483 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:53:06.483 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:53:06.483 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:53:06.483 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2024-09-18T15:53:06.483 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T15:53:06.483 INFO:tasks.workunit.client.0.smithi047.stderr:start osd.1 2024-09-18T15:53:06.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:53:06.484 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-rep-recov-eio.sh/1/whoami 2024-09-18T15:53:06.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T15:53:06.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:53:06.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:53:06.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:53:06.509 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:53:06.508+0000 7ff62748d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:53:06.511 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:53:06.510+0000 7ff62748d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:53:06.513 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:53:06.512+0000 7ff62748d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:53:06.514 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:53:06.513+0000 7ff62748d440 -1 Falling back to public interface 2024-09-18T15:53:06.903 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:53:06.902+0000 7ff62748d440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-18T15:53:06.903 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:53:06.902+0000 7ff62748d440 -1 journal do_read_entry(786432): bad header magic 2024-09-18T15:53:06.903 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:53:06.902+0000 7ff62748d440 -1 journal do_read_entry(786432): bad header magic 2024-09-18T15:53:06.907 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:53:06.906+0000 7ff62748d440 -1 osd.1 38 log_to_monitors true 2024-09-18T15:53:06.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T15:53:06.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:53:06.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:53:06.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:53:06.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:53:06.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:06.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:53:06.997 INFO:tasks.workunit.client.0.smithi047.stderr:0 2024-09-18T15:53:06.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:06.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:53:07.372 INFO:tasks.workunit.client.0.smithi047.stderr:osd.1 up in weight 1 up_from 42 up_thru 29 down_at 39 last_clean_interval [37,38) [v2:127.0.0.1:6810/445071,v1:127.0.0.1:6811/445071] [v2:127.0.0.1:6812/445071,v1:127.0.0.1:6813/445071] exists,up dadb0228-872b-47b3-855d-8193f4acf2a5 2024-09-18T15:53:07.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:53:07.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:53:07.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:53:07.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T15:53:07.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:53:07.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:53:07.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:53:07.373 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:53:07.373 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:53:07.373 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:53:07.373 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:53:07.373 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:53:07.373 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:53:07.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:53:07.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:53:07.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:53:07.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:53:07.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:53:07.464 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:53:07.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:53:07.914 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:53:07.914 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T15:53:07.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:53:07.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:53:07.914 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:53:08.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803799 2024-09-18T15:53:08.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803799 2024-09-18T15:53:08.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803799' 2024-09-18T15:53:08.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:53:08.053 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:53:08.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=180388626435 2024-09-18T15:53:08.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 180388626435 2024-09-18T15:53:08.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803799 1-180388626435' 2024-09-18T15:53:08.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:53:08.162 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:53:08.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=137438953486 2024-09-18T15:53:08.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 137438953486 2024-09-18T15:53:08.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803799 1-180388626435 2-137438953486' 2024-09-18T15:53:08.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:53:08.271 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803799 2024-09-18T15:53:08.271 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:53:08.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:53:08.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803799 2024-09-18T15:53:08.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:53:08.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803799 2024-09-18T15:53:08.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803799' 2024-09-18T15:53:08.276 INFO:tasks.workunit.client.0.smithi047.stderr:waiting osd.0 seq 25769803799 2024-09-18T15:53:08.276 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:53:08.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803799 -lt 25769803799 2024-09-18T15:53:08.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:53:08.656 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-180388626435 2024-09-18T15:53:08.656 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:53:08.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:53:08.657 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-180388626435 2024-09-18T15:53:08.657 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:53:08.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=180388626435 2024-09-18T15:53:08.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 180388626435' 2024-09-18T15:53:08.659 INFO:tasks.workunit.client.0.smithi047.stderr:waiting osd.1 seq 180388626435 2024-09-18T15:53:08.659 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:53:09.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 180388626435 2024-09-18T15:53:09.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:53:10.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:53:10.085 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:53:10.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 180388626435 -lt 180388626435 2024-09-18T15:53:10.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:53:10.464 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-137438953486 2024-09-18T15:53:10.464 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:53:10.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:53:10.465 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-137438953486 2024-09-18T15:53:10.465 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:53:10.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=137438953486 2024-09-18T15:53:10.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 137438953486' 2024-09-18T15:53:10.466 INFO:tasks.workunit.client.0.smithi047.stderr:waiting osd.2 seq 137438953486 2024-09-18T15:53:10.466 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:53:10.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 137438953486 -lt 137438953486 2024-09-18T15:53:10.870 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:53:10.870 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:53:10.870 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:53:11.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:53:11.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:53:11.379 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:53:11.379 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:53:11.379 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:53:11.379 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:53:11.380 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:53:11.380 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:53:11.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:53:11.738 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:53:11.738 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:53:11.738 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:53:12.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:53:12.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:53:12.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:53:12.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:458: TEST_rep_read_unfound: wait 2024-09-18T15:53:12.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:460: TEST_rep_read_unfound: cmp td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/tmp 2024-09-18T15:53:12.240 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:43: run: teardown td/osd-rep-recov-eio.sh 2024-09-18T15:53:12.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:53:12.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:53:12.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-09-18T15:53:12.241 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:53:12.241 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:53:12.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:53:12.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:53:12.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:53:12.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:53:12.347 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:53:12.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:53:12.349 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:53:12.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:53:12.350 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:53:12.350 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:53:12.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:53:12.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:53:12.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:53:12.351 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:53:12.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:53:12.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:53:12.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-09-18T15:53:12.365 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:53:12.365 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:53:12.365 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:53:12.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.397939 2024-09-18T15:53:12.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:53:12.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:53:12.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:35: run: for func in $funcs 2024-09-18T15:53:12.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:36: run: setup td/osd-rep-recov-eio.sh 2024-09-18T15:53:12.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:53:12.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-rep-recov-eio.sh 2024-09-18T15:53:12.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:53:12.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:53:12.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-09-18T15:53:12.368 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:53:12.368 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:53:12.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:53:12.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:53:12.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:53:12.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:53:12.369 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:53:12.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:53:12.370 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:53:12.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:53:12.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:53:12.371 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:53:12.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:53:12.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:53:12.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:53:12.372 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:53:12.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:53:12.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:53:12.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-09-18T15:53:12.375 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:53:12.375 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:53:12.375 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:53:12.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.397939 2024-09-18T15:53:12.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:53:12.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:53:12.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-rep-recov-eio.sh 2024-09-18T15:53:12.377 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T15:53:12.377 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:53:12.378 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:53:12.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.397939 2024-09-18T15:53:12.380 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T15:53:12.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T15:53:12.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T15:53:12.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-rep-recov-eio.sh 1' TERM HUP INT 2024-09-18T15:53:12.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:38: run: run_mon td/osd-rep-recov-eio.sh a --mon_osd_warn_num_repaired=10 2024-09-18T15:53:12.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:53:12.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T15:53:12.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T15:53:12.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T15:53:12.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-rep-recov-eio.sh/a 2024-09-18T15:53:12.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-rep-recov-eio.sh/a --run-dir=td/osd-rep-recov-eio.sh --mon_osd_warn_num_repaired=10 2024-09-18T15:53:12.770 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T15:53:12.770 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:53:12.770 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:53:12.770 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:53:12.770 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:53:12.770 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:53:12.770 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:53:12.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-rep-recov-eio.sh/a '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --mon-cluster-log-file=td/osd-rep-recov-eio.sh/log --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_osd_warn_num_repaired=10 2024-09-18T15:53:13.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T15:53:13.047 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T15:53:13.047 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:53:13.047 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:53:13.047 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T15:53:13.047 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T15:53:13.048 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:53:13.048 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:53:13.048 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:53:13.049 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:53:13.049 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:53:13.049 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:53:13.049 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-mon.a.asok 2024-09-18T15:53:13.049 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:53:13.049 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.397939/ceph-mon.a.asok config get fsid 2024-09-18T15:53:13.129 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T15:53:13.129 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:53:13.129 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:53:13.130 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T15:53:13.130 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T15:53:13.130 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:53:13.130 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:53:13.130 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:53:13.130 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:53:13.130 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:53:13.130 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:53:13.130 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-mon.a.asok 2024-09-18T15:53:13.130 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:53:13.130 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.397939/ceph-mon.a.asok config get mon_host 2024-09-18T15:53:13.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:39: run: run_mgr td/osd-rep-recov-eio.sh x 2024-09-18T15:53:13.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:53:13.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T15:53:13.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T15:53:13.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T15:53:13.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-rep-recov-eio.sh/x 2024-09-18T15:53:13.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T15:53:14.060 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T15:53:14.060 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:53:14.060 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:53:14.060 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:53:14.061 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:53:14.061 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:53:14.061 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:53:14.061 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:53:14.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-rep-recov-eio.sh/x '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:53:14.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:40: run: ceph osd pool create foo 8 2024-09-18T15:53:14.383 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'foo' already exists 2024-09-18T15:53:14.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:42: run: TEST_rep_recovery_unfound td/osd-rep-recov-eio.sh 2024-09-18T15:53:14.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:343: TEST_rep_recovery_unfound: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:53:14.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:344: TEST_rep_recovery_unfound: local objname=myobject 2024-09-18T15:53:14.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:345: TEST_rep_recovery_unfound: local lastobj=100 2024-09-18T15:53:14.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:347: TEST_rep_recovery_unfound: local testobj=obj75 2024-09-18T15:53:14.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:349: TEST_rep_recovery_unfound: setup_osds 3 2024-09-18T15:53:14.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:48: setup_osds: local count=3 2024-09-18T15:53:14.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:49: setup_osds: shift 2024-09-18T15:53:14.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:50: setup_osds: local type= 2024-09-18T15:53:14.394 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: expr 3 - 1 2024-09-18T15:53:14.394 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: seq 0 2 2024-09-18T15:53:14.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T15:53:14.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 0 2024-09-18T15:53:14.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:53:14.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:53:14.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T15:53:14.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:53:14.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-09-18T15:53:14.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-18T15:53:14.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:53:14.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:53:14.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:53:14.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-09-18T15:53:14.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-09-18T15:53:14.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:53:14.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:53:14.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-18T15:53:14.400 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:53:14.400 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:53:14.400 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:53:14.400 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:53:14.400 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:53:14.400 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:53:14.400 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:53:14.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:53:14.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:53:14.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:53:14.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:53:14.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-18T15:53:14.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-18T15:53:14.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:53:14.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:53:14.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:53:14.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:53:14.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:53:14.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:53:14.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-09-18T15:53:14.405 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:53:14.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6c09a867-7649-4abc-8f06-2ffadd99ea85 2024-09-18T15:53:14.405 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 6c09a867-7649-4abc-8f06-2ffadd99ea85 2024-09-18T15:53:14.406 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6c09a867-7649-4abc-8f06-2ffadd99ea85' 2024-09-18T15:53:14.406 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:53:14.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBq9+pmQ82rGBAArrZlS7T5eapw3GzBSKKRyg== 2024-09-18T15:53:14.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBq9+pmQ82rGBAArrZlS7T5eapw3GzBSKKRyg=="}' 2024-09-18T15:53:14.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6c09a867-7649-4abc-8f06-2ffadd99ea85 -i td/osd-rep-recov-eio.sh/0/new.json 2024-09-18T15:53:14.704 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:53:14.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/0/new.json 2024-09-18T15:53:14.714 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBq9+pmQ82rGBAArrZlS7T5eapw3GzBSKKRyg== --osd-uuid 6c09a867-7649-4abc-8f06-2ffadd99ea85 2024-09-18T15:53:14.741 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:53:14.740+0000 7fbc87a0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:53:14.748 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:53:14.748+0000 7fbc87a0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:53:14.749 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:53:14.749+0000 7fbc87a0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:53:14.750 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:53:14.749+0000 7fbc87a0b440 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-18T15:53:14.750 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:53:14.749+0000 7fbc87a0b440 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-18T15:53:14.750 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:53:14.749+0000 7fbc87a0b440 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-18T15:53:14.788 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:53:14.787+0000 7fbc87a0b440 -1 bluestore(td/osd-rep-recov-eio.sh/0) _read_fsid unparsable uuid 2024-09-18T15:53:20.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/0/keyring 2024-09-18T15:53:20.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:53:20.078 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T15:53:20.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T15:53:20.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:53:20.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T15:53:20.596 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T15:53:20.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:53:20.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:53:20.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:53:20.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:53:20.621 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:53:20.620+0000 7f7cd7323440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:53:20.638 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:53:20.637+0000 7f7cd7323440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:53:20.639 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:53:20.638+0000 7f7cd7323440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:53:21.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T15:53:21.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:53:21.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:53:21.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:53:21.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:53:21.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:21.043 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:53:21.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:53:21.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:21.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:53:21.176 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:53:21.175+0000 7f7cd7323440 -1 Falling back to public interface 2024-09-18T15:53:21.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:53:22.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:53:22.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:22.454 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:53:22.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:53:22.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:22.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:53:22.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:53:23.667 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:53:23.667+0000 7f7cd7323440 -1 osd.0 0 log_to_monitors true 2024-09-18T15:53:23.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:53:23.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:23.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:53:23.880 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:53:23.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:23.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:53:24.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:53:25.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:53:25.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:25.410 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:53:25.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:53:25.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:25.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:53:25.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:53:26.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:53:26.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:26.798 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:53:26.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:53:26.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:26.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:53:27.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:53:28.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:53:28.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:28.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:53:28.188 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:53:28.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:28.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:53:28.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:53:29.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:53:29.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:29.576 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:53:29.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:53:29.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:29.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:53:29.956 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:53:30.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:53:30.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:30.958 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:53:30.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:53:30.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:30.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:53:31.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:53:32.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:53:32.487 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:53:32.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:32.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:53:32.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:32.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:53:32.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:53:33.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:53:33.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:33.869 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:53:33.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:53:33.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:33.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:53:34.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:53:35.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:53:35.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:35.343 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:53:35.343 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:53:35.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:35.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:53:35.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:53:36.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:53:36.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:36.920 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:53:36.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:53:36.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:36.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:53:37.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:53:38.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:53:38.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:38.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:53:38.381 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:53:38.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:38.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:53:39.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:53:40.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:53:40.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:40.140 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:53:40.140 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:53:40.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:40.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:53:40.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:53:41.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:53:41.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:41.893 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:53:41.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:53:41.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:41.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:53:42.373 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:53:43.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:53:43.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:43.375 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:53:43.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:53:43.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:43.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:53:44.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:53:45.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:53:45.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:45.124 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:53:45.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:53:45.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:45.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:53:45.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:53:46.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:53:46.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:46.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:53:46.909 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T15:53:46.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:46.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:53:47.381 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/446676,v1:127.0.0.1:6803/446676] [v2:127.0.0.1:6804/446676,v1:127.0.0.1:6805/446676] exists,up 6c09a867-7649-4abc-8f06-2ffadd99ea85 2024-09-18T15:53:47.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:53:47.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:53:47.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:53:47.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T15:53:47.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 1 2024-09-18T15:53:47.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:53:47.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:53:47.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T15:53:47.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:53:47.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2024-09-18T15:53:47.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-18T15:53:47.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:53:47.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:53:47.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:53:47.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2024-09-18T15:53:47.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2024-09-18T15:53:47.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:53:47.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:53:47.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-18T15:53:47.386 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:53:47.386 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:53:47.386 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:53:47.386 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:53:47.387 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:53:47.387 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:53:47.387 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:53:47.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:53:47.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:53:47.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:53:47.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:53:47.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-18T15:53:47.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-18T15:53:47.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:53:47.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:53:47.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:53:47.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:53:47.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:53:47.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:53:47.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2024-09-18T15:53:47.388 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:53:47.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ce96062c-d1ba-4d35-9a17-b8997ec8ac69 2024-09-18T15:53:47.388 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 ce96062c-d1ba-4d35-9a17-b8997ec8ac69 2024-09-18T15:53:47.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ce96062c-d1ba-4d35-9a17-b8997ec8ac69' 2024-09-18T15:53:47.388 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:53:47.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCL9+pm+jPgFxAAhluchy0TcC7t5ln07+pOCw== 2024-09-18T15:53:47.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCL9+pm+jPgFxAAhluchy0TcC7t5ln07+pOCw=="}' 2024-09-18T15:53:47.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ce96062c-d1ba-4d35-9a17-b8997ec8ac69 -i td/osd-rep-recov-eio.sh/1/new.json 2024-09-18T15:53:47.924 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:53:47.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/1/new.json 2024-09-18T15:53:47.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCL9+pm+jPgFxAAhluchy0TcC7t5ln07+pOCw== --osd-uuid ce96062c-d1ba-4d35-9a17-b8997ec8ac69 2024-09-18T15:53:47.964 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:53:47.961+0000 7f5d98e09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:53:47.982 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:53:47.981+0000 7f5d98e09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:53:47.983 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:53:47.982+0000 7f5d98e09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:53:47.984 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:53:47.983+0000 7f5d98e09440 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-18T15:53:47.984 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:53:47.983+0000 7f5d98e09440 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-18T15:53:47.984 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:53:47.983+0000 7f5d98e09440 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-18T15:53:48.066 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:53:48.065+0000 7f5d98e09440 -1 bluestore(td/osd-rep-recov-eio.sh/1) _read_fsid unparsable uuid 2024-09-18T15:53:53.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/1/keyring 2024-09-18T15:53:53.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:53:53.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T15:53:53.855 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T15:53:53.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:53:54.400 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T15:53:54.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T15:53:54.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:53:54.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:53:54.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:53:54.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:53:54.425 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:53:54.424+0000 7f99e2c08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:53:54.427 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:53:54.426+0000 7f99e2c08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:53:54.428 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:53:54.427+0000 7f99e2c08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:53:54.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T15:53:54.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:53:54.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:53:54.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:53:54.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:53:54.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:54.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:53:54.772 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:53:54.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:54.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:53:54.944 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:53:54.943+0000 7f99e2c08440 -1 Falling back to public interface 2024-09-18T15:53:55.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:53:56.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:53:56.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:56.145 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:53:56.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:53:56.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:56.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:53:56.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:53:57.418 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:53:57.417+0000 7f99e2c08440 -1 osd.1 0 log_to_monitors true 2024-09-18T15:53:57.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:53:57.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:57.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:53:57.581 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:53:57.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:57.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:53:58.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:53:59.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:53:59.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:53:59.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:53:59.070 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:53:59.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:53:59.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:53:59.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:00.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:00.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:00.474 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:54:00.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:54:00.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:00.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:54:00.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:01.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:01.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:01.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:54:01.889 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:54:01.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:01.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:54:02.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:03.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:03.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:03.309 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:54:03.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:54:03.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:03.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:54:03.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:04.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:04.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:04.725 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:54:04.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:54:04.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:04.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:54:05.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:06.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:06.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:06.182 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:54:06.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:54:06.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:06.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:54:06.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:07.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:07.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:07.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:54:07.664 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:54:07.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:07.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:54:08.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:09.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:09.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:09.045 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:54:09.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:54:09.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:09.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:54:09.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:10.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:10.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:10.657 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:54:10.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:54:10.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:10.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:54:11.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:12.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:12.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:12.449 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:54:12.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:54:12.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:12.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:54:13.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:14.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:14.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:14.439 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:54:14.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:54:14.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:14.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:54:15.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:16.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:16.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:16.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:54:16.133 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:54:16.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:16.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:54:16.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:17.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:17.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:17.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:54:17.836 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:54:17.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:17.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:54:18.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:19.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:19.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:19.776 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:54:19.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:54:19.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:19.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:54:20.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:21.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:21.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:21.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:54:21.195 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T15:54:21.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:21.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:54:21.569 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 14 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/448432,v1:127.0.0.1:6811/448432] [v2:127.0.0.1:6812/448432,v1:127.0.0.1:6813/448432] exists,up ce96062c-d1ba-4d35-9a17-b8997ec8ac69 2024-09-18T15:54:21.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:54:21.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:54:21.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:54:21.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T15:54:21.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 2 2024-09-18T15:54:21.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:54:21.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:54:21.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T15:54:21.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:54:21.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/2 2024-09-18T15:54:21.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-18T15:54:21.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:54:21.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:54:21.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:54:21.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2024-09-18T15:54:21.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2024-09-18T15:54:21.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:54:21.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:54:21.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-18T15:54:21.574 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:54:21.574 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:54:21.574 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:54:21.574 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:54:21.574 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:54:21.574 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:54:21.574 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:54:21.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:54:21.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:54:21.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:54:21.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:54:21.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-18T15:54:21.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-18T15:54:21.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:54:21.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:54:21.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:54:21.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:54:21.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:54:21.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:54:21.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/2 2024-09-18T15:54:21.576 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:54:21.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b0f2b948-f6d4-4001-8d8a-7fdab0629567 2024-09-18T15:54:21.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 b0f2b948-f6d4-4001-8d8a-7fdab0629567' 2024-09-18T15:54:21.576 INFO:tasks.workunit.client.0.smithi047.stdout:add osd2 b0f2b948-f6d4-4001-8d8a-7fdab0629567 2024-09-18T15:54:21.576 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:54:21.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCt9+pm4acIIxAASBBzNVnQJ3YADwp1ZpK+SQ== 2024-09-18T15:54:21.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCt9+pm4acIIxAASBBzNVnQJ3YADwp1ZpK+SQ=="}' 2024-09-18T15:54:21.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b0f2b948-f6d4-4001-8d8a-7fdab0629567 -i td/osd-rep-recov-eio.sh/2/new.json 2024-09-18T15:54:22.095 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:54:22.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/2/new.json 2024-09-18T15:54:22.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCt9+pm4acIIxAASBBzNVnQJ3YADwp1ZpK+SQ== --osd-uuid b0f2b948-f6d4-4001-8d8a-7fdab0629567 2024-09-18T15:54:22.135 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:54:22.134+0000 7f2b85929440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:54:22.136 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:54:22.136+0000 7f2b85929440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:54:22.138 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:54:22.137+0000 7f2b85929440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:54:22.138 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:54:22.137+0000 7f2b85929440 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-18T15:54:22.138 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:54:22.137+0000 7f2b85929440 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-18T15:54:22.138 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:54:22.137+0000 7f2b85929440 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-18T15:54:22.186 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:54:22.185+0000 7f2b85929440 -1 bluestore(td/osd-rep-recov-eio.sh/2) _read_fsid unparsable uuid 2024-09-18T15:54:28.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/2/keyring 2024-09-18T15:54:28.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:54:28.003 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd2 key to auth repository 2024-09-18T15:54:28.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T15:54:28.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:54:28.519 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T15:54:28.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T15:54:28.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:54:28.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:54:28.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:54:28.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:54:28.545 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:54:28.543+0000 7f33fab23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:54:28.568 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:54:28.567+0000 7f33fab23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:54:28.569 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:54:28.568+0000 7f33fab23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:54:28.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T15:54:28.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:54:28.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:54:28.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:54:28.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:54:28.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:28.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:54:28.929 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:54:28.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:28.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:54:29.108 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:54:29.107+0000 7f33fab23440 -1 Falling back to public interface 2024-09-18T15:54:29.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:30.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:30.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:30.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:54:30.296 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:54:30.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:30.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:54:30.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:31.590 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:54:31.589+0000 7f33fab23440 -1 osd.2 0 log_to_monitors true 2024-09-18T15:54:31.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:31.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:31.675 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:54:31.676 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:54:31.676 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:31.676 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:54:32.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:33.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:33.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:33.152 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:54:33.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:54:33.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:33.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:54:33.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:34.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:34.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:34.606 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:54:34.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:54:34.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:34.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:54:34.995 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:35.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:35.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:35.997 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:54:35.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:54:35.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:35.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:54:36.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:37.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:37.412 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:37.412 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:54:37.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:54:37.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:37.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:54:37.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:38.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:38.823 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:54:38.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:38.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:54:38.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:38.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:54:39.355 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:40.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:40.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:40.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:54:40.357 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:54:40.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:40.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:54:40.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:41.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:41.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:41.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:54:41.805 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:54:41.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:41.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:54:42.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:43.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:43.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:43.321 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:54:43.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:54:43.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:43.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:54:43.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:44.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:44.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:44.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:54:44.708 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:54:44.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:44.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:54:45.514 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:46.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:46.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:46.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:54:46.516 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:54:46.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:46.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:54:47.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:48.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:48.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:48.566 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:54:48.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:54:48.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:48.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:54:49.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:50.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:50.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:50.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:54:50.063 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:54:50.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:50.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:54:51.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:52.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:52.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:52.060 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:54:52.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:54:52.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:52.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:54:52.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:53.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:53.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:53.751 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:54:53.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:54:53.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:53.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:54:54.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:54:55.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:54:55.388 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T15:54:55.389 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:54:55.389 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:54:55.389 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:54:55.389 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:54:55.770 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/450187,v1:127.0.0.1:6819/450187] [v2:127.0.0.1:6820/450187,v1:127.0.0.1:6821/450187] exists,up b0f2b948-f6d4-4001-8d8a-7fdab0629567 2024-09-18T15:54:55.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:54:55.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:54:55.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:54:55.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:55: setup_osds: wait_for_clean 2024-09-18T15:54:55.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:54:55.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:54:55.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:54:55.772 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:54:55.772 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:54:55.773 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:54:55.773 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:54:55.773 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:54:55.773 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:54:55.862 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:54:55.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:54:55.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:54:55.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:54:55.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:54:55.863 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:54:56.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:54:56.392 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:54:56.392 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T15:54:56.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:54:56.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:54:56.392 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:54:56.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803795 2024-09-18T15:54:56.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803795 2024-09-18T15:54:56.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803795' 2024-09-18T15:54:56.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:54:56.579 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:54:56.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=60129542156 2024-09-18T15:54:56.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 60129542156 2024-09-18T15:54:56.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803795 1-60129542156' 2024-09-18T15:54:56.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:54:56.754 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:54:56.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345927 2024-09-18T15:54:56.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345927 2024-09-18T15:54:56.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803795 1-60129542156 2-85899345927' 2024-09-18T15:54:56.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:54:56.863 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803795 2024-09-18T15:54:56.863 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:54:56.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:54:56.865 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803795 2024-09-18T15:54:56.865 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:54:56.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803795 2024-09-18T15:54:56.866 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 25769803795 2024-09-18T15:54:56.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803795' 2024-09-18T15:54:56.866 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:54:57.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803793 -lt 25769803795 2024-09-18T15:54:57.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:54:58.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:54:58.290 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:54:58.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803796 -lt 25769803795 2024-09-18T15:54:58.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:54:58.665 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-60129542156 2024-09-18T15:54:58.665 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:54:58.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:54:58.666 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-60129542156 2024-09-18T15:54:58.666 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:54:58.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=60129542156 2024-09-18T15:54:58.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 60129542156' 2024-09-18T15:54:58.668 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 60129542156 2024-09-18T15:54:58.668 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:54:59.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 60129542156 -lt 60129542156 2024-09-18T15:54:59.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:54:59.079 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345927 2024-09-18T15:54:59.079 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:54:59.079 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:54:59.080 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345927 2024-09-18T15:54:59.080 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:54:59.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345927 2024-09-18T15:54:59.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345927' 2024-09-18T15:54:59.082 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 85899345927 2024-09-18T15:54:59.082 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:54:59.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345927 -lt 85899345927 2024-09-18T15:54:59.452 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:54:59.452 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:54:59.452 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:54:59.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-18T15:54:59.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:54:59.974 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:54:59.974 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:54:59.974 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:54:59.974 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:54:59.974 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:54:59.974 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:55:00.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:55:00.342 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:55:00.342 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:55:00.342 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:55:00.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 8 2024-09-18T15:55:00.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' -1 2024-09-18T15:55:00.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T15:55:00.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=5 2024-09-18T15:55:00.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:55:00.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:55:00.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:55:00.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:55:00.942 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:55:00.942 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:55:00.942 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:55:00.942 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:55:00.942 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:55:00.942 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:55:01.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T15:55:01.310 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:55:01.310 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:55:01.310 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:55:01.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 8 2024-09-18T15:55:01.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' 5 2024-09-18T15:55:01.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T15:55:01.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T15:55:01.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T15:55:01.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T15:55:01.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T15:55:01.809 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T15:55:01.809 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T15:55:02.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T15:55:02.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T15:55:02.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-09-18T15:55:02.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:55:02.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-18T15:55:02.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:55:02.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:55:02.566 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:55:02.566 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:55:02.567 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:55:02.567 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:55:02.567 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:55:02.567 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:55:02.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=7 2024-09-18T15:55:02.927 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:55:02.927 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:55:02.927 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:55:03.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 7 = 8 2024-09-18T15:55:03.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 7 '!=' 5 2024-09-18T15:55:03.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T15:55:03.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=7 2024-09-18T15:55:03.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T15:55:03.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T15:55:03.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T15:55:03.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:55:03.532 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:55:03.532 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:55:03.532 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:55:03.532 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:55:03.532 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:55:03.533 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:55:03.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-18T15:55:03.889 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:55:03.890 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:55:03.890 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:55:04.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-18T15:55:04.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:55:04.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:55:04.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:351: TEST_rep_recovery_unfound: local poolname=test-pool 2024-09-18T15:55:04.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:352: TEST_rep_recovery_unfound: create_pool test-pool 1 1 2024-09-18T15:55:04.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test-pool 1 1 2024-09-18T15:55:04.948 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test-pool' already exists 2024-09-18T15:55:04.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:55:05.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:353: TEST_rep_recovery_unfound: wait_for_clean 2024-09-18T15:55:05.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:55:05.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:55:05.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:55:05.961 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:55:05.962 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:55:05.962 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:55:05.962 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:55:05.962 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:55:05.962 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:55:06.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:55:06.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:55:06.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:55:06.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:55:06.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:55:06.064 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:55:06.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:55:06.598 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:55:06.598 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T15:55:06.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:55:06.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:55:06.598 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:55:06.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803799 2024-09-18T15:55:06.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803799 2024-09-18T15:55:06.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803799' 2024-09-18T15:55:06.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:55:06.709 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:55:06.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=60129542160 2024-09-18T15:55:06.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 60129542160 2024-09-18T15:55:06.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803799 1-60129542160' 2024-09-18T15:55:06.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:55:06.857 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:55:06.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345931 2024-09-18T15:55:06.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345931 2024-09-18T15:55:06.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803799 1-60129542160 2-85899345931' 2024-09-18T15:55:06.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:55:06.968 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803799 2024-09-18T15:55:06.968 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:55:06.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:55:06.969 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803799 2024-09-18T15:55:06.969 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:55:06.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803799 2024-09-18T15:55:06.970 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 25769803799 2024-09-18T15:55:06.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803799' 2024-09-18T15:55:06.970 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:55:07.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803797 -lt 25769803799 2024-09-18T15:55:07.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:55:08.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:55:08.349 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:55:08.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803800 -lt 25769803799 2024-09-18T15:55:08.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:55:08.729 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-60129542160 2024-09-18T15:55:08.729 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:55:08.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:55:08.730 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-60129542160 2024-09-18T15:55:08.731 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:55:08.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=60129542160 2024-09-18T15:55:08.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 60129542160' 2024-09-18T15:55:08.732 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 60129542160 2024-09-18T15:55:08.732 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:55:09.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 60129542160 -lt 60129542160 2024-09-18T15:55:09.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:55:09.122 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345931 2024-09-18T15:55:09.122 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:55:09.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:55:09.123 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345931 2024-09-18T15:55:09.124 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:55:09.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345931 2024-09-18T15:55:09.125 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 85899345931 2024-09-18T15:55:09.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345931' 2024-09-18T15:55:09.125 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:55:09.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345931 -lt 85899345931 2024-09-18T15:55:09.539 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:55:09.540 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:55:09.540 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:55:10.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-18T15:55:10.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:55:10.059 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:55:10.059 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:55:10.059 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:55:10.059 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:55:10.059 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:55:10.059 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:55:10.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-09-18T15:55:10.459 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:55:10.459 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:55:10.459 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:55:10.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-09-18T15:55:10.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:55:10.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:55:10.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:355: TEST_rep_recovery_unfound: ceph pg dump pgs 2024-09-18T15:55:11.304 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T15:55:11.304 INFO:tasks.workunit.client.0.smithi047.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:54:56.632781+0000 0'0 24:25 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T15:53:14.271502+0000 0'0 2024-09-18T15:53:14.271502+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:31:48.873964+0000 0 0 2024-09-18T15:55:11.305 INFO:tasks.workunit.client.0.smithi047.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:54:56.633046+0000 0'0 24:25 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T15:53:14.271502+0000 0'0 2024-09-18T15:53:14.271502+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:42:28.305512+0000 0 0 2024-09-18T15:55:11.305 INFO:tasks.workunit.client.0.smithi047.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:54:57.394631+0000 0'0 25:14 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T15:53:14.271502+0000 0'0 2024-09-18T15:53:14.271502+0000 0 0 periodic scrub scheduled @ 2024-09-19T18:27:59.034845+0000 0 0 2024-09-18T15:55:11.305 INFO:tasks.workunit.client.0.smithi047.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:54:56.633410+0000 0'0 24:25 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T15:53:14.271502+0000 0'0 2024-09-18T15:53:14.271502+0000 0 0 periodic scrub scheduled @ 2024-09-20T02:11:17.139320+0000 0 0 2024-09-18T15:55:11.305 INFO:tasks.workunit.client.0.smithi047.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:55:06.260355+0000 0'0 25:10 [2,1,0] 2 [2,1,0] 2 0'0 2024-09-18T15:55:04.728444+0000 0'0 2024-09-18T15:55:04.728444+0000 0 0 periodic scrub scheduled @ 2024-09-19T18:56:22.324769+0000 0 0 2024-09-18T15:55:11.305 INFO:tasks.workunit.client.0.smithi047.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:54:56.632903+0000 0'0 24:25 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T15:53:14.271502+0000 0'0 2024-09-18T15:53:14.271502+0000 0 0 periodic scrub scheduled @ 2024-09-19T19:58:43.689564+0000 0 0 2024-09-18T15:55:11.305 INFO:tasks.workunit.client.0.smithi047.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:54:57.512613+0000 0'0 24:39 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-18T15:53:14.271502+0000 0'0 2024-09-18T15:53:14.271502+0000 0 0 periodic scrub scheduled @ 2024-09-19T18:21:03.821406+0000 0 0 2024-09-18T15:55:11.305 INFO:tasks.workunit.client.0.smithi047.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:54:57.512524+0000 0'0 25:14 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T15:53:14.271502+0000 0'0 2024-09-18T15:53:14.271502+0000 0 0 periodic scrub scheduled @ 2024-09-19T23:21:38.965956+0000 0 0 2024-09-18T15:55:11.305 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:54:56.633777+0000 0'0 24:25 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T15:53:14.271502+0000 0'0 2024-09-18T15:53:14.271502+0000 0 0 periodic scrub scheduled @ 2024-09-20T00:38:58.495249+0000 0 0 2024-09-18T15:55:11.305 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T15:55:11.305 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T15:55:11.306 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T15:55:11.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:357: TEST_rep_recovery_unfound: rados_put td/osd-rep-recov-eio.sh test-pool myobject 2024-09-18T15:55:11.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:55:11.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=test-pool 2024-09-18T15:55:11.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=myobject 2024-09-18T15:55:11.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:55:11.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-18T15:55:11.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:55:11.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-18T15:55:11.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:55:11.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-18T15:55:11.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T15:55:11.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-18T15:55:11.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool test-pool put myobject td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:11.468 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:359: TEST_rep_recovery_unfound: get_osds test-pool myobject 2024-09-18T15:55:11.468 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-09-18T15:55:11.468 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-18T15:55:11.468 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool myobject 2024-09-18T15:55:11.468 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:55:11.837 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-09-18T15:55:11.837 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:55:11.837 INFO:tasks.workunit.client.0.smithi047.stderr:0' 2024-09-18T15:55:11.837 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-09-18T15:55:11.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:359: TEST_rep_recovery_unfound: initial_osds=('2' '1' '0') 2024-09-18T15:55:11.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:359: TEST_rep_recovery_unfound: local -a initial_osds 2024-09-18T15:55:11.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:360: TEST_rep_recovery_unfound: local last_osd=0 2024-09-18T15:55:11.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:361: TEST_rep_recovery_unfound: kill_daemons td/osd-rep-recov-eio.sh TERM osd.0 2024-09-18T15:55:11.838 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:55:11.838 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:55:11.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:55:11.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:55:11.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:55:12.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:55:12.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:362: TEST_rep_recovery_unfound: ceph osd down 0 2024-09-18T15:55:12.952 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already down. 2024-09-18T15:55:12.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:363: TEST_rep_recovery_unfound: ceph osd out 0 2024-09-18T15:55:13.546 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already out. 2024-09-18T15:55:13.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:365: TEST_rep_recovery_unfound: ceph pg dump pgs 2024-09-18T15:55:13.926 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T15:55:13.926 INFO:tasks.workunit.client.0.smithi047.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:54:56.632781+0000 0'0 25:26 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T15:53:14.271502+0000 0'0 2024-09-18T15:53:14.271502+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:31:48.873964+0000 0 0 2024-09-18T15:55:13.926 INFO:tasks.workunit.client.0.smithi047.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:54:56.633046+0000 0'0 25:26 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T15:53:14.271502+0000 0'0 2024-09-18T15:53:14.271502+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:42:28.305512+0000 0 0 2024-09-18T15:55:13.926 INFO:tasks.workunit.client.0.smithi047.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:54:57.394631+0000 0'0 25:14 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T15:53:14.271502+0000 0'0 2024-09-18T15:53:14.271502+0000 0 0 periodic scrub scheduled @ 2024-09-19T18:27:59.034845+0000 0 0 2024-09-18T15:55:13.926 INFO:tasks.workunit.client.0.smithi047.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:54:56.633410+0000 0'0 25:26 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T15:53:14.271502+0000 0'0 2024-09-18T15:53:14.271502+0000 0 0 periodic scrub scheduled @ 2024-09-20T02:11:17.139320+0000 0 0 2024-09-18T15:55:13.926 INFO:tasks.workunit.client.0.smithi047.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:55:06.260355+0000 0'0 25:10 [2,1,0] 2 [2,1,0] 2 0'0 2024-09-18T15:55:04.728444+0000 0'0 2024-09-18T15:55:04.728444+0000 0 0 periodic scrub scheduled @ 2024-09-19T18:56:22.324769+0000 0 0 2024-09-18T15:55:13.926 INFO:tasks.workunit.client.0.smithi047.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:54:56.632903+0000 0'0 25:26 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T15:53:14.271502+0000 0'0 2024-09-18T15:53:14.271502+0000 0 0 periodic scrub scheduled @ 2024-09-19T19:58:43.689564+0000 0 0 2024-09-18T15:55:13.926 INFO:tasks.workunit.client.0.smithi047.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 stale+active+clean 2024-09-18T15:54:57.512613+0000 0'0 24:39 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-18T15:53:14.271502+0000 0'0 2024-09-18T15:53:14.271502+0000 0 0 periodic scrub scheduled @ 2024-09-19T18:21:03.821406+0000 0 0 2024-09-18T15:55:13.927 INFO:tasks.workunit.client.0.smithi047.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:54:57.512524+0000 0'0 25:14 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T15:53:14.271502+0000 0'0 2024-09-18T15:53:14.271502+0000 0 0 periodic scrub scheduled @ 2024-09-19T23:21:38.965956+0000 0 0 2024-09-18T15:55:13.927 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:54:56.633777+0000 0'0 25:26 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T15:53:14.271502+0000 0'0 2024-09-18T15:53:14.271502+0000 0 0 periodic scrub scheduled @ 2024-09-20T00:38:58.495249+0000 0 0 2024-09-18T15:55:13.927 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T15:55:13.927 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T15:55:13.927 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T15:55:13.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:367: TEST_rep_recovery_unfound: dd if=/dev/urandom of=td/osd-rep-recov-eio.sh/ORIGINAL bs=1024 count=4 2024-09-18T15:55:13.937 INFO:tasks.workunit.client.0.smithi047.stderr:4+0 records in 2024-09-18T15:55:13.937 INFO:tasks.workunit.client.0.smithi047.stderr:4+0 records out 2024-09-18T15:55:13.937 INFO:tasks.workunit.client.0.smithi047.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 8.7896e-05 s, 46.6 MB/s 2024-09-18T15:55:13.938 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: seq 1 100 2024-09-18T15:55:13.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:13.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj1 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:14.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:14.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj2 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:14.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:14.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj3 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:14.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:14.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj4 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:14.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:14.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj5 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:14.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:14.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj6 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:14.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:14.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj7 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:14.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:14.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj8 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:15.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:15.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj9 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:15.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:15.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj10 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:15.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:15.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj11 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:15.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:15.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj12 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:15.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:15.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj13 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:15.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:15.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj14 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:15.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:15.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj15 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:15.907 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:15.907 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj16 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:16.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:16.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj17 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:16.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:16.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj18 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:16.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:16.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj19 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:16.419 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:16.419 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj20 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:16.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:16.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj21 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:16.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:16.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj22 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:16.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:16.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj23 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:16.946 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:16.946 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj24 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:17.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:17.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj25 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:17.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:17.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj26 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:17.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:17.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj27 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:17.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:17.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj28 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:17.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:17.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj29 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:17.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:17.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj30 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:17.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:17.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj31 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:17.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:17.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj32 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:18.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:18.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj33 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:18.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:18.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj34 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:18.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:18.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj35 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:18.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:18.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj36 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:18.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:18.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj37 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:18.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:18.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj38 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:18.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:18.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj39 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:19.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:19.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj40 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:19.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:19.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj41 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:19.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:19.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj42 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:19.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:19.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj43 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:19.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:19.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj44 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:19.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:19.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj45 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:19.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:19.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj46 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:19.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:19.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj47 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:20.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:20.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj48 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:20.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:20.197 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj49 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:20.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:20.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj50 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:20.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:20.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj51 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:20.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:20.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj52 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:20.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:20.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj53 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:20.875 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:20.875 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj54 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:21.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:21.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj55 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:21.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:21.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj56 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:21.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:21.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj57 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:21.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:21.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj58 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:21.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:21.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj59 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:21.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:21.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj60 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:21.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:21.772 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj61 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:21.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:21.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj62 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:22.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:22.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj63 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:22.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:22.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj64 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:22.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:22.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj65 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:22.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:22.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj66 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:22.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:22.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj67 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:22.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:22.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj68 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:22.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:22.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj69 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:22.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:22.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj70 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:23.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:23.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj71 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:23.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:23.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj72 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:23.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:23.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj73 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:23.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:23.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj74 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:23.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:23.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj75 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:23.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:23.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj76 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:23.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:23.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj77 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:23.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:23.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj78 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:24.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:24.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj79 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:24.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:24.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj80 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:24.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:24.353 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj81 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:24.470 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:24.470 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj82 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:24.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:24.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj83 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:24.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:24.713 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj84 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:24.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:24.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj85 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:24.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:24.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj86 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:25.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:25.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj87 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:25.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:25.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj88 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:25.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:25.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj89 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:25.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:25.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj90 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:25.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:25.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj91 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:25.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:25.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj92 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:25.860 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:25.860 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj93 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:25.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:25.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj94 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:26.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:26.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj95 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:26.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:26.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj96 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:26.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:26.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj97 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:26.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:26.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj98 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:26.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:26.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj99 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:26.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:55:26.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj100 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-18T15:55:26.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:373: TEST_rep_recovery_unfound: inject_eio rep data test-pool obj75 td/osd-rep-recov-eio.sh 0 2024-09-18T15:55:26.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:55:26.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:55:26.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:55:26.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:55:26.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=test-pool 2024-09-18T15:55:26.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:55:26.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj75 2024-09-18T15:55:26.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:55:26.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:55:26.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:55:26.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T15:55:26.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:55:26.918 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds test-pool obj75 2024-09-18T15:55:26.918 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-09-18T15:55:26.919 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj75 2024-09-18T15:55:26.919 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool obj75 2024-09-18T15:55:26.919 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:55:27.289 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-09-18T15:55:27.289 INFO:tasks.workunit.client.0.smithi047.stderr:1' 2024-09-18T15:55:27.290 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 2024-09-18T15:55:27.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('2' '1') 2024-09-18T15:55:27.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:55:27.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=2 2024-09-18T15:55:27.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:55:27.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:55:27.290 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/2/type 2024-09-18T15:55:27.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:55:27.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 2 bluestore_debug_inject_read_err true 2024-09-18T15:55:27.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:55:27.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=2 2024-09-18T15:55:27.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:55:27.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:55:27.292 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:55:27.292 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.2 2024-09-18T15:55:27.293 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T15:55:27.293 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T15:55:27.293 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:55:27.293 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:55:27.293 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:55:27.293 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.2.asok 2024-09-18T15:55:27.293 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.2.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:55:27.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:55:27.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:55:27.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:55:27.376 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.2 2024-09-18T15:55:27.376 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T15:55:27.376 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T15:55:27.376 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:55:27.376 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:55:27.376 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:55:27.376 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.2.asok 2024-09-18T15:55:27.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:55:27.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.2.asok injectdataerr test-pool obj75 2024-09-18T15:55:27.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:374: TEST_rep_recovery_unfound: inject_eio rep data test-pool obj75 td/osd-rep-recov-eio.sh 1 2024-09-18T15:55:27.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-18T15:55:27.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T15:55:27.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T15:55:27.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T15:55:27.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=test-pool 2024-09-18T15:55:27.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T15:55:27.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj75 2024-09-18T15:55:27.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T15:55:27.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:55:27.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T15:55:27.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-18T15:55:27.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T15:55:27.461 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds test-pool obj75 2024-09-18T15:55:27.461 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-09-18T15:55:27.462 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj75 2024-09-18T15:55:27.462 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool obj75 2024-09-18T15:55:27.462 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T15:55:27.829 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-09-18T15:55:27.829 INFO:tasks.workunit.client.0.smithi047.stderr:1' 2024-09-18T15:55:27.829 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 2024-09-18T15:55:27.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('2' '1') 2024-09-18T15:55:27.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T15:55:27.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-18T15:55:27.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-18T15:55:27.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-18T15:55:27.830 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-18T15:55:27.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T15:55:27.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-18T15:55:27.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T15:55:27.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-18T15:55:27.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T15:55:27.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T15:55:27.832 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T15:55:27.832 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-18T15:55:27.832 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:55:27.832 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:55:27.832 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:55:27.832 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:55:27.832 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:55:27.832 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.1.asok 2024-09-18T15:55:27.832 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.397939/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-18T15:55:27.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T15:55:27.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T15:55:27.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T15:55:27.915 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-18T15:55:27.915 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T15:55:27.915 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T15:55:27.915 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:55:27.915 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:55:27.916 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:55:27.916 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.397939/ceph-osd.1.asok 2024-09-18T15:55:27.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T15:55:27.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.397939/ceph-osd.1.asok injectdataerr test-pool obj75 2024-09-18T15:55:28.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:376: TEST_rep_recovery_unfound: activate_osd td/osd-rep-recov-eio.sh 0 2024-09-18T15:55:28.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:55:28.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:55:28.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T15:55:28.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:55:28.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-09-18T15:55:28.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-18T15:55:28.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:55:28.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:55:28.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:55:28.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-09-18T15:55:28.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-09-18T15:55:28.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:55:28.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:55:28.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-18T15:55:28.004 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:55:28.004 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:55:28.004 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:55:28.004 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:55:28.004 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:55:28.004 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:55:28.004 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:55:28.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' 2024-09-18T15:55:28.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:55:28.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-18T15:55:28.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-18T15:55:28.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:55:28.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:55:28.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:55:28.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:55:28.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:55:28.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:55:28.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-09-18T15:55:28.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T15:55:28.006 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T15:55:28.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=19cc9b06-7e46-4b89-943e-ffc293e1c9de --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.397939/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:55:28.006 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-rep-recov-eio.sh/0/whoami 2024-09-18T15:55:28.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T15:55:28.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:55:28.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:55:28.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:55:28.027 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:55:28.026+0000 7f7566b1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:55:28.029 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:55:28.028+0000 7f7566b1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:55:28.030 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:55:28.029+0000 7f7566b1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:55:28.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T15:55:28.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:55:28.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:55:28.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:55:28.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:55:28.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:55:28.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:55:28.415 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:55:28.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:55:28.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:55:28.550 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:55:28.549+0000 7f7566b1f440 -1 Falling back to public interface 2024-09-18T15:55:28.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:55:29.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:55:29.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:55:29.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:55:29.785 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:55:29.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:55:29.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:55:30.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:55:31.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:55:31.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:55:31.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:55:31.166 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:55:31.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:55:31.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:55:31.412 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:55:31.411+0000 7f7566b1f440 -1 osd.0 26 log_to_monitors true 2024-09-18T15:55:31.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:55:32.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:55:32.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:55:32.618 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:55:32.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:55:32.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:55:32.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:55:33.248 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up out weight 0 up_from 32 up_thru 21 down_at 27 last_clean_interval [6,26) [v2:127.0.0.1:6802/454205,v1:127.0.0.1:6803/454205] [v2:127.0.0.1:6804/454205,v1:127.0.0.1:6805/454205] exists,up 6c09a867-7649-4abc-8f06-2ffadd99ea85 2024-09-18T15:55:33.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:55:33.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:55:33.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:55:33.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:377: TEST_rep_recovery_unfound: ceph osd in 0 2024-09-18T15:55:34.183 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already in. 2024-09-18T15:55:34.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:379: TEST_rep_recovery_unfound: sleep 15 2024-09-18T15:55:49.195 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:381: TEST_rep_recovery_unfound: seq 1 100 2024-09-18T15:55:49.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:381: TEST_rep_recovery_unfound: for tmp in $(seq 1 100) 2024-09-18T15:55:49.197 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:382: TEST_rep_recovery_unfound: get_state 2.0 2024-09-18T15:55:49.197 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-09-18T15:55:49.197 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-09-18T15:55:49.197 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-09-18T15:55:49.197 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-18T15:55:49.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:382: TEST_rep_recovery_unfound: state=active+recovery_unfound+degraded 2024-09-18T15:55:49.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:383: TEST_rep_recovery_unfound: echo active+recovery_unfound+degraded 2024-09-18T15:55:49.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:383: TEST_rep_recovery_unfound: grep -v recovering 2024-09-18T15:55:49.568 INFO:tasks.workunit.client.0.smithi047.stdout:active+recovery_unfound+degraded 2024-09-18T15:55:49.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:384: TEST_rep_recovery_unfound: '[' 0 = 0 ']' 2024-09-18T15:55:49.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:385: TEST_rep_recovery_unfound: break 2024-09-18T15:55:49.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:391: TEST_rep_recovery_unfound: ceph pg dump pgs 2024-09-18T15:55:49.920 INFO:tasks.workunit.client.0.smithi047.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T15:55:49.920 INFO:tasks.workunit.client.0.smithi047.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:55:34.829248+0000 0'0 39:67 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T15:53:14.271502+0000 0'0 2024-09-18T15:53:14.271502+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:33:35.782589+0000 0 0 2024-09-18T15:55:49.920 INFO:tasks.workunit.client.0.smithi047.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:55:36.187460+0000 0'0 40:69 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T15:53:14.271502+0000 0'0 2024-09-18T15:53:14.271502+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:46:45.617111+0000 0 0 2024-09-18T15:55:49.920 INFO:tasks.workunit.client.0.smithi047.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:55:36.187659+0000 0'0 40:57 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T15:53:14.271502+0000 0'0 2024-09-18T15:53:14.271502+0000 0 0 periodic scrub scheduled @ 2024-09-19T20:21:43.439956+0000 0 0 2024-09-18T15:55:49.920 INFO:tasks.workunit.client.0.smithi047.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:55:36.187553+0000 0'0 40:69 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T15:53:14.271502+0000 0'0 2024-09-18T15:53:14.271502+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:02:38.903149+0000 0 0 2024-09-18T15:55:49.920 INFO:tasks.workunit.client.0.smithi047.stdout:2.0 101 1 3 0 1 413696 0 0 101 0 101 active+recovery_unfound+degraded 2024-09-18T15:55:39.333002+0000 30'101 40:351 [2,1,0] 2 [2,1,0] 2 0'0 2024-09-18T15:55:04.728444+0000 0'0 2024-09-18T15:55:04.728444+0000 0 0 periodic scrub scheduled @ 2024-09-19T23:19:48.992470+0000 0 0 2024-09-18T15:55:49.920 INFO:tasks.workunit.client.0.smithi047.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:55:34.829180+0000 0'0 39:67 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-18T15:53:14.271502+0000 0'0 2024-09-18T15:53:14.271502+0000 0 0 periodic scrub scheduled @ 2024-09-20T00:00:54.537952+0000 0 0 2024-09-18T15:55:49.920 INFO:tasks.workunit.client.0.smithi047.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:55:35.444942+0000 0'0 39:61 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-18T15:53:14.271502+0000 0'0 2024-09-18T15:53:14.271502+0000 0 0 periodic scrub scheduled @ 2024-09-19T18:30:53.131949+0000 0 0 2024-09-18T15:55:49.920 INFO:tasks.workunit.client.0.smithi047.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:55:36.188149+0000 0'0 40:57 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-18T15:53:14.271502+0000 0'0 2024-09-18T15:53:14.271502+0000 0 0 periodic scrub scheduled @ 2024-09-19T23:20:53.202596+0000 0 0 2024-09-18T15:55:49.921 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T15:55:36.187560+0000 0'0 40:69 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T15:53:14.271502+0000 0'0 2024-09-18T15:53:14.271502+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:17:54.211053+0000 0 0 2024-09-18T15:55:49.921 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T15:55:49.921 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T15:55:49.921 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pgs 2024-09-18T15:55:49.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:392: TEST_rep_recovery_unfound: ceph pg 2.0 list_unfound 2024-09-18T15:55:49.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:392: TEST_rep_recovery_unfound: grep -q obj75 2024-09-18T15:55:50.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:395: TEST_rep_recovery_unfound: timeout 5 rados -p test-pool get obj75 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:55:55.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:396: TEST_rep_recovery_unfound: test 124 = 124 2024-09-18T15:55:55.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:398: TEST_rep_recovery_unfound: ceph pg 2.0 mark_unfound_lost delete 2024-09-18T15:55:55.236 INFO:tasks.workunit.client.0.smithi047.stderr:pg has no unfound objects 2024-09-18T15:55:55.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:400: TEST_rep_recovery_unfound: wait_for_clean 2024-09-18T15:55:55.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:55:55.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:55:55.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:55:55.246 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:55:55.246 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:55:55.246 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:55:55.246 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:55:55.247 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:55:55.247 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:55:55.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:55:55.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:55:55.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:55:55.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:55:55.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:55:55.338 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:55:55.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:55:55.721 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:55:55.721 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T15:55:55.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:55:55.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:55:55.722 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:55:55.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=137438953479 2024-09-18T15:55:55.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 137438953479 2024-09-18T15:55:55.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-137438953479' 2024-09-18T15:55:55.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:55:55.831 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:55:55.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=60129542172 2024-09-18T15:55:55.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 60129542172 2024-09-18T15:55:55.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-137438953479 1-60129542172' 2024-09-18T15:55:55.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:55:55.962 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:55:56.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345943 2024-09-18T15:55:56.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345943 2024-09-18T15:55:56.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-137438953479 1-60129542172 2-85899345943' 2024-09-18T15:55:56.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:55:56.072 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-137438953479 2024-09-18T15:55:56.072 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:55:56.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:55:56.077 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-137438953479 2024-09-18T15:55:56.077 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:55:56.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=137438953479 2024-09-18T15:55:56.077 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 137438953479 2024-09-18T15:55:56.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 137438953479' 2024-09-18T15:55:56.078 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:55:56.483 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 137438953477 -lt 137438953479 2024-09-18T15:55:56.483 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:55:57.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:55:57.485 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:55:57.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 137438953477 -lt 137438953479 2024-09-18T15:55:57.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:55:58.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-18T15:55:58.937 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:55:59.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 137438953480 -lt 137438953479 2024-09-18T15:55:59.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:55:59.354 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-60129542172 2024-09-18T15:55:59.354 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:55:59.355 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:55:59.355 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-60129542172 2024-09-18T15:55:59.356 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:55:59.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=60129542172 2024-09-18T15:55:59.357 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 60129542172 2024-09-18T15:55:59.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 60129542172' 2024-09-18T15:55:59.357 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:55:59.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 60129542172 -lt 60129542172 2024-09-18T15:55:59.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:55:59.764 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345943 2024-09-18T15:55:59.764 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:55:59.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:55:59.765 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345943 2024-09-18T15:55:59.765 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:55:59.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345943 2024-09-18T15:55:59.766 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 85899345943 2024-09-18T15:55:59.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345943' 2024-09-18T15:55:59.766 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:56:00.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345943 -lt 85899345943 2024-09-18T15:56:00.137 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:56:00.138 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:56:00.138 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:56:00.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-18T15:56:00.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:56:00.636 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:56:00.636 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:56:00.636 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:56:00.637 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:56:00.637 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:56:00.637 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:56:01.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-09-18T15:56:01.037 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:56:01.037 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:56:01.037 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:56:01.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-09-18T15:56:01.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:56:01.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:56:01.537 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: seq 1 100 2024-09-18T15:56:01.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:01.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj1 = obj75 ']' 2024-09-18T15:56:01.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj1 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:01.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:01.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:01.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj2 = obj75 ']' 2024-09-18T15:56:01.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj2 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:01.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:01.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:01.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj3 = obj75 ']' 2024-09-18T15:56:01.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj3 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:01.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:01.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:01.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj4 = obj75 ']' 2024-09-18T15:56:01.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj4 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:01.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:01.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:01.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj5 = obj75 ']' 2024-09-18T15:56:01.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj5 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:01.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:01.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:01.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj6 = obj75 ']' 2024-09-18T15:56:01.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj6 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:01.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:01.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:01.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj7 = obj75 ']' 2024-09-18T15:56:01.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj7 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:01.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:01.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:01.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj8 = obj75 ']' 2024-09-18T15:56:01.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj8 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:01.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:01.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:01.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj9 = obj75 ']' 2024-09-18T15:56:01.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj9 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:01.875 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:01.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:01.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj10 = obj75 ']' 2024-09-18T15:56:01.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj10 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:01.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:01.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:01.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj11 = obj75 ']' 2024-09-18T15:56:01.904 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj11 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:01.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:01.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:01.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj12 = obj75 ']' 2024-09-18T15:56:01.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj12 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:01.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:01.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:01.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj13 = obj75 ']' 2024-09-18T15:56:01.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj13 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:01.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:01.988 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:01.988 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj14 = obj75 ']' 2024-09-18T15:56:01.988 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj14 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.014 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:02.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj15 = obj75 ']' 2024-09-18T15:56:02.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj15 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:02.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj16 = obj75 ']' 2024-09-18T15:56:02.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj16 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:02.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj17 = obj75 ']' 2024-09-18T15:56:02.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj17 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:02.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj18 = obj75 ']' 2024-09-18T15:56:02.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj18 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:02.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj19 = obj75 ']' 2024-09-18T15:56:02.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj19 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.155 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:02.155 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj20 = obj75 ']' 2024-09-18T15:56:02.155 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj20 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:02.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj21 = obj75 ']' 2024-09-18T15:56:02.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj21 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:02.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj22 = obj75 ']' 2024-09-18T15:56:02.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj22 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:02.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj23 = obj75 ']' 2024-09-18T15:56:02.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj23 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:02.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj24 = obj75 ']' 2024-09-18T15:56:02.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj24 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:02.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj25 = obj75 ']' 2024-09-18T15:56:02.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj25 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:02.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj26 = obj75 ']' 2024-09-18T15:56:02.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj26 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:02.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj27 = obj75 ']' 2024-09-18T15:56:02.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj27 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:02.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj28 = obj75 ']' 2024-09-18T15:56:02.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj28 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:02.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj29 = obj75 ']' 2024-09-18T15:56:02.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj29 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.469 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.470 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:02.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj30 = obj75 ']' 2024-09-18T15:56:02.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj30 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:02.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj31 = obj75 ']' 2024-09-18T15:56:02.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj31 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:02.527 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj32 = obj75 ']' 2024-09-18T15:56:02.527 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj32 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:02.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj33 = obj75 ']' 2024-09-18T15:56:02.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj33 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:02.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj34 = obj75 ']' 2024-09-18T15:56:02.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj34 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:02.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj35 = obj75 ']' 2024-09-18T15:56:02.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj35 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:02.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj36 = obj75 ']' 2024-09-18T15:56:02.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj36 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.666 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:02.666 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj37 = obj75 ']' 2024-09-18T15:56:02.666 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj37 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:02.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj38 = obj75 ']' 2024-09-18T15:56:02.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj38 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:02.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj39 = obj75 ']' 2024-09-18T15:56:02.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj39 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:02.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj40 = obj75 ']' 2024-09-18T15:56:02.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj40 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:02.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj41 = obj75 ']' 2024-09-18T15:56:02.779 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj41 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:02.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj42 = obj75 ']' 2024-09-18T15:56:02.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj42 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:02.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj43 = obj75 ']' 2024-09-18T15:56:02.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj43 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.861 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.862 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:02.862 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj44 = obj75 ']' 2024-09-18T15:56:02.862 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj44 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:02.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj45 = obj75 ']' 2024-09-18T15:56:02.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj45 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:02.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj46 = obj75 ']' 2024-09-18T15:56:02.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj46 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:02.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj47 = obj75 ']' 2024-09-18T15:56:02.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj47 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:02.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:02.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj48 = obj75 ']' 2024-09-18T15:56:02.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj48 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj49 = obj75 ']' 2024-09-18T15:56:03.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj49 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj50 = obj75 ']' 2024-09-18T15:56:03.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj50 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj51 = obj75 ']' 2024-09-18T15:56:03.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj51 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj52 = obj75 ']' 2024-09-18T15:56:03.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj52 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj53 = obj75 ']' 2024-09-18T15:56:03.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj53 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj54 = obj75 ']' 2024-09-18T15:56:03.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj54 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj55 = obj75 ']' 2024-09-18T15:56:03.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj55 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.197 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj56 = obj75 ']' 2024-09-18T15:56:03.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj56 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj57 = obj75 ']' 2024-09-18T15:56:03.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj57 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj58 = obj75 ']' 2024-09-18T15:56:03.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj58 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj59 = obj75 ']' 2024-09-18T15:56:03.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj59 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj60 = obj75 ']' 2024-09-18T15:56:03.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj60 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj61 = obj75 ']' 2024-09-18T15:56:03.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj61 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj62 = obj75 ']' 2024-09-18T15:56:03.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj62 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj63 = obj75 ']' 2024-09-18T15:56:03.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj63 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj64 = obj75 ']' 2024-09-18T15:56:03.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj64 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj65 = obj75 ']' 2024-09-18T15:56:03.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj65 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.477 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj66 = obj75 ']' 2024-09-18T15:56:03.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj66 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj67 = obj75 ']' 2024-09-18T15:56:03.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj67 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj68 = obj75 ']' 2024-09-18T15:56:03.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj68 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj69 = obj75 ']' 2024-09-18T15:56:03.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj69 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj70 = obj75 ']' 2024-09-18T15:56:03.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj70 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj71 = obj75 ']' 2024-09-18T15:56:03.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj71 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj72 = obj75 ']' 2024-09-18T15:56:03.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj72 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj73 = obj75 ']' 2024-09-18T15:56:03.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj73 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj74 = obj75 ']' 2024-09-18T15:56:03.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj74 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj75 = obj75 ']' 2024-09-18T15:56:03.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:406: TEST_rep_recovery_unfound: rados -p test-pool get obj75 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.754 INFO:tasks.workunit.client.0.smithi047.stderr:error getting test-pool/obj75: (2) No such file or directory 2024-09-18T15:56:03.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj76 = obj75 ']' 2024-09-18T15:56:03.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj76 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj77 = obj75 ']' 2024-09-18T15:56:03.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj77 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj78 = obj75 ']' 2024-09-18T15:56:03.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj78 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj79 = obj75 ']' 2024-09-18T15:56:03.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj79 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj80 = obj75 ']' 2024-09-18T15:56:03.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj80 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj81 = obj75 ']' 2024-09-18T15:56:03.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj81 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj82 = obj75 ']' 2024-09-18T15:56:03.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj82 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj83 = obj75 ']' 2024-09-18T15:56:03.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj83 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.989 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:03.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:03.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj84 = obj75 ']' 2024-09-18T15:56:03.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj84 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:04.016 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:04.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:04.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj85 = obj75 ']' 2024-09-18T15:56:04.017 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj85 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:04.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:04.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:04.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj86 = obj75 ']' 2024-09-18T15:56:04.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj86 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:04.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:04.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:04.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj87 = obj75 ']' 2024-09-18T15:56:04.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj87 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:04.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:04.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:04.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj88 = obj75 ']' 2024-09-18T15:56:04.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj88 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:04.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:04.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:04.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj89 = obj75 ']' 2024-09-18T15:56:04.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj89 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:04.155 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:04.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:04.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj90 = obj75 ']' 2024-09-18T15:56:04.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj90 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:04.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:04.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:04.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj91 = obj75 ']' 2024-09-18T15:56:04.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj91 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:04.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:04.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:04.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj92 = obj75 ']' 2024-09-18T15:56:04.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj92 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:04.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:04.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:04.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj93 = obj75 ']' 2024-09-18T15:56:04.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj93 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:04.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:04.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:04.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj94 = obj75 ']' 2024-09-18T15:56:04.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj94 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:04.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:04.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:04.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj95 = obj75 ']' 2024-09-18T15:56:04.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj95 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:04.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:04.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:04.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj96 = obj75 ']' 2024-09-18T15:56:04.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj96 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:04.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:04.389 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:04.389 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj97 = obj75 ']' 2024-09-18T15:56:04.389 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj97 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:04.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:04.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:04.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj98 = obj75 ']' 2024-09-18T15:56:04.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj98 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:04.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:04.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:04.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj99 = obj75 ']' 2024-09-18T15:56:04.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj99 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:04.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:04.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T15:56:04.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj100 = obj75 ']' 2024-09-18T15:56:04.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj100 td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:04.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:04.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:413: TEST_rep_recovery_unfound: rm -f td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-18T15:56:04.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:415: TEST_rep_recovery_unfound: delete_pool test-pool 2024-09-18T15:56:04.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test-pool 2024-09-18T15:56:04.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test-pool test-pool --yes-i-really-really-mean-it 2024-09-18T15:56:05.029 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test-pool' does not exist 2024-09-18T15:56:05.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:43: run: teardown td/osd-rep-recov-eio.sh 2024-09-18T15:56:05.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:56:05.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:56:05.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-09-18T15:56:05.041 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:56:05.041 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:56:05.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:56:05.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:56:05.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:56:05.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:56:05.149 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:56:05.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:56:05.150 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:56:05.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:56:05.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:56:05.152 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:56:05.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:56:05.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:56:05.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:56:05.153 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:56:05.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:56:05.155 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:56:05.155 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-09-18T15:56:05.179 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:56:05.179 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:56:05.179 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:56:05.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.397939 2024-09-18T15:56:05.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:56:05.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:56:05.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T15:56:05.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/osd-rep-recov-eio.sh 0 2024-09-18T15:56:05.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-09-18T15:56:05.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T15:56:05.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-09-18T15:56:05.181 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:56:05.181 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:56:05.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:56:05.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:56:05.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:56:05.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:56:05.183 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:56:05.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:56:05.184 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:56:05.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:56:05.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:56:05.185 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:56:05.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:56:05.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:56:05.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:56:05.187 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:56:05.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:56:05.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T15:56:05.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-09-18T15:56:05.190 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:56:05.190 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:56:05.190 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.397939 2024-09-18T15:56:05.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.397939 2024-09-18T15:56:05.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:56:05.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:56:05.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T15:56:05.192 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T15:56:05.192 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T15:56:05.218 INFO:tasks.workunit:Running workunit osd/osd-reuse-id.sh... 2024-09-18T15:56:05.218 DEBUG:teuthology.orchestra.run.smithi047:workunit test osd/osd-reuse-id.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh 2024-09-18T15:56:05.280 INFO:tasks.workunit.client.0.smithi047.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T15:56:05.285 INFO:tasks.workunit.client.0.smithi047.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T15:56:05.285 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T15:56:05.285 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T15:56:05.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T15:56:05.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T15:56:05.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T15:56:05.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T15:56:05.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T15:56:05.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T15:56:05.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/osd-reuse-id 2024-09-18T15:56:05.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:20: run: local dir=td/osd-reuse-id 2024-09-18T15:56:05.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:21: run: shift 2024-09-18T15:56:05.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:23: run: export CEPH_MON=127.0.0.1:7123 2024-09-18T15:56:05.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:23: run: CEPH_MON=127.0.0.1:7123 2024-09-18T15:56:05.286 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:24: run: export CEPH_ARGS 2024-09-18T15:56:05.286 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:25: run: uuidgen 2024-09-18T15:56:05.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:25: run: CEPH_ARGS+='--fsid=ef149c1c-850d-450f-9329-b83add5652f7 --auth-supported=none ' 2024-09-18T15:56:05.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7123 ' 2024-09-18T15:56:05.287 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:28: run: set 2024-09-18T15:56:05.287 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:28: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-18T15:56:05.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:28: run: local funcs=TEST_reuse_id 2024-09-18T15:56:05.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:29: run: for func in $funcs 2024-09-18T15:56:05.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:30: run: setup td/osd-reuse-id 2024-09-18T15:56:05.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-reuse-id 2024-09-18T15:56:05.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-reuse-id 2024-09-18T15:56:05.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reuse-id 2024-09-18T15:56:05.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:56:05.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reuse-id KILL 2024-09-18T15:56:05.289 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:56:05.289 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:56:05.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:56:05.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:56:05.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:56:05.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:56:05.290 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:56:05.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:56:05.291 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:56:05.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:56:05.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:56:05.292 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:56:05.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:56:05.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:56:05.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:56:05.293 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:56:05.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:56:05.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:56:05.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reuse-id 2024-09-18T15:56:05.296 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:56:05.296 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:56:05.296 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.457476 2024-09-18T15:56:05.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.457476 2024-09-18T15:56:05.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:56:05.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:56:05.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-reuse-id 2024-09-18T15:56:05.298 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T15:56:05.298 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:56:05.298 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.457476 2024-09-18T15:56:05.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.457476 2024-09-18T15:56:05.300 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T15:56:05.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T15:56:05.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T15:56:05.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T15:56:05.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-reuse-id 1' TERM HUP INT 2024-09-18T15:56:05.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:31: run: TEST_reuse_id td/osd-reuse-id 2024-09-18T15:56:05.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:37: TEST_reuse_id: local dir=td/osd-reuse-id 2024-09-18T15:56:05.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:39: TEST_reuse_id: run_mon td/osd-reuse-id a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T15:56:05.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-reuse-id 2024-09-18T15:56:05.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T15:56:05.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T15:56:05.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T15:56:05.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-reuse-id/a 2024-09-18T15:56:05.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-reuse-id/a --run-dir=td/osd-reuse-id --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T15:56:05.644 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T15:56:05.644 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:56:05.644 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:56:05.644 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:56:05.644 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:56:05.644 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.457476 2024-09-18T15:56:05.645 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.457476/$cluster-$name.asok' 2024-09-18T15:56:05.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-reuse-id/a '--log-file=td/osd-reuse-id/$name.log' '--admin-socket=/tmp/ceph-asok.457476/$cluster-$name.asok' --mon-cluster-log-file=td/osd-reuse-id/log --run-dir=td/osd-reuse-id '--pid-file=td/osd-reuse-id/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T15:56:05.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T15:56:05.880 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T15:56:05.880 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:56:05.880 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:56:05.880 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T15:56:05.880 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T15:56:05.880 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:56:05.880 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:56:05.880 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:56:05.881 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:56:05.881 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:56:05.881 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.457476 2024-09-18T15:56:05.881 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.457476/ceph-mon.a.asok 2024-09-18T15:56:05.881 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:56:05.881 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.457476/ceph-mon.a.asok config get fsid 2024-09-18T15:56:05.961 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T15:56:05.961 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:56:05.962 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:56:05.962 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T15:56:05.962 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T15:56:05.962 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:56:05.962 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:56:05.962 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:56:05.962 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:56:05.962 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:56:05.962 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.457476 2024-09-18T15:56:05.962 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.457476/ceph-mon.a.asok 2024-09-18T15:56:05.962 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:56:05.962 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.457476/ceph-mon.a.asok config get mon_host 2024-09-18T15:56:06.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:40: TEST_reuse_id: run_mgr td/osd-reuse-id x 2024-09-18T15:56:06.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-reuse-id 2024-09-18T15:56:06.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T15:56:06.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T15:56:06.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T15:56:06.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-reuse-id/x 2024-09-18T15:56:06.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T15:56:06.885 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T15:56:06.886 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:56:06.886 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:56:06.886 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:56:06.886 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:56:06.886 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.457476 2024-09-18T15:56:06.886 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.457476/$cluster-$name.asok' 2024-09-18T15:56:06.886 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:56:06.887 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-reuse-id/x '--log-file=td/osd-reuse-id/$name.log' '--admin-socket=/tmp/ceph-asok.457476/$cluster-$name.asok' --run-dir=td/osd-reuse-id '--pid-file=td/osd-reuse-id/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:56:06.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:41: TEST_reuse_id: run_osd td/osd-reuse-id 0 2024-09-18T15:56:06.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-reuse-id 2024-09-18T15:56:06.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:56:06.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T15:56:06.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:56:06.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-reuse-id/0 2024-09-18T15:56:06.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ef149c1c-850d-450f-9329-b83add5652f7 --auth-supported=none --mon-host=127.0.0.1:7123 ' 2024-09-18T15:56:06.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:56:06.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:56:06.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:56:06.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-reuse-id/0' 2024-09-18T15:56:06.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-reuse-id/0/journal' 2024-09-18T15:56:06.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:56:06.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:56:06.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-reuse-id' 2024-09-18T15:56:06.913 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:56:06.913 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:56:06.913 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:56:06.913 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:56:06.913 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:56:06.913 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.457476 2024-09-18T15:56:06.914 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.457476/$cluster-$name.asok' 2024-09-18T15:56:06.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.457476/$cluster-$name.asok' 2024-09-18T15:56:06.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:56:06.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:56:06.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:56:06.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-reuse-id/$name.log' 2024-09-18T15:56:06.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-reuse-id/$name.pid' 2024-09-18T15:56:06.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:56:06.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:56:06.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:56:06.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:56:06.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:56:06.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:56:06.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-reuse-id/0 2024-09-18T15:56:06.914 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:56:06.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c87d6959-2eab-42d1-a750-c140730c97e5 2024-09-18T15:56:06.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c87d6959-2eab-42d1-a750-c140730c97e5' 2024-09-18T15:56:06.915 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 c87d6959-2eab-42d1-a750-c140730c97e5 2024-09-18T15:56:06.916 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:56:06.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAW+OpmUPUrNxAAIUNYamcbpOQ6/tnNhFx4WA== 2024-09-18T15:56:06.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAW+OpmUPUrNxAAIUNYamcbpOQ6/tnNhFx4WA=="}' 2024-09-18T15:56:06.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c87d6959-2eab-42d1-a750-c140730c97e5 -i td/osd-reuse-id/0/new.json 2024-09-18T15:56:07.226 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:56:07.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-reuse-id/0/new.json 2024-09-18T15:56:07.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=ef149c1c-850d-450f-9329-b83add5652f7 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/0 --osd-journal=td/osd-reuse-id/0/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.457476/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAW+OpmUPUrNxAAIUNYamcbpOQ6/tnNhFx4WA== --osd-uuid c87d6959-2eab-42d1-a750-c140730c97e5 2024-09-18T15:56:07.259 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:56:07.258+0000 7f4acea34440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:56:07.271 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:56:07.271+0000 7f4acea34440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:56:07.273 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:56:07.272+0000 7f4acea34440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:56:07.273 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:56:07.272+0000 7f4acea34440 -1 bluestore(td/osd-reuse-id/0/block) _read_bdev_label failed to open td/osd-reuse-id/0/block: (2) No such file or directory 2024-09-18T15:56:07.273 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:56:07.272+0000 7f4acea34440 -1 bluestore(td/osd-reuse-id/0/block) _read_bdev_label failed to open td/osd-reuse-id/0/block: (2) No such file or directory 2024-09-18T15:56:07.273 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:56:07.272+0000 7f4acea34440 -1 bluestore(td/osd-reuse-id/0/block) _read_bdev_label failed to open td/osd-reuse-id/0/block: (2) No such file or directory 2024-09-18T15:56:07.312 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:56:07.311+0000 7f4acea34440 -1 bluestore(td/osd-reuse-id/0) _read_fsid unparsable uuid 2024-09-18T15:56:12.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-reuse-id/0/keyring 2024-09-18T15:56:12.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:56:12.557 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T15:56:12.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T15:56:12.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-reuse-id/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:56:13.072 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T15:56:13.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T15:56:13.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=ef149c1c-850d-450f-9329-b83add5652f7 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/0 --osd-journal=td/osd-reuse-id/0/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.457476/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:56:13.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:56:13.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:56:13.073 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:56:13.092 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:56:13.091+0000 7efd38762440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:56:13.094 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:56:13.093+0000 7efd38762440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:56:13.095 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:56:13.094+0000 7efd38762440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:56:13.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T15:56:13.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:56:13.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:56:13.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:56:13.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:56:13.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:13.435 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:56:13.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:56:13.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:13.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:56:13.610 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:56:13.609+0000 7efd38762440 -1 Falling back to public interface 2024-09-18T15:56:13.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:14.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:14.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:14.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:56:14.810 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:56:14.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:14.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:56:15.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:16.091 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:56:16.090+0000 7efd38762440 -1 osd.0 0 log_to_monitors true 2024-09-18T15:56:16.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:16.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:16.223 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:56:16.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:56:16.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:16.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:56:16.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:17.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:17.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:17.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:56:17.685 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:56:17.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:17.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:56:18.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:19.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:19.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:19.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:56:19.114 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:56:19.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:19.114 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:56:19.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:20.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:20.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:20.520 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:56:20.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:56:20.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:20.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:56:20.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:21.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:21.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:21.899 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:56:21.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:56:21.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:21.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:56:22.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:23.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:23.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:23.279 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:56:23.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:56:23.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:23.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:56:23.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:24.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:24.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:24.810 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:56:24.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:56:24.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:24.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:56:25.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:26.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:26.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:26.210 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:56:26.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:56:26.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:26.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:56:26.598 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:27.599 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:27.599 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:27.600 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:56:27.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:56:27.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:27.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:56:28.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:29.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:29.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:29.605 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:56:29.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:56:29.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:29.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:56:30.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:31.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:31.192 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:56:31.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:31.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:56:31.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:31.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:56:31.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:32.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:32.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:32.967 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:56:32.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:56:32.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:32.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:56:34.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:35.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:35.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:35.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:56:35.165 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:56:35.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:35.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:56:36.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:37.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:37.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:37.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:56:37.274 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:56:37.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:37.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:56:38.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:39.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:39.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:39.245 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:56:39.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:56:39.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:39.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:56:39.683 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/458567,v1:127.0.0.1:6803/458567] [v2:127.0.0.1:6804/458567,v1:127.0.0.1:6805/458567] exists,up c87d6959-2eab-42d1-a750-c140730c97e5 2024-09-18T15:56:39.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:56:39.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:56:39.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:56:39.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:42: TEST_reuse_id: run_osd td/osd-reuse-id 1 2024-09-18T15:56:39.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-reuse-id 2024-09-18T15:56:39.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:56:39.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T15:56:39.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:56:39.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-reuse-id/1 2024-09-18T15:56:39.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ef149c1c-850d-450f-9329-b83add5652f7 --auth-supported=none --mon-host=127.0.0.1:7123 ' 2024-09-18T15:56:39.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:56:39.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:56:39.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:56:39.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-reuse-id/1' 2024-09-18T15:56:39.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-reuse-id/1/journal' 2024-09-18T15:56:39.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:56:39.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:56:39.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-reuse-id' 2024-09-18T15:56:39.689 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:56:39.689 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:56:39.689 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:56:39.689 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:56:39.689 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:56:39.689 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.457476 2024-09-18T15:56:39.689 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.457476/$cluster-$name.asok' 2024-09-18T15:56:39.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.457476/$cluster-$name.asok' 2024-09-18T15:56:39.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:56:39.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:56:39.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:56:39.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-reuse-id/$name.log' 2024-09-18T15:56:39.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-reuse-id/$name.pid' 2024-09-18T15:56:39.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:56:39.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:56:39.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:56:39.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:56:39.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:56:39.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:56:39.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-reuse-id/1 2024-09-18T15:56:39.690 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:56:39.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b64cf34d-de91-4de3-bb73-526e0c5f8b9f 2024-09-18T15:56:39.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b64cf34d-de91-4de3-bb73-526e0c5f8b9f' 2024-09-18T15:56:39.690 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 b64cf34d-de91-4de3-bb73-526e0c5f8b9f 2024-09-18T15:56:39.691 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:56:39.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA3+OpmhHmzKRAAYX6T+v/o1b12+tNajM2Dyg== 2024-09-18T15:56:39.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA3+OpmhHmzKRAAYX6T+v/o1b12+tNajM2Dyg=="}' 2024-09-18T15:56:39.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b64cf34d-de91-4de3-bb73-526e0c5f8b9f -i td/osd-reuse-id/1/new.json 2024-09-18T15:56:40.216 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:56:40.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-reuse-id/1/new.json 2024-09-18T15:56:40.228 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=ef149c1c-850d-450f-9329-b83add5652f7 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/1 --osd-journal=td/osd-reuse-id/1/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.457476/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA3+OpmhHmzKRAAYX6T+v/o1b12+tNajM2Dyg== --osd-uuid b64cf34d-de91-4de3-bb73-526e0c5f8b9f 2024-09-18T15:56:40.251 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:56:40.250+0000 7f4312c0d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:56:40.253 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:56:40.252+0000 7f4312c0d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:56:40.257 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:56:40.253+0000 7f4312c0d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:56:40.257 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:56:40.253+0000 7f4312c0d440 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-09-18T15:56:40.257 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:56:40.253+0000 7f4312c0d440 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-09-18T15:56:40.257 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:56:40.253+0000 7f4312c0d440 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-09-18T15:56:40.331 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:56:40.331+0000 7f4312c0d440 -1 bluestore(td/osd-reuse-id/1) _read_fsid unparsable uuid 2024-09-18T15:56:46.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-reuse-id/1/keyring 2024-09-18T15:56:46.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:56:46.163 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T15:56:46.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T15:56:46.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-reuse-id/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:56:46.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T15:56:46.674 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T15:56:46.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=ef149c1c-850d-450f-9329-b83add5652f7 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/1 --osd-journal=td/osd-reuse-id/1/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.457476/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:56:46.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:56:46.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:56:46.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:56:46.695 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:56:46.694+0000 7f6ed083f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:56:46.697 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:56:46.696+0000 7f6ed083f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:56:46.698 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:56:46.697+0000 7f6ed083f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:56:47.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T15:56:47.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:56:47.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:56:47.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:56:47.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:56:47.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:47.077 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:56:47.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:56:47.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:47.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:56:47.220 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:56:47.219+0000 7f6ed083f440 -1 Falling back to public interface 2024-09-18T15:56:47.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:48.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:48.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:48.443 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:56:48.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:56:48.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:48.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:56:48.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:49.617 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:56:49.616+0000 7f6ed083f440 -1 osd.1 0 log_to_monitors true 2024-09-18T15:56:49.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:49.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:49.829 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:56:49.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:56:49.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:49.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:56:50.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:51.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:51.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:51.304 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:56:51.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:56:51.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:51.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:56:51.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:52.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:52.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:52.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:56:52.834 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:56:52.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:52.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:56:53.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:54.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:54.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:54.245 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:56:54.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:56:54.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:54.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:56:54.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:55.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:55.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:55.638 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:56:55.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:56:55.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:55.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:56:56.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:57.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:57.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:57.061 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:56:57.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:56:57.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:57.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:56:57.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:56:58.642 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:56:58.642 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:56:58.642 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:56:58.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:56:58.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:56:58.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:56:59.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:00.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:00.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:00.049 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:57:00.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:57:00.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:00.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:57:00.469 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:01.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:01.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:01.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:57:01.471 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:57:01.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:01.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:57:02.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:03.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:03.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:03.070 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:57:03.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:57:03.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:03.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:57:03.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:04.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:04.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:04.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:57:04.796 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:57:04.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:04.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:57:05.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:06.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:06.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:06.365 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:57:06.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:57:06.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:06.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:57:06.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:07.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:07.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:07.837 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:57:07.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:57:07.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:07.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:57:08.642 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:09.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:09.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:09.644 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:57:09.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:57:09.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:09.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:57:10.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:11.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:11.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:11.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:57:11.259 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:57:11.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:11.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:57:11.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:12.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:12.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:12.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T15:57:12.646 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T15:57:12.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:12.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:57:13.030 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/460296,v1:127.0.0.1:6811/460296] [v2:127.0.0.1:6812/460296,v1:127.0.0.1:6813/460296] exists,up b64cf34d-de91-4de3-bb73-526e0c5f8b9f 2024-09-18T15:57:13.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:57:13.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:57:13.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:57:13.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:43: TEST_reuse_id: create_rbd_pool 2024-09-18T15:57:13.031 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T15:57:13.462 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'rbd' does not exist 2024-09-18T15:57:13.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T15:57:13.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T15:57:14.089 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'rbd' already exists 2024-09-18T15:57:14.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T15:57:15.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T15:57:16.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:44: TEST_reuse_id: wait_for_clean 2024-09-18T15:57:16.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:57:16.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:57:16.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:57:16.398 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:57:16.398 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:57:16.398 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:57:16.399 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:57:16.399 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:57:16.399 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:57:16.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:57:16.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:57:16.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:57:16.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:57:16.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:57:16.497 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:57:16.862 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:57:16.862 INFO:tasks.workunit.client.0.smithi047.stderr:1' 2024-09-18T15:57:16.862 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:57:16.862 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:57:16.863 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:57:16.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836491 2024-09-18T15:57:16.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836491 2024-09-18T15:57:16.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836491' 2024-09-18T15:57:16.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:57:16.997 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:57:17.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672969 2024-09-18T15:57:17.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672969 2024-09-18T15:57:17.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836491 1-42949672969' 2024-09-18T15:57:17.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:57:17.146 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836491 2024-09-18T15:57:17.146 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:57:17.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:57:17.148 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836491 2024-09-18T15:57:17.148 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:57:17.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836491 2024-09-18T15:57:17.149 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836491 2024-09-18T15:57:17.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2024-09-18T15:57:17.149 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:57:17.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836489 -lt 21474836491 2024-09-18T15:57:17.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:57:18.551 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:57:18.551 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:57:18.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836491 -lt 21474836491 2024-09-18T15:57:18.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:57:18.925 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672969 2024-09-18T15:57:18.925 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:57:18.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:57:18.927 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672969 2024-09-18T15:57:18.927 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:57:18.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672969 2024-09-18T15:57:18.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-09-18T15:57:18.928 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672969 2024-09-18T15:57:18.928 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:57:19.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672969 -lt 42949672969 2024-09-18T15:57:19.289 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:57:19.289 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:57:19.289 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:57:19.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T15:57:19.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:57:19.777 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:57:19.777 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:57:19.777 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:57:19.777 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:57:19.777 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:57:19.777 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:57:20.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T15:57:20.148 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:57:20.148 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:57:20.148 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:57:20.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T15:57:20.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:57:20.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:57:20.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:45: TEST_reuse_id: destroy_osd td/osd-reuse-id 1 2024-09-18T15:57:20.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:784: destroy_osd: local dir=td/osd-reuse-id 2024-09-18T15:57:20.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:785: destroy_osd: local id=1 2024-09-18T15:57:20.654 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:787: destroy_osd: ceph osd out osd.1 2024-09-18T15:57:21.247 INFO:tasks.workunit.client.0.smithi047.stderr:osd.1 is already out. 2024-09-18T15:57:21.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:788: destroy_osd: kill_daemons td/osd-reuse-id TERM osd.1 2024-09-18T15:57:21.258 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:57:21.258 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:57:21.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:57:21.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:57:21.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:57:21.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:57:21.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:789: destroy_osd: ceph osd down osd.1 2024-09-18T15:57:22.044 INFO:tasks.workunit.client.0.smithi047.stderr:osd.1 is already down. 2024-09-18T15:57:22.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:790: destroy_osd: ceph osd purge osd.1 --yes-i-really-mean-it 2024-09-18T15:57:22.529 INFO:tasks.workunit.client.0.smithi047.stderr:osd.1 does not exist 2024-09-18T15:57:22.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:791: destroy_osd: teardown td/osd-reuse-id/1 2024-09-18T15:57:22.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reuse-id/1 2024-09-18T15:57:22.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:57:22.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reuse-id/1 KILL 2024-09-18T15:57:22.541 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:57:22.541 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:57:22.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:57:22.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:57:22.541 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:57:22.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:57:22.542 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:57:22.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:57:22.543 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:57:22.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:57:22.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:57:22.545 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:57:22.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:57:22.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:57:22.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:57:22.546 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:57:22.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:57:22.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:57:22.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reuse-id/1 2024-09-18T15:57:22.551 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:57:22.552 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:57:22.552 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.457476 2024-09-18T15:57:22.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.457476 2024-09-18T15:57:22.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:57:22.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:57:22.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:792: destroy_osd: rm -fr td/osd-reuse-id/1 2024-09-18T15:57:22.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:46: TEST_reuse_id: run_osd td/osd-reuse-id 1 2024-09-18T15:57:22.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-reuse-id 2024-09-18T15:57:22.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:57:22.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T15:57:22.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:57:22.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-reuse-id/1 2024-09-18T15:57:22.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ef149c1c-850d-450f-9329-b83add5652f7 --auth-supported=none --mon-host=127.0.0.1:7123 ' 2024-09-18T15:57:22.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:57:22.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:57:22.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:57:22.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-reuse-id/1' 2024-09-18T15:57:22.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-reuse-id/1/journal' 2024-09-18T15:57:22.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:57:22.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:57:22.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-reuse-id' 2024-09-18T15:57:22.557 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:57:22.557 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:57:22.557 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:57:22.558 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:57:22.558 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:57:22.558 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.457476 2024-09-18T15:57:22.558 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.457476/$cluster-$name.asok' 2024-09-18T15:57:22.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.457476/$cluster-$name.asok' 2024-09-18T15:57:22.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:57:22.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:57:22.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:57:22.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-reuse-id/$name.log' 2024-09-18T15:57:22.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-reuse-id/$name.pid' 2024-09-18T15:57:22.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:57:22.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:57:22.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:57:22.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:57:22.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:57:22.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:57:22.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-reuse-id/1 2024-09-18T15:57:22.559 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:57:22.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f72df949-1e69-4eb5-9ddf-d0befecbfb6f 2024-09-18T15:57:22.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f72df949-1e69-4eb5-9ddf-d0befecbfb6f' 2024-09-18T15:57:22.560 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 f72df949-1e69-4eb5-9ddf-d0befecbfb6f 2024-09-18T15:57:22.560 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:57:22.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBi+OpmaZ7uIRAAMBVqEkxWgT3Uop3gwZdl0w== 2024-09-18T15:57:22.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBi+OpmaZ7uIRAAMBVqEkxWgT3Uop3gwZdl0w=="}' 2024-09-18T15:57:22.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f72df949-1e69-4eb5-9ddf-d0befecbfb6f -i td/osd-reuse-id/1/new.json 2024-09-18T15:57:23.098 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:57:23.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-reuse-id/1/new.json 2024-09-18T15:57:23.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=ef149c1c-850d-450f-9329-b83add5652f7 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/1 --osd-journal=td/osd-reuse-id/1/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.457476/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBi+OpmaZ7uIRAAMBVqEkxWgT3Uop3gwZdl0w== --osd-uuid f72df949-1e69-4eb5-9ddf-d0befecbfb6f 2024-09-18T15:57:23.133 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:57:23.133+0000 7fdf3a720440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:57:23.135 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:57:23.135+0000 7fdf3a720440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:57:23.137 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:57:23.136+0000 7fdf3a720440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:57:23.137 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:57:23.136+0000 7fdf3a720440 -1 asok(0x55ecb207a000) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/tmp/ceph-asok.457476/ceph-osd.1.asok': (2) No such file or directory 2024-09-18T15:57:23.137 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:57:23.136+0000 7fdf3a720440 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-09-18T15:57:23.137 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:57:23.136+0000 7fdf3a720440 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-09-18T15:57:23.137 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:57:23.136+0000 7fdf3a720440 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-09-18T15:57:23.189 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:57:23.188+0000 7fdf3a720440 -1 bluestore(td/osd-reuse-id/1) _read_fsid unparsable uuid 2024-09-18T15:57:28.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-reuse-id/1/keyring 2024-09-18T15:57:28.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:57:28.393 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T15:57:28.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T15:57:28.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-reuse-id/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:57:28.898 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T15:57:28.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T15:57:28.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=ef149c1c-850d-450f-9329-b83add5652f7 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/1 --osd-journal=td/osd-reuse-id/1/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.457476/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:57:28.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:57:28.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:57:28.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:57:28.919 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:57:28.918+0000 7f74e360c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:57:28.943 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:57:28.942+0000 7f74e360c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:57:28.944 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:57:28.943+0000 7f74e360c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:57:28.945 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:57:28.944+0000 7f74e360c440 -1 asok(0x5637d5672000) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/tmp/ceph-asok.457476/ceph-osd.1.asok': (2) No such file or directory 2024-09-18T15:57:29.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T15:57:29.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:57:29.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:57:29.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:57:29.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:57:29.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:29.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:57:29.325 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:57:29.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:29.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:57:29.480 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:57:29.479+0000 7f74e360c440 -1 Falling back to public interface 2024-09-18T15:57:29.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:30.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:30.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:30.726 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:57:30.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:57:30.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:30.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:57:31.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:32.058 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:57:32.057+0000 7f74e360c440 -1 osd.1 0 log_to_monitors true 2024-09-18T15:57:32.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:32.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:32.122 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:57:32.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:57:32.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:32.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:57:32.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:33.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:33.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:33.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:57:33.635 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:57:33.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:33.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:57:34.045 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/462486,v1:127.0.0.1:6811/462486] [v2:127.0.0.1:6812/462486,v1:127.0.0.1:6813/462486] exists,up f72df949-1e69-4eb5-9ddf-d0befecbfb6f 2024-09-18T15:57:34.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:57:34.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:57:34.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:57:34.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:32: run: teardown td/osd-reuse-id 2024-09-18T15:57:34.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reuse-id 2024-09-18T15:57:34.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:57:34.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reuse-id KILL 2024-09-18T15:57:34.047 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:57:34.047 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:57:34.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:57:34.047 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:57:34.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:57:34.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:57:34.153 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:57:34.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:57:34.154 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:57:34.155 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:57:34.155 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:57:34.155 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:57:34.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:57:34.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:57:34.157 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:57:34.157 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:57:34.157 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:57:34.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:57:34.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reuse-id 2024-09-18T15:57:34.166 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:57:34.167 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:57:34.167 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.457476 2024-09-18T15:57:34.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.457476 2024-09-18T15:57:34.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:57:34.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:57:34.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T15:57:34.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/osd-reuse-id 0 2024-09-18T15:57:34.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reuse-id 2024-09-18T15:57:34.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T15:57:34.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reuse-id KILL 2024-09-18T15:57:34.169 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:57:34.169 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:57:34.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:57:34.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:57:34.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:57:34.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:57:34.170 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:57:34.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:57:34.171 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:57:34.172 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:57:34.172 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:57:34.172 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:57:34.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:57:34.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:57:34.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:57:34.174 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:57:34.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:57:34.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T15:57:34.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reuse-id 2024-09-18T15:57:34.176 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:57:34.176 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:57:34.176 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.457476 2024-09-18T15:57:34.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.457476 2024-09-18T15:57:34.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:57:34.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:57:34.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T15:57:34.181 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T15:57:34.182 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T15:57:34.246 INFO:tasks.workunit:Running workunit osd/pg-split-merge.sh... 2024-09-18T15:57:34.247 DEBUG:teuthology.orchestra.run.smithi047:workunit test osd/pg-split-merge.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh 2024-09-18T15:57:34.309 INFO:tasks.workunit.client.0.smithi047.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T15:57:34.314 INFO:tasks.workunit.client.0.smithi047.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T15:57:34.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T15:57:34.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T15:57:34.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T15:57:34.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T15:57:34.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T15:57:34.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T15:57:34.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T15:57:34.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T15:57:34.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/pg-split-merge 2024-09-18T15:57:34.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:5: run: local dir=td/pg-split-merge 2024-09-18T15:57:34.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:6: run: shift 2024-09-18T15:57:34.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:8: run: export CEPH_MON=127.0.0.1:7147 2024-09-18T15:57:34.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:8: run: CEPH_MON=127.0.0.1:7147 2024-09-18T15:57:34.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:9: run: export CEPH_ARGS 2024-09-18T15:57:34.315 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:10: run: uuidgen 2024-09-18T15:57:34.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:10: run: CEPH_ARGS+='--fsid=1a671e43-8866-44a5-abe3-c1e1f4ff765b --auth-supported=none ' 2024-09-18T15:57:34.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:11: run: CEPH_ARGS+='--mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-18T15:57:34.316 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:13: run: set 2024-09-18T15:57:34.316 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:13: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-18T15:57:34.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:13: run: local 'funcs=TEST_a_merge_empty 2024-09-18T15:57:34.318 INFO:tasks.workunit.client.0.smithi047.stderr:TEST_import_after_merge_and_gap 2024-09-18T15:57:34.318 INFO:tasks.workunit.client.0.smithi047.stderr:TEST_import_after_split' 2024-09-18T15:57:34.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:14: run: for func in $funcs 2024-09-18T15:57:34.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:15: run: setup td/pg-split-merge 2024-09-18T15:57:34.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/pg-split-merge 2024-09-18T15:57:34.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/pg-split-merge 2024-09-18T15:57:34.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-09-18T15:57:34.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T15:57:34.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-09-18T15:57:34.319 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:57:34.319 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:57:34.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:57:34.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:57:34.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:57:34.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:57:34.319 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T15:57:34.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T15:57:34.320 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T15:57:34.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T15:57:34.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T15:57:34.321 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T15:57:34.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:57:34.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T15:57:34.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T15:57:34.322 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T15:57:34.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T15:57:34.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T15:57:34.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-09-18T15:57:34.325 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T15:57:34.325 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:57:34.325 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.463129 2024-09-18T15:57:34.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.463129 2024-09-18T15:57:34.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T15:57:34.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T15:57:34.326 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/pg-split-merge 2024-09-18T15:57:34.327 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T15:57:34.328 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:57:34.328 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.463129 2024-09-18T15:57:34.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.463129 2024-09-18T15:57:34.329 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T15:57:34.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T15:57:34.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T15:57:34.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T15:57:34.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/pg-split-merge 1' TERM HUP INT 2024-09-18T15:57:34.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:16: run: TEST_a_merge_empty td/pg-split-merge 2024-09-18T15:57:34.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:22: TEST_a_merge_empty: local dir=td/pg-split-merge 2024-09-18T15:57:34.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:24: TEST_a_merge_empty: run_mon td/pg-split-merge a --osd_pool_default_size=3 2024-09-18T15:57:34.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/pg-split-merge 2024-09-18T15:57:34.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T15:57:34.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T15:57:34.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T15:57:34.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/pg-split-merge/a 2024-09-18T15:57:34.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/pg-split-merge/a --run-dir=td/pg-split-merge --osd_pool_default_size=3 2024-09-18T15:57:34.683 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T15:57:34.683 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:57:34.683 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:57:34.683 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:57:34.683 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:57:34.683 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.463129 2024-09-18T15:57:34.683 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.463129/$cluster-$name.asok' 2024-09-18T15:57:34.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/pg-split-merge/a '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.463129/$cluster-$name.asok' --mon-cluster-log-file=td/pg-split-merge/log --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-18T15:57:34.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T15:57:34.926 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T15:57:34.926 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:57:34.927 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:57:34.927 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T15:57:34.927 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T15:57:34.927 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:57:34.927 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:57:34.927 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:57:34.927 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:57:34.927 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:57:34.927 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.463129 2024-09-18T15:57:34.927 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.463129/ceph-mon.a.asok 2024-09-18T15:57:34.928 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:57:34.928 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.463129/ceph-mon.a.asok config get fsid 2024-09-18T15:57:35.005 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T15:57:35.005 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T15:57:35.005 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T15:57:35.005 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T15:57:35.005 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T15:57:35.005 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T15:57:35.006 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T15:57:35.006 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T15:57:35.006 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T15:57:35.006 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:57:35.006 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.463129 2024-09-18T15:57:35.006 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.463129/ceph-mon.a.asok 2024-09-18T15:57:35.006 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T15:57:35.006 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.463129/ceph-mon.a.asok config get mon_host 2024-09-18T15:57:35.080 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:25: TEST_a_merge_empty: run_mgr td/pg-split-merge x 2024-09-18T15:57:35.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/pg-split-merge 2024-09-18T15:57:35.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T15:57:35.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T15:57:35.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T15:57:35.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/pg-split-merge/x 2024-09-18T15:57:35.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T15:57:35.938 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T15:57:35.939 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:57:35.939 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:57:35.939 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:57:35.939 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:57:35.939 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.463129 2024-09-18T15:57:35.939 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.463129/$cluster-$name.asok' 2024-09-18T15:57:35.939 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:57:35.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/pg-split-merge/x '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.463129/$cluster-$name.asok' --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T15:57:35.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:26: TEST_a_merge_empty: run_osd td/pg-split-merge 0 2024-09-18T15:57:35.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/pg-split-merge 2024-09-18T15:57:35.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:57:35.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T15:57:35.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:57:35.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/pg-split-merge/0 2024-09-18T15:57:35.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a671e43-8866-44a5-abe3-c1e1f4ff765b --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-18T15:57:35.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:57:35.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:57:35.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:57:35.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-09-18T15:57:35.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-09-18T15:57:35.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:57:35.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:57:35.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-09-18T15:57:35.968 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:57:35.968 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:57:35.968 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:57:35.968 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:57:35.968 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:57:35.968 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.463129 2024-09-18T15:57:35.968 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.463129/$cluster-$name.asok' 2024-09-18T15:57:35.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.463129/$cluster-$name.asok' 2024-09-18T15:57:35.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:57:35.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:57:35.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:57:35.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-09-18T15:57:35.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-09-18T15:57:35.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:57:35.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:57:35.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:57:35.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:57:35.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:57:35.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:57:35.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/pg-split-merge/0 2024-09-18T15:57:35.969 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:57:35.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8866341f-2121-4db5-b7fb-4ee124bad37d 2024-09-18T15:57:35.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8866341f-2121-4db5-b7fb-4ee124bad37d' 2024-09-18T15:57:35.970 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 8866341f-2121-4db5-b7fb-4ee124bad37d 2024-09-18T15:57:35.970 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:57:35.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBv+OpmyVSMOhAAyQfKGY3iYCOxGMEW5LDtkA== 2024-09-18T15:57:35.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBv+OpmyVSMOhAAyQfKGY3iYCOxGMEW5LDtkA=="}' 2024-09-18T15:57:35.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8866341f-2121-4db5-b7fb-4ee124bad37d -i td/pg-split-merge/0/new.json 2024-09-18T15:57:36.282 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:57:36.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/pg-split-merge/0/new.json 2024-09-18T15:57:36.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1a671e43-8866-44a5-abe3-c1e1f4ff765b --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.463129/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBv+OpmyVSMOhAAyQfKGY3iYCOxGMEW5LDtkA== --osd-uuid 8866341f-2121-4db5-b7fb-4ee124bad37d 2024-09-18T15:57:36.318 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:57:36.318+0000 7f9f3a51c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:57:36.326 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:57:36.326+0000 7f9f3a51c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:57:36.327 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:57:36.327+0000 7f9f3a51c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:57:36.328 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:57:36.327+0000 7f9f3a51c440 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-09-18T15:57:36.328 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:57:36.327+0000 7f9f3a51c440 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-09-18T15:57:36.328 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:57:36.327+0000 7f9f3a51c440 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-09-18T15:57:36.367 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:57:36.366+0000 7f9f3a51c440 -1 bluestore(td/pg-split-merge/0) _read_fsid unparsable uuid 2024-09-18T15:57:41.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/pg-split-merge/0/keyring 2024-09-18T15:57:41.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:57:41.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T15:57:41.684 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T15:57:41.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/pg-split-merge/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:57:42.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T15:57:42.200 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T15:57:42.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1a671e43-8866-44a5-abe3-c1e1f4ff765b --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.463129/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:57:42.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:57:42.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:57:42.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:57:42.224 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:57:42.223+0000 7f5f5100a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:57:42.248 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:57:42.247+0000 7f5f5100a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:57:42.249 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:57:42.248+0000 7f5f5100a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:57:42.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T15:57:42.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:57:42.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T15:57:42.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:57:42.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:57:42.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:42.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:57:42.610 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:57:42.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:42.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:57:42.782 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:57:42.781+0000 7f5f5100a440 -1 Falling back to public interface 2024-09-18T15:57:43.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:44.021 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:44.022 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:44.022 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:57:44.022 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:57:44.022 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:44.022 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:57:44.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:45.177 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:57:45.176+0000 7f5f5100a440 -1 osd.0 0 log_to_monitors true 2024-09-18T15:57:45.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:45.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:45.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:57:45.449 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:57:45.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:45.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:57:45.918 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:46.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:46.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:46.920 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:57:46.920 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:57:46.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:46.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:57:47.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:48.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:48.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:48.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:57:48.342 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:57:48.343 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:48.343 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:57:48.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:49.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:49.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:49.900 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:57:49.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:57:49.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:49.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:57:50.281 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:51.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:51.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:51.283 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:57:51.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:57:51.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:51.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:57:51.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:52.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:52.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:52.701 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:57:52.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:57:52.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:52.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:57:53.194 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:54.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:54.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:54.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:57:54.196 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:57:54.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:54.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:57:54.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:55.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:55.606 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:55.606 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:57:55.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:57:55.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:55.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:57:56.021 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:57.022 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:57.022 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:57.022 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:57:57.022 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:57:57.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:57.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:57:57.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:57:58.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:57:58.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:57:58.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:57:58.631 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:57:58.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:57:58.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:57:59.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:00.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:00.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:00.618 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:58:00.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:58:00.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:00.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:58:01.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:02.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:02.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:02.296 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:58:02.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:58:02.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:02.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:58:03.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:04.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:04.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:04.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:58:04.068 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:58:04.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:04.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:58:04.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:05.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:05.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:05.689 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:58:05.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:58:05.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:05.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:58:06.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:07.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:07.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:07.088 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:58:07.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:58:07.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:07.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T15:58:07.525 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/464220,v1:127.0.0.1:6803/464220] [v2:127.0.0.1:6804/464220,v1:127.0.0.1:6805/464220] exists,up 8866341f-2121-4db5-b7fb-4ee124bad37d 2024-09-18T15:58:07.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:58:07.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:58:07.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:58:07.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:27: TEST_a_merge_empty: run_osd td/pg-split-merge 1 2024-09-18T15:58:07.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/pg-split-merge 2024-09-18T15:58:07.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:58:07.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T15:58:07.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:58:07.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/pg-split-merge/1 2024-09-18T15:58:07.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a671e43-8866-44a5-abe3-c1e1f4ff765b --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-18T15:58:07.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:58:07.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:58:07.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:58:07.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/1' 2024-09-18T15:58:07.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/1/journal' 2024-09-18T15:58:07.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:58:07.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:58:07.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-09-18T15:58:07.531 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:58:07.531 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:58:07.531 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:58:07.531 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:58:07.531 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:58:07.531 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.463129 2024-09-18T15:58:07.531 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.463129/$cluster-$name.asok' 2024-09-18T15:58:07.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.463129/$cluster-$name.asok' 2024-09-18T15:58:07.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:58:07.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:58:07.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:58:07.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-09-18T15:58:07.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-09-18T15:58:07.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:58:07.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:58:07.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:58:07.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:58:07.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:58:07.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:58:07.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/pg-split-merge/1 2024-09-18T15:58:07.533 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:58:07.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=35114e79-89ba-4a6a-bf3e-39d529bd5d3c 2024-09-18T15:58:07.533 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 35114e79-89ba-4a6a-bf3e-39d529bd5d3c 2024-09-18T15:58:07.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 35114e79-89ba-4a6a-bf3e-39d529bd5d3c' 2024-09-18T15:58:07.533 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:58:07.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCP+OpmQm9mIBAAg9UWIiGzSL5Iacyg3ZQ1wg== 2024-09-18T15:58:07.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCP+OpmQm9mIBAAg9UWIiGzSL5Iacyg3ZQ1wg=="}' 2024-09-18T15:58:07.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 35114e79-89ba-4a6a-bf3e-39d529bd5d3c -i td/pg-split-merge/1/new.json 2024-09-18T15:58:08.091 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:58:08.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/pg-split-merge/1/new.json 2024-09-18T15:58:08.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1a671e43-8866-44a5-abe3-c1e1f4ff765b --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/1 --osd-journal=td/pg-split-merge/1/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.463129/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCP+OpmQm9mIBAAg9UWIiGzSL5Iacyg3ZQ1wg== --osd-uuid 35114e79-89ba-4a6a-bf3e-39d529bd5d3c 2024-09-18T15:58:08.129 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:58:08.128+0000 7f3a48d25440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:58:08.131 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:58:08.130+0000 7f3a48d25440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:58:08.132 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:58:08.131+0000 7f3a48d25440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:58:08.132 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:58:08.131+0000 7f3a48d25440 -1 bluestore(td/pg-split-merge/1/block) _read_bdev_label failed to open td/pg-split-merge/1/block: (2) No such file or directory 2024-09-18T15:58:08.132 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:58:08.131+0000 7f3a48d25440 -1 bluestore(td/pg-split-merge/1/block) _read_bdev_label failed to open td/pg-split-merge/1/block: (2) No such file or directory 2024-09-18T15:58:08.132 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:58:08.131+0000 7f3a48d25440 -1 bluestore(td/pg-split-merge/1/block) _read_bdev_label failed to open td/pg-split-merge/1/block: (2) No such file or directory 2024-09-18T15:58:08.198 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:58:08.197+0000 7f3a48d25440 -1 bluestore(td/pg-split-merge/1) _read_fsid unparsable uuid 2024-09-18T15:58:13.776 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/pg-split-merge/1/keyring 2024-09-18T15:58:13.777 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:58:13.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T15:58:13.778 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T15:58:13.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/pg-split-merge/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:58:14.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T15:58:14.288 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T15:58:14.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1a671e43-8866-44a5-abe3-c1e1f4ff765b --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/1 --osd-journal=td/pg-split-merge/1/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.463129/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:58:14.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:58:14.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:58:14.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:58:14.312 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:58:14.311+0000 7f22d557b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:58:14.314 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:58:14.314+0000 7f22d557b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:58:14.315 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:58:14.315+0000 7f22d557b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:58:14.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T15:58:14.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:58:14.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T15:58:14.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:58:14.663 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:58:14.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:58:14.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:14.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:58:14.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:14.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:58:14.832 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:58:14.831+0000 7f22d557b440 -1 Falling back to public interface 2024-09-18T15:58:15.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:16.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:16.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:16.035 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:58:16.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:58:16.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:16.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:58:16.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:17.292 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:58:17.291+0000 7f22d557b440 -1 osd.1 0 log_to_monitors true 2024-09-18T15:58:17.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:17.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:17.430 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:58:17.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:58:17.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:17.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:58:17.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:18.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:18.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:18.880 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:58:18.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:58:18.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:18.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:58:19.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:20.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:20.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:20.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:58:20.383 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:58:20.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:20.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:58:20.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:21.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:21.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:21.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:58:21.881 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:58:21.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:21.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:58:22.355 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:23.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:23.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:23.357 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:58:23.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:58:23.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:23.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:58:23.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:24.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:24.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:24.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:58:24.736 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:58:24.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:24.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:58:25.182 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:26.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:26.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:26.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:58:26.184 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:58:26.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:26.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:58:26.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:27.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:27.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:27.697 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:58:27.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:58:27.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:27.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:58:28.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:29.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:29.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:29.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:58:29.189 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:58:29.189 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:29.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:58:30.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:31.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:31.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:31.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:58:31.005 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:58:31.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:31.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:58:31.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:32.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:32.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:32.507 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:58:32.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:58:32.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:32.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:58:33.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:34.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:34.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:34.207 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:58:34.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:58:34.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:34.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:58:35.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:36.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:36.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:36.279 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:58:36.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:58:36.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:36.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:58:36.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:37.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:37.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:37.908 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:58:37.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:58:37.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:37.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:58:38.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:39.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:39.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:39.774 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:58:39.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:58:39.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:39.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T15:58:40.157 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/465949,v1:127.0.0.1:6811/465949] [v2:127.0.0.1:6812/465949,v1:127.0.0.1:6813/465949] exists,up 35114e79-89ba-4a6a-bf3e-39d529bd5d3c 2024-09-18T15:58:40.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:58:40.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:58:40.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:58:40.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:28: TEST_a_merge_empty: run_osd td/pg-split-merge 2 2024-09-18T15:58:40.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/pg-split-merge 2024-09-18T15:58:40.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T15:58:40.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T15:58:40.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T15:58:40.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/pg-split-merge/2 2024-09-18T15:58:40.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a671e43-8866-44a5-abe3-c1e1f4ff765b --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-18T15:58:40.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:58:40.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:58:40.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:58:40.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/2' 2024-09-18T15:58:40.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/2/journal' 2024-09-18T15:58:40.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T15:58:40.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T15:58:40.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-09-18T15:58:40.162 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T15:58:40.162 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:58:40.162 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:58:40.162 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:58:40.163 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:58:40.163 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.463129 2024-09-18T15:58:40.163 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.463129/$cluster-$name.asok' 2024-09-18T15:58:40.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.463129/$cluster-$name.asok' 2024-09-18T15:58:40.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:58:40.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T15:58:40.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T15:58:40.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-09-18T15:58:40.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-09-18T15:58:40.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:58:40.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:58:40.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:58:40.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:58:40.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T15:58:40.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T15:58:40.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/pg-split-merge/2 2024-09-18T15:58:40.164 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T15:58:40.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3047ef09-2eed-40bf-9c34-62adf1977248 2024-09-18T15:58:40.164 INFO:tasks.workunit.client.0.smithi047.stdout:add osd2 3047ef09-2eed-40bf-9c34-62adf1977248 2024-09-18T15:58:40.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 3047ef09-2eed-40bf-9c34-62adf1977248' 2024-09-18T15:58:40.165 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T15:58:40.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCw+OpmN9yEChAARx9KsyJtnFfZ8nhzHyTc8Q== 2024-09-18T15:58:40.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCw+OpmN9yEChAARx9KsyJtnFfZ8nhzHyTc8Q=="}' 2024-09-18T15:58:40.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3047ef09-2eed-40bf-9c34-62adf1977248 -i td/pg-split-merge/2/new.json 2024-09-18T15:58:40.749 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:58:40.760 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/pg-split-merge/2/new.json 2024-09-18T15:58:40.761 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1a671e43-8866-44a5-abe3-c1e1f4ff765b --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/2 --osd-journal=td/pg-split-merge/2/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.463129/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCw+OpmN9yEChAARx9KsyJtnFfZ8nhzHyTc8Q== --osd-uuid 3047ef09-2eed-40bf-9c34-62adf1977248 2024-09-18T15:58:40.787 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:58:40.786+0000 7f7fe391f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:58:40.789 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:58:40.788+0000 7f7fe391f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:58:40.790 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:58:40.789+0000 7f7fe391f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:58:40.790 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:58:40.789+0000 7f7fe391f440 -1 bluestore(td/pg-split-merge/2/block) _read_bdev_label failed to open td/pg-split-merge/2/block: (2) No such file or directory 2024-09-18T15:58:40.790 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:58:40.789+0000 7f7fe391f440 -1 bluestore(td/pg-split-merge/2/block) _read_bdev_label failed to open td/pg-split-merge/2/block: (2) No such file or directory 2024-09-18T15:58:40.790 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:58:40.789+0000 7f7fe391f440 -1 bluestore(td/pg-split-merge/2/block) _read_bdev_label failed to open td/pg-split-merge/2/block: (2) No such file or directory 2024-09-18T15:58:40.829 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:58:40.828+0000 7f7fe391f440 -1 bluestore(td/pg-split-merge/2) _read_fsid unparsable uuid 2024-09-18T15:58:46.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/pg-split-merge/2/keyring 2024-09-18T15:58:46.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T15:58:46.807 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd2 key to auth repository 2024-09-18T15:58:46.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T15:58:46.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/pg-split-merge/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T15:58:47.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T15:58:47.324 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T15:58:47.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1a671e43-8866-44a5-abe3-c1e1f4ff765b --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/2 --osd-journal=td/pg-split-merge/2/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.463129/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:58:47.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T15:58:47.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T15:58:47.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T15:58:47.348 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:58:47.347+0000 7f4e23639440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:58:47.380 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:58:47.379+0000 7f4e23639440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:58:47.381 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:58:47.380+0000 7f4e23639440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:58:47.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T15:58:47.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:58:47.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:58:47.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:58:47.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:58:47.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:47.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:58:47.692 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:58:47.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:47.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:58:47.910 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:58:47.909+0000 7f4e23639440 -1 Falling back to public interface 2024-09-18T15:58:48.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:49.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:49.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:49.088 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:58:49.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:58:49.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:49.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:58:49.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:50.452 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:58:50.451+0000 7f4e23639440 -1 osd.2 0 log_to_monitors true 2024-09-18T15:58:50.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:50.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:50.505 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:58:50.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:58:50.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:50.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:58:51.096 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:52.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:52.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:52.098 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T15:58:52.098 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T15:58:52.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:52.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:58:52.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:53.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:53.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:53.553 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T15:58:53.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T15:58:53.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:53.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:58:53.947 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:54.949 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:54.949 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:54.949 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T15:58:54.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T15:58:54.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:54.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:58:55.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:56.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:56.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:56.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T15:58:56.331 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T15:58:56.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:56.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:58:56.760 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:57.761 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:57.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:57.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T15:58:57.762 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T15:58:57.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:57.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:58:58.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:58:59.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:58:59.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:58:59.169 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T15:58:59.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T15:58:59.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:58:59.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:58:59.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:59:00.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:59:00.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:59:00.548 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T15:59:00.549 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T15:59:00.549 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:59:00.549 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:59:00.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:59:01.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:59:01.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:59:01.954 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T15:59:01.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T15:59:01.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:59:01.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:59:02.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:59:03.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:59:03.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:59:03.711 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T15:59:03.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T15:59:03.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:59:03.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:59:04.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:59:05.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:59:05.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:59:05.629 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T15:59:05.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T15:59:05.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:59:05.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:59:06.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:59:07.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:59:07.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:59:07.167 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T15:59:07.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T15:59:07.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:59:07.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:59:07.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:59:08.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:59:08.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:59:08.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T15:59:08.810 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T15:59:08.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:59:08.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:59:09.552 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:59:10.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:59:10.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:59:10.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T15:59:10.554 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T15:59:10.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:59:10.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:59:11.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:59:12.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:59:12.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:59:12.482 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T15:59:12.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T15:59:12.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:59:12.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:59:12.950 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/467679,v1:127.0.0.1:6819/467679] [v2:127.0.0.1:6820/467679,v1:127.0.0.1:6821/467679] exists,up 3047ef09-2eed-40bf-9c34-62adf1977248 2024-09-18T15:59:12.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:59:12.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:59:12.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:59:12.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:30: TEST_a_merge_empty: ceph osd pool create foo 2 2024-09-18T15:59:13.594 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'foo' already exists 2024-09-18T15:59:13.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:31: TEST_a_merge_empty: ceph osd pool set foo pgp_num 1 2024-09-18T15:59:14.533 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 1 pgp_num to 1 2024-09-18T15:59:14.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:33: TEST_a_merge_empty: wait_for_clean 2024-09-18T15:59:14.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:59:14.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:59:14.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:59:14.545 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:59:14.545 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:59:14.545 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:59:14.545 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:59:14.545 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:59:14.545 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:59:14.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:59:14.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:59:14.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:59:14.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:59:14.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:59:14.634 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:59:15.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:59:15.274 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:59:15.274 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T15:59:15.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:59:15.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:59:15.274 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:59:15.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836498 2024-09-18T15:59:15.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836498 2024-09-18T15:59:15.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836498' 2024-09-18T15:59:15.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:59:15.423 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:59:15.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672972 2024-09-18T15:59:15.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672972 2024-09-18T15:59:15.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836498 1-42949672972' 2024-09-18T15:59:15.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:59:15.533 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:59:15.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509445 2024-09-18T15:59:15.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509445 2024-09-18T15:59:15.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836498 1-42949672972 2-64424509445' 2024-09-18T15:59:15.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:59:15.644 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836498 2024-09-18T15:59:15.644 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:59:15.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:59:15.645 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836498 2024-09-18T15:59:15.645 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:59:15.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836498 2024-09-18T15:59:15.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-09-18T15:59:15.647 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836498 2024-09-18T15:59:15.647 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:59:16.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836498 -lt 21474836498 2024-09-18T15:59:16.011 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:59:16.012 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672972 2024-09-18T15:59:16.012 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:59:16.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:59:16.013 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672972 2024-09-18T15:59:16.013 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:59:16.014 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672972 2024-09-18T15:59:16.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672972 2024-09-18T15:59:16.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-09-18T15:59:16.015 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:59:16.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672972 -lt 42949672972 2024-09-18T15:59:16.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:59:16.383 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509445 2024-09-18T15:59:16.383 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:59:16.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:59:16.385 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509445 2024-09-18T15:59:16.385 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:59:16.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509445 2024-09-18T15:59:16.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-09-18T15:59:16.386 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509445 2024-09-18T15:59:16.386 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:59:16.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509443 -lt 64424509445 2024-09-18T15:59:16.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:59:17.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:59:17.822 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:59:18.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509445 -lt 64424509445 2024-09-18T15:59:18.323 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:59:18.323 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:59:18.323 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:59:18.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-09-18T15:59:18.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:59:18.869 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:59:18.869 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:59:18.869 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:59:18.869 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:59:18.869 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:59:18.869 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:59:19.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:59:19.225 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:59:19.225 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:59:19.225 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:59:19.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-09-18T15:59:19.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:59:19.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:59:19.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:37: TEST_a_merge_empty: rados -p foo put foo1 /etc/passwd 2024-09-18T15:59:19.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:38: TEST_a_merge_empty: rados -p foo put foo2 /etc/passwd 2024-09-18T15:59:20.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:39: TEST_a_merge_empty: rados -p foo put foo3 /etc/passwd 2024-09-18T15:59:20.172 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:40: TEST_a_merge_empty: rados -p foo put foo4 /etc/passwd 2024-09-18T15:59:20.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:42: TEST_a_merge_empty: rados -p foo put foo5 /etc/passwd 2024-09-18T15:59:20.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:43: TEST_a_merge_empty: rados -p foo put foo6 /etc/passwd 2024-09-18T15:59:20.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:44: TEST_a_merge_empty: rados -p foo put foo8 /etc/passwd 2024-09-18T15:59:20.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:45: TEST_a_merge_empty: rados -p foo put foo10 /etc/passwd 2024-09-18T15:59:20.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:46: TEST_a_merge_empty: rados -p foo put foo11 /etc/passwd 2024-09-18T15:59:21.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:47: TEST_a_merge_empty: rados -p foo put foo12 /etc/passwd 2024-09-18T15:59:21.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:48: TEST_a_merge_empty: rados -p foo put foo16 /etc/passwd 2024-09-18T15:59:21.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:50: TEST_a_merge_empty: wait_for_clean 2024-09-18T15:59:21.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:59:21.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:59:21.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:59:21.331 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:59:21.331 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:59:21.331 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:59:21.332 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:59:21.332 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:59:21.332 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:59:21.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:59:21.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:59:21.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:59:21.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:59:21.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:59:21.425 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:59:21.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:59:21.841 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:59:21.841 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T15:59:21.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:59:21.842 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:59:21.842 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:59:21.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836501 2024-09-18T15:59:21.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836501 2024-09-18T15:59:21.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836501' 2024-09-18T15:59:21.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:59:21.955 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:59:22.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672975 2024-09-18T15:59:22.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672975 2024-09-18T15:59:22.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836501 1-42949672975' 2024-09-18T15:59:22.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:59:22.065 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:59:22.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509448 2024-09-18T15:59:22.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509448 2024-09-18T15:59:22.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836501 1-42949672975 2-64424509448' 2024-09-18T15:59:22.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:59:22.175 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836501 2024-09-18T15:59:22.175 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:59:22.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:59:22.177 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836501 2024-09-18T15:59:22.177 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:59:22.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836501 2024-09-18T15:59:22.178 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836501 2024-09-18T15:59:22.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-09-18T15:59:22.178 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:59:22.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836499 -lt 21474836501 2024-09-18T15:59:22.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:59:23.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:59:23.545 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:59:23.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836502 -lt 21474836501 2024-09-18T15:59:23.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:59:23.936 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672975 2024-09-18T15:59:23.936 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:59:23.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:59:23.938 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672975 2024-09-18T15:59:23.938 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:59:23.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672975 2024-09-18T15:59:23.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-09-18T15:59:23.939 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672975 2024-09-18T15:59:23.939 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:59:24.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672975 -lt 42949672975 2024-09-18T15:59:24.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:59:24.354 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509448 2024-09-18T15:59:24.354 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:59:24.355 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:59:24.356 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509448 2024-09-18T15:59:24.356 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:59:24.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509448 2024-09-18T15:59:24.357 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509448 2024-09-18T15:59:24.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-09-18T15:59:24.358 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:59:24.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509448 -lt 64424509448 2024-09-18T15:59:24.806 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:59:24.806 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:59:24.806 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:59:25.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-09-18T15:59:25.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:59:25.302 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:59:25.302 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:59:25.302 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:59:25.303 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:59:25.303 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:59:25.303 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:59:25.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:59:25.698 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:59:25.698 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:59:25.698 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:59:26.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-09-18T15:59:26.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:59:26.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:59:26.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:52: TEST_a_merge_empty: ceph tell osd.1 config set osd_debug_no_purge_strays true 2024-09-18T15:59:26.295 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:59:26.295 INFO:tasks.workunit.client.0.smithi047.stdout: "success": "" 2024-09-18T15:59:26.295 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:59:26.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:53: TEST_a_merge_empty: ceph osd pool set foo size 2 2024-09-18T15:59:27.116 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 1 size to 2 2024-09-18T15:59:27.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:54: TEST_a_merge_empty: wait_for_clean 2024-09-18T15:59:27.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:59:27.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:59:27.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:59:27.129 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:59:27.129 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:59:27.129 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:59:27.129 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:59:27.129 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:59:27.129 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:59:27.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:59:27.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:59:27.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:59:27.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:59:27.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:59:27.219 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:59:27.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:59:27.634 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:59:27.634 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T15:59:27.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:59:27.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:59:27.635 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:59:27.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836505 2024-09-18T15:59:27.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836505 2024-09-18T15:59:27.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505' 2024-09-18T15:59:27.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:59:27.771 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:59:27.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672978 2024-09-18T15:59:27.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672978 2024-09-18T15:59:27.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505 1-42949672978' 2024-09-18T15:59:27.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:59:27.889 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:59:27.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509451 2024-09-18T15:59:27.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509451 2024-09-18T15:59:27.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505 1-42949672978 2-64424509451' 2024-09-18T15:59:27.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:59:27.998 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836505 2024-09-18T15:59:27.998 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:59:27.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:59:28.000 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836505 2024-09-18T15:59:28.000 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:59:28.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836505 2024-09-18T15:59:28.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-09-18T15:59:28.001 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836505 2024-09-18T15:59:28.001 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:59:28.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836503 -lt 21474836505 2024-09-18T15:59:28.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:59:29.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:59:29.452 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:59:29.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836505 -lt 21474836505 2024-09-18T15:59:29.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:59:29.868 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672978 2024-09-18T15:59:29.868 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:59:29.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:59:29.870 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672978 2024-09-18T15:59:29.870 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:59:29.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672978 2024-09-18T15:59:29.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-09-18T15:59:29.871 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672978 2024-09-18T15:59:29.871 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:59:30.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672978 -lt 42949672978 2024-09-18T15:59:30.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:59:30.277 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509451 2024-09-18T15:59:30.277 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:59:30.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:59:30.278 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509451 2024-09-18T15:59:30.278 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:59:30.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509451 2024-09-18T15:59:30.279 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509451 2024-09-18T15:59:30.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-09-18T15:59:30.280 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:59:30.642 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509452 -lt 64424509451 2024-09-18T15:59:30.643 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:59:30.643 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:59:30.644 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:59:31.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-09-18T15:59:31.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:59:31.144 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:59:31.144 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:59:31.145 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:59:31.145 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:59:31.145 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:59:31.145 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:59:31.503 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:59:31.503 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:59:31.503 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:59:31.504 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:59:31.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-09-18T15:59:31.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:59:31.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:59:31.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:56: TEST_a_merge_empty: kill_daemons td/pg-split-merge TERM osd.2 2024-09-18T15:59:31.999 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T15:59:32.000 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T15:59:32.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T15:59:32.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T15:59:32.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T15:59:32.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T15:59:32.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:57: TEST_a_merge_empty: ceph-objectstore-tool --data-path td/pg-split-merge/2 --op remove --pgid 1.1 --force 2024-09-18T15:59:33.507 INFO:tasks.workunit.client.0.smithi047.stdout: marking collection for removal 2024-09-18T15:59:33.507 INFO:tasks.workunit.client.0.smithi047.stdout:setting '_remove' omap key 2024-09-18T15:59:33.507 INFO:tasks.workunit.client.0.smithi047.stdout:finish_remove_pgs 1.1_head removing 1.1 2024-09-18T15:59:33.627 INFO:tasks.workunit.client.0.smithi047.stdout:Remove successful 2024-09-18T15:59:34.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:58: TEST_a_merge_empty: activate_osd td/pg-split-merge 2 2024-09-18T15:59:34.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-09-18T15:59:34.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T15:59:34.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T15:59:34.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T15:59:34.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/2 2024-09-18T15:59:34.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1a671e43-8866-44a5-abe3-c1e1f4ff765b --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-18T15:59:34.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T15:59:34.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T15:59:34.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T15:59:34.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/2' 2024-09-18T15:59:34.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/2/journal' 2024-09-18T15:59:34.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T15:59:34.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T15:59:34.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-09-18T15:59:34.026 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T15:59:34.026 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T15:59:34.026 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T15:59:34.026 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T15:59:34.026 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T15:59:34.026 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.463129 2024-09-18T15:59:34.026 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.463129/$cluster-$name.asok' 2024-09-18T15:59:34.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.463129/$cluster-$name.asok' 2024-09-18T15:59:34.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T15:59:34.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-09-18T15:59:34.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-09-18T15:59:34.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T15:59:34.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T15:59:34.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T15:59:34.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T15:59:34.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T15:59:34.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T15:59:34.027 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/2 2024-09-18T15:59:34.027 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T15:59:34.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T15:59:34.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=1a671e43-8866-44a5-abe3-c1e1f4ff765b --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/2 --osd-journal=td/pg-split-merge/2/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.463129/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T15:59:34.029 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/2/whoami 2024-09-18T15:59:34.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T15:59:34.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T15:59:34.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T15:59:34.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T15:59:34.053 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:59:34.052+0000 7fc17a957440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:59:34.062 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:59:34.061+0000 7fc17a957440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:59:34.063 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:59:34.062+0000 7fc17a957440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T15:59:34.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T15:59:34.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T15:59:34.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T15:59:34.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T15:59:34.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T15:59:34.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:59:34.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T15:59:34.468 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T15:59:34.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:59:34.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:59:34.578 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:59:34.577+0000 7fc17a957440 -1 Falling back to public interface 2024-09-18T15:59:34.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:59:35.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:59:35.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:59:35.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T15:59:35.895 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T15:59:35.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:59:35.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:59:36.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T15:59:37.195 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T15:59:37.194+0000 7fc17a957440 -1 osd.2 25 log_to_monitors true 2024-09-18T15:59:37.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T15:59:37.305 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T15:59:37.305 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T15:59:37.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T15:59:37.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T15:59:37.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T15:59:37.818 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 28 up_thru 17 down_at 26 last_clean_interval [15,25) [v2:127.0.0.1:6818/470284,v1:127.0.0.1:6819/470284] [v2:127.0.0.1:6820/470284,v1:127.0.0.1:6821/470284] exists,up 3047ef09-2eed-40bf-9c34-62adf1977248 2024-09-18T15:59:37.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T15:59:37.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T15:59:37.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T15:59:37.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:60: TEST_a_merge_empty: wait_for_clean 2024-09-18T15:59:37.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:59:37.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:59:37.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:59:37.820 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:59:37.820 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:59:37.821 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:59:37.821 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:59:37.821 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:59:37.821 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:59:37.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:59:37.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:59:37.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:59:37.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:59:37.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:59:37.908 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:59:38.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:59:38.301 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:59:38.301 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T15:59:38.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:59:38.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:59:38.301 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:59:38.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836509 2024-09-18T15:59:38.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836509 2024-09-18T15:59:38.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509' 2024-09-18T15:59:38.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:59:38.411 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:59:38.528 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672982 2024-09-18T15:59:38.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672982 2024-09-18T15:59:38.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509 1-42949672982' 2024-09-18T15:59:38.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:59:38.529 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:59:38.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084291 2024-09-18T15:59:38.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084291 2024-09-18T15:59:38.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509 1-42949672982 2-120259084291' 2024-09-18T15:59:38.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:59:38.638 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836509 2024-09-18T15:59:38.638 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:59:38.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:59:38.639 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836509 2024-09-18T15:59:38.639 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:59:38.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836509 2024-09-18T15:59:38.641 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836509 2024-09-18T15:59:38.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-09-18T15:59:38.641 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:59:39.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836507 -lt 21474836509 2024-09-18T15:59:39.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:59:40.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:59:40.049 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:59:40.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836509 -lt 21474836509 2024-09-18T15:59:40.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:59:40.450 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672982 2024-09-18T15:59:40.450 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:59:40.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:59:40.451 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672982 2024-09-18T15:59:40.451 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:59:40.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672982 2024-09-18T15:59:40.452 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672982 2024-09-18T15:59:40.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-09-18T15:59:40.453 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:59:40.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672982 -lt 42949672982 2024-09-18T15:59:40.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:59:40.860 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-120259084291 2024-09-18T15:59:40.860 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:59:40.861 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:59:40.861 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-120259084291 2024-09-18T15:59:40.861 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:59:40.862 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084291 2024-09-18T15:59:40.863 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 120259084291 2024-09-18T15:59:40.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 120259084291' 2024-09-18T15:59:40.863 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:59:41.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084291 -lt 120259084291 2024-09-18T15:59:41.269 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:59:41.270 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:59:41.270 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:59:41.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-09-18T15:59:41.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:59:41.770 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:59:41.770 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:59:41.771 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:59:41.771 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:59:41.771 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:59:41.771 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:59:42.168 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T15:59:42.168 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:59:42.169 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:59:42.169 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:59:42.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-09-18T15:59:42.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:59:42.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:59:42.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:66: TEST_a_merge_empty: ceph tell osd.2 config set osd_debug_no_acting_change true 2024-09-18T15:59:42.802 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:59:42.802 INFO:tasks.workunit.client.0.smithi047.stdout: "success": "" 2024-09-18T15:59:42.803 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:59:42.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:67: TEST_a_merge_empty: ceph osd out 0 1 2024-09-18T15:59:43.591 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already out. osd.1 is already out. 2024-09-18T15:59:43.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:68: TEST_a_merge_empty: ceph osd pool set foo pg_num 1 2024-09-18T15:59:44.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:69: TEST_a_merge_empty: sleep 5 2024-09-18T15:59:49.542 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:70: TEST_a_merge_empty: ceph tell osd.2 config set osd_debug_no_acting_change false 2024-09-18T15:59:49.656 INFO:tasks.workunit.client.0.smithi047.stdout:{ 2024-09-18T15:59:49.656 INFO:tasks.workunit.client.0.smithi047.stdout: "success": "" 2024-09-18T15:59:49.656 INFO:tasks.workunit.client.0.smithi047.stdout:} 2024-09-18T15:59:49.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:74: TEST_a_merge_empty: ceph osd in 0 1 2024-09-18T15:59:50.267 INFO:tasks.workunit.client.0.smithi047.stderr:osd.0 is already in. osd.1 is already in. 2024-09-18T15:59:50.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:75: TEST_a_merge_empty: ceph osd pool set foo size 3 2024-09-18T15:59:51.886 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 1 size to 3 2024-09-18T15:59:51.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:77: TEST_a_merge_empty: wait_for_clean 2024-09-18T15:59:51.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T15:59:51.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T15:59:51.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T15:59:51.901 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T15:59:51.901 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T15:59:51.901 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T15:59:51.901 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T15:59:51.901 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T15:59:51.901 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T15:59:51.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T15:59:51.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T15:59:51.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T15:59:51.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T15:59:51.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T15:59:51.996 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T15:59:52.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T15:59:52.566 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T15:59:52.566 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T15:59:52.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T15:59:52.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:59:52.566 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T15:59:52.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836514 2024-09-18T15:59:52.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836514 2024-09-18T15:59:52.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836514' 2024-09-18T15:59:52.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:59:52.767 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T15:59:52.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672987 2024-09-18T15:59:52.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672987 2024-09-18T15:59:52.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836514 1-42949672987' 2024-09-18T15:59:52.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T15:59:52.878 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T15:59:53.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084296 2024-09-18T15:59:53.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084296 2024-09-18T15:59:53.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836514 1-42949672987 2-120259084296' 2024-09-18T15:59:53.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:59:53.019 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836514 2024-09-18T15:59:53.019 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:59:53.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T15:59:53.020 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836514 2024-09-18T15:59:53.020 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:59:53.021 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836514 2024-09-18T15:59:53.021 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836514 2024-09-18T15:59:53.021 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2024-09-18T15:59:53.022 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:59:53.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836511 -lt 21474836514 2024-09-18T15:59:53.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T15:59:54.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T15:59:54.422 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T15:59:54.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836514 -lt 21474836514 2024-09-18T15:59:54.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:59:54.838 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672987 2024-09-18T15:59:54.838 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:59:54.839 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T15:59:54.839 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672987 2024-09-18T15:59:54.839 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:59:54.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672987 2024-09-18T15:59:54.840 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672987 2024-09-18T15:59:54.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2024-09-18T15:59:54.841 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T15:59:55.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672987 -lt 42949672987 2024-09-18T15:59:55.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T15:59:55.206 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-120259084296 2024-09-18T15:59:55.206 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T15:59:55.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T15:59:55.207 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-120259084296 2024-09-18T15:59:55.207 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T15:59:55.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084296 2024-09-18T15:59:55.209 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 120259084296 2024-09-18T15:59:55.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 120259084296' 2024-09-18T15:59:55.209 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T15:59:55.623 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084296 -lt 120259084296 2024-09-18T15:59:55.624 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T15:59:55.624 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:59:55.624 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:59:56.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T15:59:56.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T15:59:56.160 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T15:59:56.160 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T15:59:56.160 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T15:59:56.160 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T15:59:56.160 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T15:59:56.160 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T15:59:56.550 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T15:59:56.550 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T15:59:56.551 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T15:59:56.551 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T15:59:57.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T15:59:57.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T15:59:57.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T15:59:57.043 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:81: TEST_a_merge_empty: ceph pg scrub 1.0 2024-09-18T15:59:57.412 INFO:tasks.workunit.client.0.smithi047.stderr:instructing pg 1.0 on osd.1 to scrub 2024-09-18T15:59:57.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:82: TEST_a_merge_empty: sleep 10 2024-09-18T16:00:07.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:83: TEST_a_merge_empty: ceph log last debug 2024-09-18T16:00:07.924 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T15:59:51.010696+0000 mon.a (mon.0) 385 : cluster [DBG] osdmap e37: 3 total, 3 up, 3 in 2024-09-18T16:00:07.924 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T15:59:51.466425+0000 mon.a (mon.0) 387 : cluster [DBG] osdmap e38: 3 total, 3 up, 3 in 2024-09-18T16:00:07.924 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T15:59:51.885828+0000 mon.a (mon.0) 389 : cluster [DBG] osdmap e39: 3 total, 3 up, 3 in 2024-09-18T16:00:07.925 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T15:59:51.619803+0000 mgr.x (mgr.4100) 66 : cluster [DBG] pgmap v95: 1 pgs: 1 unknown; 88 KiB data, 291 MiB used, 100 GiB / 100 GiB avail 2024-09-18T16:00:07.925 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T15:59:52.153448+0000 mon.a (mon.0) 390 : cluster [DBG] osdmap e40: 3 total, 3 up, 3 in 2024-09-18T16:00:07.925 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T15:59:52.497076+0000 mon.a (mon.0) 392 : cluster [DBG] osdmap e41: 3 total, 3 up, 3 in 2024-09-18T16:00:07.925 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T15:59:52.579720+0000 osd.1 (osd.1) 3 : cluster [INF] 1.0 continuing backfill to osd.2 from (23'7,23'7] MIN to 23'7 2024-09-18T16:00:07.925 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T15:59:53.285718+0000 mon.a (mon.0) 394 : cluster [DBG] osdmap e42: 3 total, 3 up, 3 in 2024-09-18T16:00:07.925 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T15:59:53.662966+0000 mon.a (mon.0) 397 : cluster [DBG] osdmap e43: 3 total, 3 up, 3 in 2024-09-18T16:00:07.925 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T15:59:53.720323+0000 mon.a (mon.0) 398 : cluster [WRN] Health check failed: Degraded data redundancy: 4/33 objects degraded (12.121%), 1 pg degraded (PG_DEGRADED) 2024-09-18T16:00:07.925 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T15:59:53.620116+0000 mgr.x (mgr.4100) 67 : cluster [DBG] pgmap v100: 1 pgs: 1 active+undersized+degraded+remapped+backfilling; 138 KiB data, 873 MiB used, 299 GiB / 300 GiB avail; 23 KiB/s wr, 1 op/s; 4/33 objects degraded (12.121%) 2024-09-18T16:00:07.925 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T15:59:55.668836+0000 mon.a (mon.0) 405 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/33 objects degraded (12.121%), 1 pg degraded) 2024-09-18T16:00:07.925 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T15:59:55.620363+0000 mgr.x (mgr.4100) 68 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 138 KiB data, 873 MiB used, 299 GiB / 300 GiB avail; 14 KiB/s wr, 1 op/s; 13 KiB/s, 1 objects/s recovering 2024-09-18T16:00:07.925 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T15:59:57.620630+0000 mgr.x (mgr.4100) 73 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 138 KiB data, 873 MiB used, 299 GiB / 300 GiB avail; 9.4 KiB/s wr, 0 op/s; 9.1 KiB/s, 0 objects/s recovering 2024-09-18T16:00:07.925 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T15:59:59.621082+0000 mgr.x (mgr.4100) 74 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 138 KiB data, 873 MiB used, 299 GiB / 300 GiB avail; 7.3 KiB/s wr, 0 op/s; 7.1 KiB/s, 0 objects/s recovering 2024-09-18T16:00:07.925 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T16:00:01.621548+0000 mgr.x (mgr.4100) 75 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 138 KiB data, 873 MiB used, 299 GiB / 300 GiB avail; 6.0 KiB/s, 0 objects/s recovering 2024-09-18T16:00:07.925 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T16:00:03.621891+0000 mgr.x (mgr.4100) 76 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 138 KiB data, 873 MiB used, 299 GiB / 300 GiB avail; 5.0 KiB/s, 0 objects/s recovering 2024-09-18T16:00:07.925 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T16:00:05.070930+0000 osd.1 (osd.1) 4 : cluster [DBG] 1.0 scrub starts 2024-09-18T16:00:07.926 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T16:00:05.073222+0000 osd.1 (osd.1) 5 : cluster [DBG] 1.0 scrub ok 2024-09-18T16:00:07.926 INFO:tasks.workunit.client.0.smithi047.stdout:2024-09-18T16:00:05.622370+0000 mgr.x (mgr.4100) 77 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 138 KiB data, 873 MiB used, 299 GiB / 300 GiB avail; 4.2 KiB/s, 0 objects/s recovering 2024-09-18T16:00:07.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:84: TEST_a_merge_empty: ceph pg ls 2024-09-18T16:00:08.311 INFO:tasks.workunit.client.0.smithi047.stdout:PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS STATE SINCE VERSION REPORTED UP ACTING SCRUB_STAMP DEEP_SCRUB_STAMP LAST_SCRUB_DURATION SCRUB_SCHEDULING 2024-09-18T16:00:08.311 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 11 0 0 0 141625 0 0 0 11 active+clean 14s 23'7 43:88 [1,0,2]p1 [1,0,2]p1 2024-09-18T15:59:13.265647+0000 2024-09-18T15:59:13.265647+0000 0 queued for scrub 2024-09-18T16:00:08.312 INFO:tasks.workunit.client.0.smithi047.stdout: 2024-09-18T16:00:08.312 INFO:tasks.workunit.client.0.smithi047.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T16:00:08.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:85: TEST_a_merge_empty: ceph pg ls 2024-09-18T16:00:08.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:85: TEST_a_merge_empty: grep ' active.clean ' 2024-09-18T16:00:08.682 INFO:tasks.workunit.client.0.smithi047.stdout:1.0 11 0 0 0 141625 0 0 0 11 active+clean 14s 23'7 43:88 [1,0,2]p1 [1,0,2]p1 2024-09-18T15:59:13.265647+0000 2024-09-18T15:59:13.265647+0000 0 queued for scrub 2024-09-18T16:00:08.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:17: run: teardown td/pg-split-merge 2024-09-18T16:00:08.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-09-18T16:00:08.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:00:08.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-09-18T16:00:08.683 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:00:08.684 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:00:08.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:00:08.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:00:08.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:00:08.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:00:08.791 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:00:08.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:00:08.792 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:00:08.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:00:08.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:00:08.793 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:00:08.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:00:08.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:00:08.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:00:08.794 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:00:08.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:00:08.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:00:08.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-09-18T16:00:08.813 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:00:08.813 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:00:08.813 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.463129 2024-09-18T16:00:08.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.463129 2024-09-18T16:00:08.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:00:08.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:00:08.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:14: run: for func in $funcs 2024-09-18T16:00:08.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:15: run: setup td/pg-split-merge 2024-09-18T16:00:08.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/pg-split-merge 2024-09-18T16:00:08.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/pg-split-merge 2024-09-18T16:00:08.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-09-18T16:00:08.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:00:08.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-09-18T16:00:08.816 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:00:08.816 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:00:08.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:00:08.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:00:08.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:00:08.816 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:00:08.817 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:00:08.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:00:08.817 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:00:08.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:00:08.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:00:08.819 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:00:08.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:00:08.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:00:08.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:00:08.820 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:00:08.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:00:08.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:00:08.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-09-18T16:00:08.823 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:00:08.823 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:00:08.823 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.463129 2024-09-18T16:00:08.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.463129 2024-09-18T16:00:08.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:00:08.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:00:08.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/pg-split-merge 2024-09-18T16:00:08.825 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T16:00:08.825 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:00:08.825 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.463129 2024-09-18T16:00:08.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.463129 2024-09-18T16:00:08.828 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T16:00:08.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T16:00:08.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T16:00:08.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/pg-split-merge 1' TERM HUP INT 2024-09-18T16:00:08.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:16: run: TEST_import_after_merge_and_gap td/pg-split-merge 2024-09-18T16:00:08.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:89: TEST_import_after_merge_and_gap: local dir=td/pg-split-merge 2024-09-18T16:00:08.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:91: TEST_import_after_merge_and_gap: run_mon td/pg-split-merge a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T16:00:08.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/pg-split-merge 2024-09-18T16:00:08.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T16:00:08.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T16:00:08.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T16:00:08.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/pg-split-merge/a 2024-09-18T16:00:08.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/pg-split-merge/a --run-dir=td/pg-split-merge --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T16:00:09.123 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T16:00:09.123 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:00:09.124 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:00:09.124 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:00:09.124 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:00:09.124 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.463129 2024-09-18T16:00:09.124 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.463129/$cluster-$name.asok' 2024-09-18T16:00:09.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/pg-split-merge/a '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.463129/$cluster-$name.asok' --mon-cluster-log-file=td/pg-split-merge/log --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T16:00:09.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T16:00:09.364 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T16:00:09.364 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:00:09.364 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:00:09.364 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T16:00:09.364 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T16:00:09.364 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:00:09.365 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:00:09.365 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:00:09.365 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:00:09.365 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:00:09.366 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.463129 2024-09-18T16:00:09.366 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.463129/ceph-mon.a.asok 2024-09-18T16:00:09.366 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:00:09.366 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.463129/ceph-mon.a.asok config get fsid 2024-09-18T16:00:09.443 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T16:00:09.443 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:00:09.443 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:00:09.443 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T16:00:09.443 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T16:00:09.443 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:00:09.443 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:00:09.443 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:00:09.443 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:00:09.443 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:00:09.443 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.463129 2024-09-18T16:00:09.444 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.463129/ceph-mon.a.asok 2024-09-18T16:00:09.444 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:00:09.444 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.463129/ceph-mon.a.asok config get mon_host 2024-09-18T16:00:09.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:92: TEST_import_after_merge_and_gap: run_mgr td/pg-split-merge x 2024-09-18T16:00:09.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/pg-split-merge 2024-09-18T16:00:09.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T16:00:09.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T16:00:09.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T16:00:09.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/pg-split-merge/x 2024-09-18T16:00:09.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T16:00:10.370 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T16:00:10.370 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:00:10.370 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:00:10.370 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:00:10.370 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:00:10.371 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.463129 2024-09-18T16:00:10.371 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.463129/$cluster-$name.asok' 2024-09-18T16:00:10.371 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:00:10.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/pg-split-merge/x '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.463129/$cluster-$name.asok' --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:00:10.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:93: TEST_import_after_merge_and_gap: run_osd td/pg-split-merge 0 2024-09-18T16:00:10.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/pg-split-merge 2024-09-18T16:00:10.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:00:10.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T16:00:10.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:00:10.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/pg-split-merge/0 2024-09-18T16:00:10.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a671e43-8866-44a5-abe3-c1e1f4ff765b --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-18T16:00:10.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:00:10.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:00:10.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:00:10.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-09-18T16:00:10.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-09-18T16:00:10.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:00:10.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:00:10.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-09-18T16:00:10.401 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:00:10.401 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:00:10.401 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:00:10.401 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:00:10.401 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:00:10.401 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.463129 2024-09-18T16:00:10.401 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.463129/$cluster-$name.asok' 2024-09-18T16:00:10.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.463129/$cluster-$name.asok' 2024-09-18T16:00:10.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:00:10.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:00:10.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:00:10.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-09-18T16:00:10.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-09-18T16:00:10.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:00:10.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:00:10.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:00:10.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:00:10.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:00:10.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:00:10.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/pg-split-merge/0 2024-09-18T16:00:10.403 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:00:10.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1b6f27f6-2f00-44c7-8d93-1f3fa5ab4594 2024-09-18T16:00:10.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1b6f27f6-2f00-44c7-8d93-1f3fa5ab4594' 2024-09-18T16:00:10.403 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 1b6f27f6-2f00-44c7-8d93-1f3fa5ab4594 2024-09-18T16:00:10.403 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:00:10.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAK+epmzaLcGBAAl+f5qlAFR8zQrFAehqZW8Q== 2024-09-18T16:00:10.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAK+epmzaLcGBAAl+f5qlAFR8zQrFAehqZW8Q=="}' 2024-09-18T16:00:10.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1b6f27f6-2f00-44c7-8d93-1f3fa5ab4594 -i td/pg-split-merge/0/new.json 2024-09-18T16:00:10.747 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:00:10.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/pg-split-merge/0/new.json 2024-09-18T16:00:10.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1a671e43-8866-44a5-abe3-c1e1f4ff765b --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.463129/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAK+epmzaLcGBAAl+f5qlAFR8zQrFAehqZW8Q== --osd-uuid 1b6f27f6-2f00-44c7-8d93-1f3fa5ab4594 2024-09-18T16:00:10.784 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:00:10.783+0000 7f7b38780440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:00:10.786 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:00:10.785+0000 7f7b38780440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:00:10.787 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:00:10.786+0000 7f7b38780440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:00:10.788 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:00:10.787+0000 7f7b38780440 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-09-18T16:00:10.788 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:00:10.787+0000 7f7b38780440 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-09-18T16:00:10.788 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:00:10.787+0000 7f7b38780440 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-09-18T16:00:10.863 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:00:10.862+0000 7f7b38780440 -1 bluestore(td/pg-split-merge/0) _read_fsid unparsable uuid 2024-09-18T16:00:16.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/pg-split-merge/0/keyring 2024-09-18T16:00:16.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:00:16.160 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T16:00:16.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T16:00:16.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/pg-split-merge/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:00:16.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T16:00:16.679 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T16:00:16.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1a671e43-8866-44a5-abe3-c1e1f4ff765b --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.463129/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:00:16.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:00:16.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:00:16.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:00:16.703 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:00:16.702+0000 7f1663816440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:00:16.711 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:00:16.710+0000 7f1663816440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:00:16.712 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:00:16.711+0000 7f1663816440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:00:17.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T16:00:17.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:00:17.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T16:00:17.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:00:17.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:00:17.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:17.100 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:00:17.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:00:17.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:17.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:17.243 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:00:17.242+0000 7f1663816440 -1 Falling back to public interface 2024-09-18T16:00:17.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:18.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:18.518 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:18.518 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T16:00:18.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:00:18.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:18.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:18.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:19.714 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:00:19.713+0000 7f1663816440 -1 osd.0 0 log_to_monitors true 2024-09-18T16:00:19.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:19.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:19.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:00:19.899 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T16:00:19.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:19.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:20.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:21.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:21.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:21.424 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:00:21.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:00:21.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:21.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:21.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:22.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:22.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:22.804 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T16:00:22.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:00:22.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:22.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:23.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:24.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:24.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:24.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:00:24.186 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T16:00:24.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:24.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:24.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:25.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:25.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:25.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:00:25.693 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T16:00:25.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:25.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:26.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:27.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:27.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:27.090 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T16:00:27.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:00:27.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:27.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:27.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:28.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:28.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:28.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:00:28.524 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T16:00:28.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:28.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:28.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:29.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:29.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:29.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:00:29.911 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T16:00:29.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:29.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:30.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:31.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:31.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:31.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:00:31.387 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T16:00:31.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:31.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:32.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:33.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:33.146 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T16:00:33.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:33.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:00:33.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:33.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:33.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:34.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:34.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:34.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:00:34.659 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T16:00:34.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:34.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:35.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:36.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:36.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:36.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:00:36.335 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T16:00:36.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:36.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:37.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:38.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:38.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:38.264 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:00:38.264 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T16:00:38.264 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:38.264 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:38.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:39.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:39.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:39.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:00:39.722 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T16:00:39.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:39.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:40.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:41.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:41.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:41.278 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T16:00:41.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:00:41.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:41.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:41.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:42.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:42.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:42.696 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T16:00:42.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:00:42.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:42.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:43.099 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/472837,v1:127.0.0.1:6803/472837] [v2:127.0.0.1:6804/472837,v1:127.0.0.1:6805/472837] exists,up 1b6f27f6-2f00-44c7-8d93-1f3fa5ab4594 2024-09-18T16:00:43.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:00:43.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:00:43.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:00:43.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:95: TEST_import_after_merge_and_gap: ceph osd pool create foo 2 2024-09-18T16:00:43.708 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'foo' already exists 2024-09-18T16:00:43.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:96: TEST_import_after_merge_and_gap: wait_for_clean 2024-09-18T16:00:43.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:00:43.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:00:43.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:00:43.720 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:00:43.720 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:00:43.721 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:00:43.721 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:00:43.721 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:00:43.721 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:00:43.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T16:00:43.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:00:43.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:00:43.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:00:43.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:00:43.809 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:00:44.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T16:00:44.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:00:44.307 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:00:44.307 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:00:44.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836486 2024-09-18T16:00:44.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836486 2024-09-18T16:00:44.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836486' 2024-09-18T16:00:44.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:00:44.417 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836486 2024-09-18T16:00:44.417 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:00:44.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:00:44.418 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836486 2024-09-18T16:00:44.418 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:00:44.419 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836486 2024-09-18T16:00:44.420 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836486 2024-09-18T16:00:44.420 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-09-18T16:00:44.420 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:00:44.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836484 -lt 21474836486 2024-09-18T16:00:44.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:00:45.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:00:45.811 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:00:46.194 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836486 -lt 21474836486 2024-09-18T16:00:46.194 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:00:46.194 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:00:46.194 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:00:46.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-09-18T16:00:46.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:00:46.736 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:00:46.736 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:00:46.736 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:00:46.736 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:00:46.736 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:00:46.736 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:00:47.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T16:00:47.093 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:00:47.094 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:00:47.094 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:00:47.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-09-18T16:00:47.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:00:47.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:00:47.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:97: TEST_import_after_merge_and_gap: rados -p foo bench 3 write -b 1024 --no-cleanup 2024-09-18T16:00:47.626 INFO:tasks.workunit.client.0.smithi047.stdout:hints = 1 2024-09-18T16:00:47.626 INFO:tasks.workunit.client.0.smithi047.stdout:Maintaining 16 concurrent writes of 1024 bytes to objects of size 1024 for up to 3 seconds or 0 objects 2024-09-18T16:00:47.626 INFO:tasks.workunit.client.0.smithi047.stdout:Object prefix: benchmark_data_smithi047_474001 2024-09-18T16:00:47.626 INFO:tasks.workunit.client.0.smithi047.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T16:00:47.626 INFO:tasks.workunit.client.0.smithi047.stdout: 0 0 0 0 0 0 - 0 2024-09-18T16:00:48.626 INFO:tasks.workunit.client.0.smithi047.stdout: 1 16 144 128 0.124988 0.125 0.168242 0.120208 2024-09-18T16:00:49.627 INFO:tasks.workunit.client.0.smithi047.stdout: 2 16 289 273 0.133284 0.141602 0.116981 0.112018 2024-09-18T16:00:50.627 INFO:tasks.workunit.client.0.smithi047.stdout: 3 16 432 416 0.135398 0.139648 0.167749 0.11243 2024-09-18T16:00:51.627 INFO:tasks.workunit.client.0.smithi047.stdout:Total time run: 3.06418 2024-09-18T16:00:51.627 INFO:tasks.workunit.client.0.smithi047.stdout:Total writes made: 432 2024-09-18T16:00:51.627 INFO:tasks.workunit.client.0.smithi047.stdout:Write size: 1024 2024-09-18T16:00:51.627 INFO:tasks.workunit.client.0.smithi047.stdout:Object size: 1024 2024-09-18T16:00:51.627 INFO:tasks.workunit.client.0.smithi047.stdout:Bandwidth (MB/sec): 0.13768 2024-09-18T16:00:51.627 INFO:tasks.workunit.client.0.smithi047.stdout:Stddev Bandwidth: 0.0090738 2024-09-18T16:00:51.627 INFO:tasks.workunit.client.0.smithi047.stdout:Max bandwidth (MB/sec): 0.141602 2024-09-18T16:00:51.627 INFO:tasks.workunit.client.0.smithi047.stdout:Min bandwidth (MB/sec): 0.125 2024-09-18T16:00:51.627 INFO:tasks.workunit.client.0.smithi047.stdout:Average IOPS: 140 2024-09-18T16:00:51.628 INFO:tasks.workunit.client.0.smithi047.stdout:Stddev IOPS: 9.29157 2024-09-18T16:00:51.628 INFO:tasks.workunit.client.0.smithi047.stdout:Max IOPS: 145 2024-09-18T16:00:51.628 INFO:tasks.workunit.client.0.smithi047.stdout:Min IOPS: 128 2024-09-18T16:00:51.628 INFO:tasks.workunit.client.0.smithi047.stdout:Average Latency(s): 0.11335 2024-09-18T16:00:51.628 INFO:tasks.workunit.client.0.smithi047.stdout:Stddev Latency(s): 0.0230432 2024-09-18T16:00:51.628 INFO:tasks.workunit.client.0.smithi047.stdout:Max latency(s): 0.192885 2024-09-18T16:00:51.628 INFO:tasks.workunit.client.0.smithi047.stdout:Min latency(s): 0.0798269 2024-09-18T16:00:51.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:99: TEST_import_after_merge_and_gap: kill_daemons td/pg-split-merge TERM osd.0 2024-09-18T16:00:51.668 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:00:51.668 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:00:51.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:00:51.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:00:51.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:00:51.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:00:51.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:100: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op export --pgid 1.1 --file td/pg-split-merge/1.1 --force 2024-09-18T16:00:53.301 INFO:tasks.workunit.client.0.smithi047.stderr:Exporting 1.1 info 1.1( v 9'229 (0'0,9'229] local-lis/les=7/8 n=229 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-09-18T16:00:53.302 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:806a8ff0:::benchmark_data_smithi047_474001_object14:head# 2024-09-18T16:00:53.303 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:827184ed:::benchmark_data_smithi047_474001_object202:head# 2024-09-18T16:00:53.305 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:83cb4be8:::benchmark_data_smithi047_474001_object189:head# 2024-09-18T16:00:53.306 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:83d7b3d6:::benchmark_data_smithi047_474001_object427:head# 2024-09-18T16:00:53.306 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:83e6a7ba:::benchmark_data_smithi047_474001_object326:head# 2024-09-18T16:00:53.308 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:83e714c1:::benchmark_data_smithi047_474001_object384:head# 2024-09-18T16:00:53.309 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:84fc4a07:::benchmark_data_smithi047_474001_object235:head# 2024-09-18T16:00:53.309 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8635a002:::benchmark_data_smithi047_474001_object8:head# 2024-09-18T16:00:53.318 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8659e9ad:::benchmark_data_smithi047_474001_object111:head# 2024-09-18T16:00:53.328 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:865f4a69:::benchmark_data_smithi047_474001_object312:head# 2024-09-18T16:00:53.330 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:87201b49:::benchmark_data_smithi047_474001_object185:head# 2024-09-18T16:00:53.330 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:889f60a0:::benchmark_data_smithi047_474001_object109:head# 2024-09-18T16:00:53.331 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8904e269:::benchmark_data_smithi047_474001_object228:head# 2024-09-18T16:00:53.331 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:89ec4c33:::benchmark_data_smithi047_474001_object133:head# 2024-09-18T16:00:53.331 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8a2258fa:::benchmark_data_smithi047_474001_object119:head# 2024-09-18T16:00:53.332 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8aba1a3a:::benchmark_data_smithi047_474001_object291:head# 2024-09-18T16:00:53.333 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8cb0b5a1:::benchmark_data_smithi047_474001_object3:head# 2024-09-18T16:00:53.333 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8d65845a:::benchmark_data_smithi047_474001_object324:head# 2024-09-18T16:00:53.333 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8e21472c:::benchmark_data_smithi047_474001_object110:head# 2024-09-18T16:00:53.334 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8efa467b:::benchmark_data_smithi047_474001_object247:head# 2024-09-18T16:00:53.334 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8f162f27:::benchmark_data_smithi047_474001_object222:head# 2024-09-18T16:00:53.335 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8f2ab4de:::benchmark_data_smithi047_474001_object112:head# 2024-09-18T16:00:53.335 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8f3016de:::benchmark_data_smithi047_474001_object337:head# 2024-09-18T16:00:53.335 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:906470c5:::benchmark_data_smithi047_474001_object108:head# 2024-09-18T16:00:53.336 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:90a988ef:::benchmark_data_smithi047_474001_object391:head# 2024-09-18T16:00:53.337 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:91461d4c:::benchmark_data_smithi047_474001_object252:head# 2024-09-18T16:00:53.337 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:92438533:::benchmark_data_smithi047_474001_object17:head# 2024-09-18T16:00:53.337 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9256e853:::benchmark_data_smithi047_474001_object314:head# 2024-09-18T16:00:53.338 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:928bb93a:::benchmark_data_smithi047_474001_object51:head# 2024-09-18T16:00:53.338 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:92b0a2c6:::benchmark_data_smithi047_474001_object250:head# 2024-09-18T16:00:53.338 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9324d622:::benchmark_data_smithi047_474001_object95:head# 2024-09-18T16:00:53.339 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:939c1aa8:::benchmark_data_smithi047_474001_object382:head# 2024-09-18T16:00:53.339 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:945bc203:::benchmark_data_smithi047_474001_object177:head# 2024-09-18T16:00:53.340 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:94707a46:::benchmark_data_smithi047_474001_object138:head# 2024-09-18T16:00:53.340 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:94f79b31:::benchmark_data_smithi047_474001_object253:head# 2024-09-18T16:00:53.341 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9502e979:::benchmark_data_smithi047_474001_object50:head# 2024-09-18T16:00:53.341 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:953c3a4c:::benchmark_data_smithi047_474001_object74:head# 2024-09-18T16:00:53.341 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:95edf5ad:::benchmark_data_smithi047_474001_object345:head# 2024-09-18T16:00:53.342 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9612cb9d:::benchmark_data_smithi047_474001_object59:head# 2024-09-18T16:00:53.342 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9670fdab:::benchmark_data_smithi047_474001_object116:head# 2024-09-18T16:00:53.343 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:978a53fd:::benchmark_data_smithi047_474001_object358:head# 2024-09-18T16:00:53.343 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:986d3d20:::benchmark_data_smithi047_474001_object38:head# 2024-09-18T16:00:53.343 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9a0edcb5:::benchmark_data_smithi047_474001_object89:head# 2024-09-18T16:00:53.343 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9bef3576:::benchmark_data_smithi047_474001_object180:head# 2024-09-18T16:00:53.344 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9dafb813:::benchmark_data_smithi047_474001_object226:head# 2024-09-18T16:00:53.344 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9dc7b4a6:::benchmark_data_smithi047_474001_object346:head# 2024-09-18T16:00:53.344 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9de9d04a:::benchmark_data_smithi047_474001_object338:head# 2024-09-18T16:00:53.345 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9e17b853:::benchmark_data_smithi047_474001_object124:head# 2024-09-18T16:00:53.345 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9e534a18:::benchmark_data_smithi047_474001_object70:head# 2024-09-18T16:00:53.346 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9e5ea7f3:::benchmark_data_smithi047_474001_object348:head# 2024-09-18T16:00:53.346 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a0254336:::benchmark_data_smithi047_474001_object188:head# 2024-09-18T16:00:53.346 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a0c154fe:::benchmark_data_smithi047_474001_object383:head# 2024-09-18T16:00:53.347 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a1981567:::benchmark_data_smithi047_474001_object414:head# 2024-09-18T16:00:53.351 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a23c3345:::benchmark_data_smithi047_474001_object408:head# 2024-09-18T16:00:53.351 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a244fe5e:::benchmark_data_smithi047_474001_object307:head# 2024-09-18T16:00:53.352 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a256b83b:::benchmark_data_smithi047_474001_object417:head# 2024-09-18T16:00:53.353 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a331eaa2:::benchmark_data_smithi047_474001_object103:head# 2024-09-18T16:00:53.353 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a36fdbde:::benchmark_data_smithi047_474001_object125:head# 2024-09-18T16:00:53.354 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a388d7f6:::benchmark_data_smithi047_474001_object46:head# 2024-09-18T16:00:53.354 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a473749a:::benchmark_data_smithi047_474001_object396:head# 2024-09-18T16:00:53.354 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a546fc38:::benchmark_data_smithi047_474001_object344:head# 2024-09-18T16:00:53.355 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a56684c3:::benchmark_data_smithi047_474001_object73:head# 2024-09-18T16:00:53.355 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a6a637ca:::benchmark_data_smithi047_474001_object352:head# 2024-09-18T16:00:53.356 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a6ac7233:::benchmark_data_smithi047_474001_object273:head# 2024-09-18T16:00:53.356 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a6be61c0:::benchmark_data_smithi047_474001_object212:head# 2024-09-18T16:00:53.357 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a6d03d77:::benchmark_data_smithi047_474001_object429:head# 2024-09-18T16:00:53.357 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a7029afc:::benchmark_data_smithi047_474001_object57:head# 2024-09-18T16:00:53.357 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a87abb60:::benchmark_data_smithi047_474001_object28:head# 2024-09-18T16:00:53.358 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a99c9566:::benchmark_data_smithi047_474001_object285:head# 2024-09-18T16:00:53.358 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a9ee7f09:::benchmark_data_smithi047_474001_object60:head# 2024-09-18T16:00:53.358 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:aaf069cf:::benchmark_data_smithi047_474001_object426:head# 2024-09-18T16:00:53.359 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ac9a95a5:::benchmark_data_smithi047_474001_object327:head# 2024-09-18T16:00:53.359 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ad75c20a:::benchmark_data_smithi047_474001_object322:head# 2024-09-18T16:00:53.360 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:add97b11:::benchmark_data_smithi047_474001_object120:head# 2024-09-18T16:00:53.360 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ae2bc6da:::benchmark_data_smithi047_474001_object374:head# 2024-09-18T16:00:53.360 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ae32eaa9:::benchmark_data_smithi047_474001_object217:head# 2024-09-18T16:00:53.361 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ae5f08a5:::benchmark_data_smithi047_474001_object156:head# 2024-09-18T16:00:53.361 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ae90731a:::benchmark_data_smithi047_474001_object275:head# 2024-09-18T16:00:53.362 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:aeb96a61:::benchmark_data_smithi047_474001_object35:head# 2024-09-18T16:00:53.362 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:af8861c2:::benchmark_data_smithi047_474001_object19:head# 2024-09-18T16:00:53.362 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:afa79d73:::benchmark_data_smithi047_474001_object297:head# 2024-09-18T16:00:53.363 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b06f5ff6:::benchmark_data_smithi047_474001_object96:head# 2024-09-18T16:00:53.363 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b09fcb0b:::benchmark_data_smithi047_474001_object330:head# 2024-09-18T16:00:53.363 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b0f4cb26:::benchmark_data_smithi047_474001_object262:head# 2024-09-18T16:00:53.364 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b1589877:::benchmark_data_smithi047_474001_object332:head# 2024-09-18T16:00:53.364 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b1dd1fef:::benchmark_data_smithi047_474001_object134:head# 2024-09-18T16:00:53.365 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b3f3163f:::benchmark_data_smithi047_474001_object341:head# 2024-09-18T16:00:53.366 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b5265e20:::benchmark_data_smithi047_474001_object159:head# 2024-09-18T16:00:53.366 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b533e7e6:::benchmark_data_smithi047_474001_object78:head# 2024-09-18T16:00:53.366 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b585c96f:::benchmark_data_smithi047_474001_object422:head# 2024-09-18T16:00:53.366 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b5e588a2:::benchmark_data_smithi047_474001_object30:head# 2024-09-18T16:00:53.367 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b606ef85:::benchmark_data_smithi047_474001_object12:head# 2024-09-18T16:00:53.367 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b61ac151:::benchmark_data_smithi047_474001_object361:head# 2024-09-18T16:00:53.368 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b707ad78:::benchmark_data_smithi047_474001_object33:head# 2024-09-18T16:00:53.368 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b72090f3:::benchmark_data_smithi047_474001_object225:head# 2024-09-18T16:00:53.368 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b74f9085:::benchmark_data_smithi047_474001_object224:head# 2024-09-18T16:00:53.368 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b854e709:::benchmark_data_smithi047_474001_object279:head# 2024-09-18T16:00:53.369 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b8615e71:::benchmark_last_metadata:head# 2024-09-18T16:00:53.370 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b88427b9:::benchmark_data_smithi047_474001_object204:head# 2024-09-18T16:00:53.370 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b8a01441:::benchmark_data_smithi047_474001_object99:head# 2024-09-18T16:00:53.370 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b8d632ef:::benchmark_data_smithi047_474001_object49:head# 2024-09-18T16:00:53.371 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b9a03ab3:::benchmark_data_smithi047_474001_object41:head# 2024-09-18T16:00:53.371 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ba233a46:::benchmark_data_smithi047_474001_object368:head# 2024-09-18T16:00:53.371 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:bb9dcf18:::benchmark_data_smithi047_474001_object306:head# 2024-09-18T16:00:53.372 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:bc2e5e68:::benchmark_data_smithi047_474001_object24:head# 2024-09-18T16:00:53.372 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:bd291308:::benchmark_data_smithi047_474001_object316:head# 2024-09-18T16:00:53.373 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:bd68e671:::benchmark_data_smithi047_474001_object44:head# 2024-09-18T16:00:53.373 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:bdd1c304:::benchmark_data_smithi047_474001_object237:head# 2024-09-18T16:00:53.373 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:be3f8098:::benchmark_data_smithi047_474001_object423:head# 2024-09-18T16:00:53.373 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:be77543b:::benchmark_data_smithi047_474001_object77:head# 2024-09-18T16:00:53.374 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:be86e7ec:::benchmark_data_smithi047_474001_object122:head# 2024-09-18T16:00:53.374 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:bed16cbf:::benchmark_data_smithi047_474001_object137:head# 2024-09-18T16:00:53.374 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:bf89fe68:::benchmark_data_smithi047_474001_object54:head# 2024-09-18T16:00:53.375 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c08db5a2:::benchmark_data_smithi047_474001_object236:head# 2024-09-18T16:00:53.375 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c18ab8fc:::benchmark_data_smithi047_474001_object131:head# 2024-09-18T16:00:53.375 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c1fcb644:::benchmark_data_smithi047_474001_object158:head# 2024-09-18T16:00:53.376 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c24a7c46:::benchmark_data_smithi047_474001_object282:head# 2024-09-18T16:00:53.376 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c313dcbc:::benchmark_data_smithi047_474001_object321:head# 2024-09-18T16:00:53.376 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c35a5205:::benchmark_data_smithi047_474001_object303:head# 2024-09-18T16:00:53.377 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c3d61783:::benchmark_data_smithi047_474001_object288:head# 2024-09-18T16:00:53.377 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c3fe223d:::benchmark_data_smithi047_474001_object356:head# 2024-09-18T16:00:53.378 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c4b79889:::benchmark_data_smithi047_474001_object325:head# 2024-09-18T16:00:53.378 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c60c9d19:::benchmark_data_smithi047_474001_object146:head# 2024-09-18T16:00:53.379 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c6123b6f:::benchmark_data_smithi047_474001_object55:head# 2024-09-18T16:00:53.379 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c6869eae:::benchmark_data_smithi047_474001_object407:head# 2024-09-18T16:00:53.380 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c849831c:::benchmark_data_smithi047_474001_object118:head# 2024-09-18T16:00:53.380 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c9eb3ea0:::benchmark_data_smithi047_474001_object87:head# 2024-09-18T16:00:53.381 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ca4e33a3:::benchmark_data_smithi047_474001_object283:head# 2024-09-18T16:00:53.381 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:cb6747e8:::benchmark_data_smithi047_474001_object357:head# 2024-09-18T16:00:53.381 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:cb84b5d4:::benchmark_data_smithi047_474001_object173:head# 2024-09-18T16:00:53.382 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:cbdaedd2:::benchmark_data_smithi047_474001_object179:head# 2024-09-18T16:00:53.382 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:cd271f33:::benchmark_data_smithi047_474001_object166:head# 2024-09-18T16:00:53.383 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:cd28a341:::benchmark_data_smithi047_474001_object150:head# 2024-09-18T16:00:53.383 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:cd4d6810:::benchmark_data_smithi047_474001_object363:head# 2024-09-18T16:00:53.384 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:cd5cc2dc:::benchmark_data_smithi047_474001_object88:head# 2024-09-18T16:00:53.384 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:cd77e455:::benchmark_data_smithi047_474001_object404:head# 2024-09-18T16:00:53.385 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:cdedbf36:::benchmark_data_smithi047_474001_object155:head# 2024-09-18T16:00:53.385 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d007de6f:::benchmark_data_smithi047_474001_object151:head# 2024-09-18T16:00:53.386 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d275e6f0:::benchmark_data_smithi047_474001_object419:head# 2024-09-18T16:00:53.386 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d295be0f:::benchmark_data_smithi047_474001_object167:head# 2024-09-18T16:00:53.386 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d2992f10:::benchmark_data_smithi047_474001_object175:head# 2024-09-18T16:00:53.387 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d2cdad48:::benchmark_data_smithi047_474001_object90:head# 2024-09-18T16:00:53.387 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d43afad7:::benchmark_data_smithi047_474001_object376:head# 2024-09-18T16:00:53.387 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d6762583:::benchmark_data_smithi047_474001_object15:head# 2024-09-18T16:00:53.388 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d6d0e63b:::benchmark_data_smithi047_474001_object117:head# 2024-09-18T16:00:53.388 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d74454f2:::benchmark_data_smithi047_474001_object191:head# 2024-09-18T16:00:53.389 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d7ba857c:::benchmark_data_smithi047_474001_object160:head# 2024-09-18T16:00:53.389 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d7ca34da:::benchmark_data_smithi047_474001_object40:head# 2024-09-18T16:00:53.389 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d8141741:::benchmark_data_smithi047_474001_object287:head# 2024-09-18T16:00:53.390 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:da6a8dc2:::benchmark_data_smithi047_474001_object18:head# 2024-09-18T16:00:53.390 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:da6c2c4d:::benchmark_data_smithi047_474001_object31:head# 2024-09-18T16:00:53.391 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:db2f6939:::benchmark_data_smithi047_474001_object400:head# 2024-09-18T16:00:53.391 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:dc4893ce:::benchmark_data_smithi047_474001_object232:head# 2024-09-18T16:00:53.391 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:dc6daefe:::benchmark_data_smithi047_474001_object276:head# 2024-09-18T16:00:53.392 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:dcdad940:::benchmark_data_smithi047_474001_object182:head# 2024-09-18T16:00:53.392 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:dd4ec68f:::benchmark_data_smithi047_474001_object56:head# 2024-09-18T16:00:53.393 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ddc4bb3d:::benchmark_data_smithi047_474001_object199:head# 2024-09-18T16:00:53.393 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ddece86f:::benchmark_data_smithi047_474001_object43:head# 2024-09-18T16:00:53.394 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:df65649c:::benchmark_data_smithi047_474001_object176:head# 2024-09-18T16:00:53.394 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:dfa0b551:::benchmark_data_smithi047_474001_object421:head# 2024-09-18T16:00:53.395 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e07f15b6:::benchmark_data_smithi047_474001_object10:head# 2024-09-18T16:00:53.395 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e1757a69:::benchmark_data_smithi047_474001_object366:head# 2024-09-18T16:00:53.395 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e1859c12:::benchmark_data_smithi047_474001_object102:head# 2024-09-18T16:00:53.396 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e1d3a554:::benchmark_data_smithi047_474001_object369:head# 2024-09-18T16:00:53.396 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e437b5d2:::benchmark_data_smithi047_474001_object377:head# 2024-09-18T16:00:53.396 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e445afb5:::benchmark_data_smithi047_474001_object355:head# 2024-09-18T16:00:53.397 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e61a1975:::benchmark_data_smithi047_474001_object296:head# 2024-09-18T16:00:53.397 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e65db1db:::benchmark_data_smithi047_474001_object329:head# 2024-09-18T16:00:53.397 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e65e637a:::benchmark_data_smithi047_474001_object424:head# 2024-09-18T16:00:53.398 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e6b693e8:::benchmark_data_smithi047_474001_object308:head# 2024-09-18T16:00:53.398 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e6d85c1d:::benchmark_data_smithi047_474001_object220:head# 2024-09-18T16:00:53.398 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e6f3dc67:::benchmark_data_smithi047_474001_object293:head# 2024-09-18T16:00:53.399 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e6f460e2:::benchmark_data_smithi047_474001_object130:head# 2024-09-18T16:00:53.399 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e6f5814f:::benchmark_data_smithi047_474001_object64:head# 2024-09-18T16:00:53.400 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e7a2c4e1:::benchmark_data_smithi047_474001_object94:head# 2024-09-18T16:00:53.400 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e945e46a:::benchmark_data_smithi047_474001_object171:head# 2024-09-18T16:00:53.401 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e9b3c7f8:::benchmark_data_smithi047_474001_object181:head# 2024-09-18T16:00:53.401 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ea0dab4b:::benchmark_data_smithi047_474001_object406:head# 2024-09-18T16:00:53.402 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ea0dd8c2:::benchmark_data_smithi047_474001_object381:head# 2024-09-18T16:00:53.402 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ea20ab79:::benchmark_data_smithi047_474001_object178:head# 2024-09-18T16:00:53.403 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ea95d8c2:::benchmark_data_smithi047_474001_object238:head# 2024-09-18T16:00:53.403 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ecaeb389:::benchmark_data_smithi047_474001_object135:head# 2024-09-18T16:00:53.404 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ed907482:::benchmark_data_smithi047_474001_object34:head# 2024-09-18T16:00:53.404 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ee898c6a:::benchmark_data_smithi047_474001_object184:head# 2024-09-18T16:00:53.405 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:eeceac30:::benchmark_data_smithi047_474001_object97:head# 2024-09-18T16:00:53.405 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:eee17d95:::benchmark_data_smithi047_474001_object187:head# 2024-09-18T16:00:53.405 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:eefc7324:::benchmark_data_smithi047_474001_object91:head# 2024-09-18T16:00:53.406 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ef61b0b9:::benchmark_data_smithi047_474001_object428:head# 2024-09-18T16:00:53.406 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:efbee6f8:::benchmark_data_smithi047_474001_object211:head# 2024-09-18T16:00:53.407 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:eff64798:::benchmark_data_smithi047_474001_object93:head# 2024-09-18T16:00:53.407 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f09ab14b:::benchmark_data_smithi047_474001_object339:head# 2024-09-18T16:00:53.408 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f09c2c79:::benchmark_data_smithi047_474001_object258:head# 2024-09-18T16:00:53.408 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f0a2b2e0:::benchmark_data_smithi047_474001_object234:head# 2024-09-18T16:00:53.408 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f0e21778:::benchmark_data_smithi047_474001_object210:head# 2024-09-18T16:00:53.409 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f12591b2:::benchmark_data_smithi047_474001_object142:head# 2024-09-18T16:00:53.409 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f15f0e45:::benchmark_data_smithi047_474001_object353:head# 2024-09-18T16:00:53.410 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f17effda:::benchmark_data_smithi047_474001_object121:head# 2024-09-18T16:00:53.410 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f188e48b:::benchmark_data_smithi047_474001_object82:head# 2024-09-18T16:00:53.411 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f286bf95:::benchmark_data_smithi047_474001_object115:head# 2024-09-18T16:00:53.411 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f400c3e0:::benchmark_data_smithi047_474001_object305:head# 2024-09-18T16:00:53.411 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f40d5282:::benchmark_data_smithi047_474001_object48:head# 2024-09-18T16:00:53.412 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f4d9af26:::benchmark_data_smithi047_474001_object311:head# 2024-09-18T16:00:53.413 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f50aa0e5:::benchmark_data_smithi047_474001_object65:head# 2024-09-18T16:00:53.413 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f58a2ed4:::benchmark_data_smithi047_474001_object393:head# 2024-09-18T16:00:53.413 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f61edbb2:::benchmark_data_smithi047_474001_object140:head# 2024-09-18T16:00:53.413 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f6596924:::benchmark_data_smithi047_474001_object256:head# 2024-09-18T16:00:53.414 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f68bfbc5:::benchmark_data_smithi047_474001_object295:head# 2024-09-18T16:00:53.414 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f7031fa9:::benchmark_data_smithi047_474001_object318:head# 2024-09-18T16:00:53.415 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f8576353:::benchmark_data_smithi047_474001_object336:head# 2024-09-18T16:00:53.415 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f8e14a8e:::benchmark_data_smithi047_474001_object63:head# 2024-09-18T16:00:53.415 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f9e01ed3:::benchmark_data_smithi047_474001_object20:head# 2024-09-18T16:00:53.416 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fa676313:::benchmark_data_smithi047_474001_object268:head# 2024-09-18T16:00:53.416 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fb2704e8:::benchmark_data_smithi047_474001_object294:head# 2024-09-18T16:00:53.416 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fb557b5c:::benchmark_data_smithi047_474001_object25:head# 2024-09-18T16:00:53.417 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fba861de:::benchmark_data_smithi047_474001_object233:head# 2024-09-18T16:00:53.417 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fc48c5f5:::benchmark_data_smithi047_474001_object128:head# 2024-09-18T16:00:53.417 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fcc00bd9:::benchmark_data_smithi047_474001_object267:head# 2024-09-18T16:00:53.418 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fcc267e2:::benchmark_data_smithi047_474001_object420:head# 2024-09-18T16:00:53.418 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fcd59592:::benchmark_data_smithi047_474001_object387:head# 2024-09-18T16:00:53.419 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fcff1581:::benchmark_data_smithi047_474001_object229:head# 2024-09-18T16:00:53.419 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fd5c7c49:::benchmark_data_smithi047_474001_object375:head# 2024-09-18T16:00:53.420 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fdd0ff05:::benchmark_data_smithi047_474001_object67:head# 2024-09-18T16:00:53.420 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fdf729e9:::benchmark_data_smithi047_474001_object230:head# 2024-09-18T16:00:53.420 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fe6c9d0e:::benchmark_data_smithi047_474001_object52:head# 2024-09-18T16:00:53.421 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fe938a07:::benchmark_data_smithi047_474001_object207:head# 2024-09-18T16:00:53.421 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fea2446c:::benchmark_data_smithi047_474001_object105:head# 2024-09-18T16:00:53.422 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fee86c8a:::benchmark_data_smithi047_474001_object412:head# 2024-09-18T16:00:53.422 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ff03a3e6:::benchmark_data_smithi047_474001_object37:head# 2024-09-18T16:00:53.422 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ff8f6396:::benchmark_data_smithi047_474001_object114:head# 2024-09-18T16:00:53.423 INFO:tasks.workunit.client.0.smithi047.stderr:Export successful 2024-09-18T16:00:53.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:101: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op export --pgid 1.0 --file td/pg-split-merge/1.0 --force 2024-09-18T16:00:54.642 INFO:tasks.workunit.client.0.smithi047.stderr:Exporting 1.0 info 1.0( v 9'204 (0'0,9'204] local-lis/les=7/8 n=204 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-09-18T16:00:54.660 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0118b086:::benchmark_data_smithi047_474001_object299:head# 2024-09-18T16:00:54.666 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:03a83bbd:::benchmark_data_smithi047_474001_object340:head# 2024-09-18T16:00:54.666 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:03ef44ea:::benchmark_data_smithi047_474001_object136:head# 2024-09-18T16:00:54.678 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:04ce55e6:::benchmark_data_smithi047_474001_object104:head# 2024-09-18T16:00:54.687 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:04e7bb72:::benchmark_data_smithi047_474001_object259:head# 2024-09-18T16:00:54.691 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0522b134:::benchmark_data_smithi047_474001_object390:head# 2024-09-18T16:00:54.691 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:065742a4:::benchmark_data_smithi047_474001_object240:head# 2024-09-18T16:00:54.696 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0667ed83:::benchmark_data_smithi047_474001_object29:head# 2024-09-18T16:00:54.696 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:067cfdc0:::benchmark_data_smithi047_474001_object373:head# 2024-09-18T16:00:54.708 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:06873111:::benchmark_data_smithi047_474001_object270:head# 2024-09-18T16:00:54.716 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:06fbd7c2:::benchmark_data_smithi047_474001_object127:head# 2024-09-18T16:00:54.717 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:07ded5f1:::benchmark_data_smithi047_474001_object266:head# 2024-09-18T16:00:54.733 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0846b0e1:::benchmark_data_smithi047_474001_object255:head# 2024-09-18T16:00:54.741 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0a425810:::benchmark_data_smithi047_474001_object410:head# 2024-09-18T16:00:54.743 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0b99baba:::benchmark_data_smithi047_474001_object215:head# 2024-09-18T16:00:54.758 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0cfa8c38:::benchmark_data_smithi047_474001_object163:head# 2024-09-18T16:00:54.766 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0e30d732:::benchmark_data_smithi047_474001_object402:head# 2024-09-18T16:00:54.768 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0e508c23:::benchmark_data_smithi047_474001_object196:head# 2024-09-18T16:00:54.783 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0e642663:::benchmark_data_smithi047_474001_object80:head# 2024-09-18T16:00:54.784 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0e743ace:::benchmark_data_smithi047_474001_object21:head# 2024-09-18T16:00:54.792 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0eaf1576:::benchmark_data_smithi047_474001_object172:head# 2024-09-18T16:00:54.793 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0fcd5078:::benchmark_data_smithi047_474001_object364:head# 2024-09-18T16:00:54.809 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0fe45a41:::benchmark_data_smithi047_474001_object333:head# 2024-09-18T16:00:54.817 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:104ccd46:::benchmark_data_smithi047_474001_object365:head# 2024-09-18T16:00:54.818 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:10c24a87:::benchmark_data_smithi047_474001_object141:head# 2024-09-18T16:00:54.818 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:11014d3d:::benchmark_data_smithi047_474001_object281:head# 2024-09-18T16:00:54.818 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1161e93e:::benchmark_data_smithi047_474001_object278:head# 2024-09-18T16:00:54.820 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:129bbd1b:::benchmark_data_smithi047_474001_object389:head# 2024-09-18T16:00:54.820 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:131e86da:::benchmark_data_smithi047_474001_object203:head# 2024-09-18T16:00:54.820 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1444d7df:::benchmark_data_smithi047_474001_object271:head# 2024-09-18T16:00:54.820 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:14ce8c64:::benchmark_data_smithi047_474001_object246:head# 2024-09-18T16:00:54.820 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:15241e69:::benchmark_data_smithi047_474001_object148:head# 2024-09-18T16:00:54.820 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1593a397:::benchmark_data_smithi047_474001_object347:head# 2024-09-18T16:00:54.820 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:17f3626e:::benchmark_data_smithi047_474001_object42:head# 2024-09-18T16:00:54.820 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:180c940a:::benchmark_data_smithi047_474001_object372:head# 2024-09-18T16:00:54.820 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:183bb03e:::benchmark_data_smithi047_474001_object323:head# 2024-09-18T16:00:54.821 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:18747883:::benchmark_data_smithi047_474001_object192:head# 2024-09-18T16:00:54.821 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1a19ff87:::benchmark_data_smithi047_474001_object2:head# 2024-09-18T16:00:54.822 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1a2c0aea:::benchmark_data_smithi047_474001_object274:head# 2024-09-18T16:00:54.822 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1bd588e8:::benchmark_data_smithi047_474001_object320:head# 2024-09-18T16:00:54.822 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1cc6e811:::benchmark_data_smithi047_474001_object149:head# 2024-09-18T16:00:54.822 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1ce6bbbf:::benchmark_data_smithi047_474001_object164:head# 2024-09-18T16:00:54.824 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1d83d665:::benchmark_data_smithi047_474001_object153:head# 2024-09-18T16:00:54.824 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1d849551:::benchmark_data_smithi047_474001_object395:head# 2024-09-18T16:00:54.824 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1df00925:::benchmark_data_smithi047_474001_object86:head# 2024-09-18T16:00:54.824 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1ea58aa2:::benchmark_data_smithi047_474001_object265:head# 2024-09-18T16:00:54.824 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:20be7217:::benchmark_data_smithi047_474001_object145:head# 2024-09-18T16:00:54.824 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:21c5c01f:::benchmark_data_smithi047_474001_object69:head# 2024-09-18T16:00:54.826 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:221f844d:::benchmark_data_smithi047_474001_object350:head# 2024-09-18T16:00:54.826 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:22d3be20:::benchmark_data_smithi047_474001_object315:head# 2024-09-18T16:00:54.826 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2334c1fe:::benchmark_data_smithi047_474001_object84:head# 2024-09-18T16:00:54.826 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:24393629:::benchmark_data_smithi047_474001_object239:head# 2024-09-18T16:00:54.826 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:25482445:::benchmark_data_smithi047_474001_object53:head# 2024-09-18T16:00:54.826 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2555ae90:::benchmark_data_smithi047_474001_object370:head# 2024-09-18T16:00:54.827 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:26a1c85e:::benchmark_data_smithi047_474001_object32:head# 2024-09-18T16:00:54.827 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:26b7b87f:::benchmark_data_smithi047_474001_object107:head# 2024-09-18T16:00:54.827 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:27190702:::benchmark_data_smithi047_474001_object218:head# 2024-09-18T16:00:54.827 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:27f9a01e:::benchmark_data_smithi047_474001_object349:head# 2024-09-18T16:00:54.828 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:284c9f3a:::benchmark_data_smithi047_474001_object254:head# 2024-09-18T16:00:54.828 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2856b42f:::benchmark_data_smithi047_474001_object81:head# 2024-09-18T16:00:54.828 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:29494f84:::benchmark_data_smithi047_474001_object248:head# 2024-09-18T16:00:54.829 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2a2243e8:::benchmark_data_smithi047_474001_object1:head# 2024-09-18T16:00:54.829 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2abea481:::benchmark_data_smithi047_474001_object23:head# 2024-09-18T16:00:54.830 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2ced8b80:::benchmark_data_smithi047_474001_object401:head# 2024-09-18T16:00:54.830 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2e2d9e2b:::benchmark_data_smithi047_474001_object367:head# 2024-09-18T16:00:54.831 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2e9ea224:::benchmark_data_smithi047_474001_object106:head# 2024-09-18T16:00:54.831 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2fb8dbb3:::benchmark_data_smithi047_474001_object170:head# 2024-09-18T16:00:54.831 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:300fdc07:::benchmark_data_smithi047_474001_object227:head# 2024-09-18T16:00:54.833 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3078741e:::benchmark_data_smithi047_474001_object139:head# 2024-09-18T16:00:54.833 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:31669bfe:::benchmark_data_smithi047_474001_object319:head# 2024-09-18T16:00:54.833 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:321ae169:::benchmark_data_smithi047_474001_object244:head# 2024-09-18T16:00:54.833 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:32ff3ac8:::benchmark_data_smithi047_474001_object261:head# 2024-09-18T16:00:54.833 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:33134995:::benchmark_data_smithi047_474001_object310:head# 2024-09-18T16:00:54.833 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:33b1fdf8:::benchmark_data_smithi047_474001_object161:head# 2024-09-18T16:00:54.834 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:33b93a31:::benchmark_data_smithi047_474001_object403:head# 2024-09-18T16:00:54.834 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:33fa48ba:::benchmark_data_smithi047_474001_object289:head# 2024-09-18T16:00:54.834 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:34348ae4:::benchmark_data_smithi047_474001_object6:head# 2024-09-18T16:00:54.835 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:34e0f433:::benchmark_data_smithi047_474001_object342:head# 2024-09-18T16:00:54.835 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:355eddbc:::benchmark_data_smithi047_474001_object198:head# 2024-09-18T16:00:54.836 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:35841d04:::benchmark_data_smithi047_474001_object186:head# 2024-09-18T16:00:54.836 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:35c32413:::benchmark_data_smithi047_474001_object413:head# 2024-09-18T16:00:54.836 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:35e67788:::benchmark_data_smithi047_474001_object388:head# 2024-09-18T16:00:54.836 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3613c729:::benchmark_data_smithi047_474001_object380:head# 2024-09-18T16:00:54.837 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:37ccfdd9:::benchmark_data_smithi047_474001_object219:head# 2024-09-18T16:00:54.838 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:380af478:::benchmark_data_smithi047_474001_object359:head# 2024-09-18T16:00:54.838 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3812ffca:::benchmark_data_smithi047_474001_object286:head# 2024-09-18T16:00:54.839 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:381ae33c:::benchmark_data_smithi047_474001_object152:head# 2024-09-18T16:00:54.839 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3994ed2b:::benchmark_data_smithi047_474001_object162:head# 2024-09-18T16:00:54.839 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3ad1e94f:::benchmark_data_smithi047_474001_object26:head# 2024-09-18T16:00:54.839 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3b2460a7:::benchmark_data_smithi047_474001_object418:head# 2024-09-18T16:00:54.840 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3b70d910:::benchmark_data_smithi047_474001_object157:head# 2024-09-18T16:00:54.840 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3b976a95:::benchmark_data_smithi047_474001_object392:head# 2024-09-18T16:00:54.840 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3c01f944:::benchmark_data_smithi047_474001_object85:head# 2024-09-18T16:00:54.841 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3c08c862:::benchmark_data_smithi047_474001_object209:head# 2024-09-18T16:00:54.841 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3c3b4221:::benchmark_data_smithi047_474001_object302:head# 2024-09-18T16:00:54.841 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3e2bc2d1:::benchmark_data_smithi047_474001_object290:head# 2024-09-18T16:00:54.842 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3f0792f2:::benchmark_data_smithi047_474001_object205:head# 2024-09-18T16:00:54.842 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3f0bbc7e:::benchmark_data_smithi047_474001_object214:head# 2024-09-18T16:00:54.842 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:40e84920:::benchmark_data_smithi047_474001_object201:head# 2024-09-18T16:00:54.843 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:410710e5:::benchmark_data_smithi047_474001_object360:head# 2024-09-18T16:00:54.843 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4286a502:::benchmark_data_smithi047_474001_object280:head# 2024-09-18T16:00:54.844 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:429b03e9:::benchmark_data_smithi047_474001_object249:head# 2024-09-18T16:00:54.844 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:43c8723a:::benchmark_data_smithi047_474001_object4:head# 2024-09-18T16:00:54.845 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:43c8e265:::benchmark_data_smithi047_474001_object208:head# 2024-09-18T16:00:54.845 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:43d378b4:::benchmark_data_smithi047_474001_object425:head# 2024-09-18T16:00:54.845 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:43fb4408:::benchmark_data_smithi047_474001_object399:head# 2024-09-18T16:00:54.846 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:44ab53e5:::benchmark_data_smithi047_474001_object27:head# 2024-09-18T16:00:54.847 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4515f62e:::benchmark_data_smithi047_474001_object143:head# 2024-09-18T16:00:54.847 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:457b93bc:::benchmark_data_smithi047_474001_object300:head# 2024-09-18T16:00:54.847 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:457e7435:::benchmark_data_smithi047_474001_object231:head# 2024-09-18T16:00:54.848 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:468b56fb:::benchmark_data_smithi047_474001_object242:head# 2024-09-18T16:00:54.848 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:47454808:::benchmark_data_smithi047_474001_object394:head# 2024-09-18T16:00:54.848 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4757d830:::benchmark_data_smithi047_474001_object147:head# 2024-09-18T16:00:54.849 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4790fa42:::benchmark_data_smithi047_474001_object16:head# 2024-09-18T16:00:54.849 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:481edef6:::benchmark_data_smithi047_474001_object129:head# 2024-09-18T16:00:54.849 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:498127df:::benchmark_data_smithi047_474001_object409:head# 2024-09-18T16:00:54.850 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4a94b2bf:::benchmark_data_smithi047_474001_object331:head# 2024-09-18T16:00:54.851 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4b0f17cd:::benchmark_data_smithi047_474001_object194:head# 2024-09-18T16:00:54.851 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4ccceada:::benchmark_data_smithi047_474001_object213:head# 2024-09-18T16:00:54.851 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4d145d82:::benchmark_data_smithi047_474001_object206:head# 2024-09-18T16:00:54.851 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4e67dcba:::benchmark_data_smithi047_474001_object144:head# 2024-09-18T16:00:54.851 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4e6bc7a4:::benchmark_data_smithi047_474001_object165:head# 2024-09-18T16:00:54.851 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4e7fe7b2:::benchmark_data_smithi047_474001_object431:head# 2024-09-18T16:00:54.852 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4f5e76df:::benchmark_data_smithi047_474001_object415:head# 2024-09-18T16:00:54.852 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5030ebeb:::benchmark_data_smithi047_474001_object335:head# 2024-09-18T16:00:54.853 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:505987dd:::benchmark_data_smithi047_474001_object313:head# 2024-09-18T16:00:54.854 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:50a1999f:::benchmark_data_smithi047_474001_object190:head# 2024-09-18T16:00:54.854 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:51905d78:::benchmark_data_smithi047_474001_object371:head# 2024-09-18T16:00:54.854 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:53516fa7:::benchmark_data_smithi047_474001_object47:head# 2024-09-18T16:00:54.855 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:53a75224:::benchmark_data_smithi047_474001_object277:head# 2024-09-18T16:00:54.856 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:53bc5374:::benchmark_data_smithi047_474001_object304:head# 2024-09-18T16:00:54.856 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:54037397:::benchmark_data_smithi047_474001_object168:head# 2024-09-18T16:00:54.856 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:54e93f24:::benchmark_data_smithi047_474001_object351:head# 2024-09-18T16:00:54.856 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5549c508:::benchmark_data_smithi047_474001_object154:head# 2024-09-18T16:00:54.856 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:55a8f195:::benchmark_data_smithi047_474001_object183:head# 2024-09-18T16:00:54.856 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:58bb18eb:::benchmark_data_smithi047_474001_object92:head# 2024-09-18T16:00:54.857 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:58e3824a:::benchmark_data_smithi047_474001_object22:head# 2024-09-18T16:00:54.857 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:58eacbbf:::benchmark_data_smithi047_474001_object100:head# 2024-09-18T16:00:54.857 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:590d7cd0:::benchmark_data_smithi047_474001_object328:head# 2024-09-18T16:00:54.857 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:59701efb:::benchmark_data_smithi047_474001_object243:head# 2024-09-18T16:00:54.858 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:59852438:::benchmark_data_smithi047_474001_object83:head# 2024-09-18T16:00:54.858 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5a123740:::benchmark_data_smithi047_474001_object36:head# 2024-09-18T16:00:54.859 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5b73d6c3:::benchmark_data_smithi047_474001_object284:head# 2024-09-18T16:00:54.859 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5c8e3731:::benchmark_data_smithi047_474001_object132:head# 2024-09-18T16:00:54.859 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5caa7c9f:::benchmark_data_smithi047_474001_object416:head# 2024-09-18T16:00:54.859 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5cf5ea91:::benchmark_data_smithi047_474001_object397:head# 2024-09-18T16:00:54.860 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5d1b58ad:::benchmark_data_smithi047_474001_object269:head# 2024-09-18T16:00:54.860 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5e669bb0:::benchmark_data_smithi047_474001_object405:head# 2024-09-18T16:00:54.860 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5eac6d5a:::benchmark_data_smithi047_474001_object411:head# 2024-09-18T16:00:54.861 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5ef5c1d9:::benchmark_data_smithi047_474001_object385:head# 2024-09-18T16:00:54.861 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5f7398fd:::benchmark_data_smithi047_474001_object68:head# 2024-09-18T16:00:54.862 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5f8ba50b:::benchmark_data_smithi047_474001_object174:head# 2024-09-18T16:00:54.862 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5fb38f21:::benchmark_data_smithi047_474001_object75:head# 2024-09-18T16:00:54.862 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:60a5ff1a:::benchmark_data_smithi047_474001_object298:head# 2024-09-18T16:00:54.862 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6159ad07:::benchmark_data_smithi047_474001_object221:head# 2024-09-18T16:00:54.863 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:62273e63:::benchmark_data_smithi047_474001_object62:head# 2024-09-18T16:00:54.863 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:63cf18a2:::benchmark_data_smithi047_474001_object272:head# 2024-09-18T16:00:54.863 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:63f415e6:::benchmark_data_smithi047_474001_object58:head# 2024-09-18T16:00:54.864 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:64ff073d:::benchmark_data_smithi047_474001_object362:head# 2024-09-18T16:00:54.864 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:657bccc4:::benchmark_data_smithi047_474001_object169:head# 2024-09-18T16:00:54.864 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:660836d4:::benchmark_data_smithi047_474001_object39:head# 2024-09-18T16:00:54.864 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:66682ebd:::benchmark_data_smithi047_474001_object101:head# 2024-09-18T16:00:54.866 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:66e730e2:::benchmark_data_smithi047_474001_object61:head# 2024-09-18T16:00:54.866 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:66fbc3ec:::benchmark_data_smithi047_474001_object71:head# 2024-09-18T16:00:54.866 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6717adea:::benchmark_data_smithi047_474001_object317:head# 2024-09-18T16:00:54.866 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:68107ec1:::benchmark_data_smithi047_474001_object195:head# 2024-09-18T16:00:54.866 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6812f8cb:::benchmark_data_smithi047_474001_object264:head# 2024-09-18T16:00:54.867 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:68f519f4:::benchmark_data_smithi047_474001_object343:head# 2024-09-18T16:00:54.867 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6a39a8d4:::benchmark_data_smithi047_474001_object66:head# 2024-09-18T16:00:54.867 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6ab23d26:::benchmark_data_smithi047_474001_object9:head# 2024-09-18T16:00:54.867 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6b25d55a:::benchmark_data_smithi047_474001_object72:head# 2024-09-18T16:00:54.867 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6b5fc221:::benchmark_data_smithi047_474001_object334:head# 2024-09-18T16:00:54.868 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6c2d825c:::benchmark_data_smithi047_474001_object5:head# 2024-09-18T16:00:54.869 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6caa7eaf:::benchmark_data_smithi047_474001_object263:head# 2024-09-18T16:00:54.869 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6dfd6f4e:::benchmark_data_smithi047_474001_object251:head# 2024-09-18T16:00:54.869 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6e9248a9:::benchmark_data_smithi047_474001_object245:head# 2024-09-18T16:00:54.870 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6f58cdd1:::benchmark_data_smithi047_474001_object430:head# 2024-09-18T16:00:54.870 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:71084a82:::benchmark_data_smithi047_474001_object398:head# 2024-09-18T16:00:54.871 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:722633c0:::benchmark_data_smithi047_474001_object7:head# 2024-09-18T16:00:54.871 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:73078ec4:::benchmark_data_smithi047_474001_object200:head# 2024-09-18T16:00:54.871 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7319939a:::benchmark_data_smithi047_474001_object378:head# 2024-09-18T16:00:54.872 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:732bbbc2:::benchmark_data_smithi047_474001_object113:head# 2024-09-18T16:00:54.872 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:73694cfd:::benchmark_data_smithi047_474001_object257:head# 2024-09-18T16:00:54.873 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:739a0e00:::benchmark_data_smithi047_474001_object11:head# 2024-09-18T16:00:54.873 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:748a50de:::benchmark_data_smithi047_474001_object223:head# 2024-09-18T16:00:54.873 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:74cbfe52:::benchmark_data_smithi047_474001_object309:head# 2024-09-18T16:00:54.874 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:751b68bd:::benchmark_data_smithi047_474001_object45:head# 2024-09-18T16:00:54.875 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:76583154:::benchmark_data_smithi047_474001_object13:head# 2024-09-18T16:00:54.875 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7669c90e:::benchmark_data_smithi047_474001_object216:head# 2024-09-18T16:00:54.875 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:76dc48ec:::benchmark_data_smithi047_474001_object79:head# 2024-09-18T16:00:54.875 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:77226950:::benchmark_data_smithi047_474001_object386:head# 2024-09-18T16:00:54.876 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:77a33037:::benchmark_data_smithi047_474001_object197:head# 2024-09-18T16:00:54.876 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:77f7974a:::benchmark_data_smithi047_474001_object123:head# 2024-09-18T16:00:54.876 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:78d90d4e:::benchmark_data_smithi047_474001_object301:head# 2024-09-18T16:00:54.876 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7b36e97b:::benchmark_data_smithi047_474001_object292:head# 2024-09-18T16:00:54.877 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7c8233bd:::benchmark_data_smithi047_474001_object76:head# 2024-09-18T16:00:54.877 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7cce9f64:::benchmark_data_smithi047_474001_object0:head# 2024-09-18T16:00:54.877 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7d8e31eb:::benchmark_data_smithi047_474001_object193:head# 2024-09-18T16:00:54.877 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7dc806f2:::benchmark_data_smithi047_474001_object126:head# 2024-09-18T16:00:54.878 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7dfaf25d:::benchmark_data_smithi047_474001_object260:head# 2024-09-18T16:00:54.878 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7e00f5f7:::benchmark_data_smithi047_474001_object98:head# 2024-09-18T16:00:54.879 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7f679bd9:::benchmark_data_smithi047_474001_object379:head# 2024-09-18T16:00:54.879 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7fa61220:::benchmark_data_smithi047_474001_object354:head# 2024-09-18T16:00:54.879 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7fb32424:::benchmark_data_smithi047_474001_object241:head# 2024-09-18T16:00:54.880 INFO:tasks.workunit.client.0.smithi047.stderr:Export successful 2024-09-18T16:00:55.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:102: TEST_import_after_merge_and_gap: activate_osd td/pg-split-merge 0 2024-09-18T16:00:55.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-09-18T16:00:55.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T16:00:55.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T16:00:55.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T16:00:55.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/0 2024-09-18T16:00:55.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1a671e43-8866-44a5-abe3-c1e1f4ff765b --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-18T16:00:55.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:00:55.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:00:55.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:00:55.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-09-18T16:00:55.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-09-18T16:00:55.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T16:00:55.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T16:00:55.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-09-18T16:00:55.149 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T16:00:55.149 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:00:55.149 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:00:55.149 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:00:55.149 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:00:55.149 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.463129 2024-09-18T16:00:55.149 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.463129/$cluster-$name.asok' 2024-09-18T16:00:55.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.463129/$cluster-$name.asok' 2024-09-18T16:00:55.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:00:55.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-09-18T16:00:55.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-09-18T16:00:55.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:00:55.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:00:55.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:00:55.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:00:55.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T16:00:55.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T16:00:55.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/0 2024-09-18T16:00:55.151 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T16:00:55.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T16:00:55.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1a671e43-8866-44a5-abe3-c1e1f4ff765b --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.463129/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:00:55.152 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/0/whoami 2024-09-18T16:00:55.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T16:00:55.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T16:00:55.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T16:00:55.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T16:00:55.177 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:00:55.175+0000 7fef31a0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:00:55.179 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:00:55.178+0000 7fef31a0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:00:55.180 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:00:55.179+0000 7fef31a0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:00:55.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T16:00:55.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:00:55.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T16:00:55.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:00:55.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:00:55.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:55.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:00:55.525 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:00:55.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:55.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:55.694 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:00:55.693+0000 7fef31a0b440 -1 Falling back to public interface 2024-09-18T16:00:55.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:56.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:56.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:56.932 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T16:00:56.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:00:56.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:56.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:57.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:00:58.035 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:00:58.034+0000 7fef31a0b440 -1 osd.0 9 log_to_monitors true 2024-09-18T16:00:58.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:00:58.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:00:58.310 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T16:00:58.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:00:58.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:00:58.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:00:58.781 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 12 up_thru 12 down_at 10 last_clean_interval [5,9) [v2:127.0.0.1:6802/474816,v1:127.0.0.1:6803/474816] [v2:127.0.0.1:6804/474816,v1:127.0.0.1:6805/474816] exists,up 1b6f27f6-2f00-44c7-8d93-1f3fa5ab4594 2024-09-18T16:00:58.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:00:58.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:00:58.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:00:58.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:104: TEST_import_after_merge_and_gap: ceph osd pool set foo pg_num 1 2024-09-18T16:00:59.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:105: TEST_import_after_merge_and_gap: sleep 5 2024-09-18T16:01:04.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:106: TEST_import_after_merge_and_gap: ceph daemon osd.0 perf dump 2024-09-18T16:01:04.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:106: TEST_import_after_merge_and_gap: jq .osd.numpg 2024-09-18T16:01:04.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:106: TEST_import_after_merge_and_gap: grep 2 2024-09-18T16:01:04.547 INFO:tasks.workunit.client.0.smithi047.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-18T16:01:04.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:107: TEST_import_after_merge_and_gap: wait_for_clean 2024-09-18T16:01:04.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:01:04.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:01:04.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:01:04.549 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:01:04.549 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:01:04.549 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:01:04.549 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:01:04.549 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:01:04.550 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:01:04.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T16:01:04.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:01:04.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:01:04.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:01:04.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:01:04.641 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:01:05.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T16:01:05.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:01:05.006 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:01:05.006 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:01:05.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=51539607556 2024-09-18T16:01:05.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 51539607556 2024-09-18T16:01:05.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-51539607556' 2024-09-18T16:01:05.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:01:05.116 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-51539607556 2024-09-18T16:01:05.116 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:01:05.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:01:05.117 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-51539607556 2024-09-18T16:01:05.117 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:01:05.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=51539607556 2024-09-18T16:01:05.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 51539607556' 2024-09-18T16:01:05.118 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 51539607556 2024-09-18T16:01:05.118 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:01:05.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 51539607554 -lt 51539607556 2024-09-18T16:01:05.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:01:06.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:01:06.522 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:01:06.930 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 51539607556 -lt 51539607556 2024-09-18T16:01:06.931 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:01:06.931 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:01:06.931 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:01:07.432 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:01:07.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:01:07.433 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:01:07.433 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:01:07.433 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:01:07.433 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:01:07.433 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:01:07.433 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:01:07.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:01:07.836 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:01:07.836 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:01:07.836 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:01:08.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:01:08.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:01:08.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:01:08.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:110: TEST_import_after_merge_and_gap: kill_daemons td/pg-split-merge TERM osd.0 2024-09-18T16:01:08.369 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:01:08.369 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:01:08.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:01:08.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:01:08.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:01:08.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:01:08.674 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:111: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op remove --pgid 1.0 --force 2024-09-18T16:01:09.877 INFO:tasks.workunit.client.0.smithi047.stdout: marking collection for removal 2024-09-18T16:01:09.877 INFO:tasks.workunit.client.0.smithi047.stdout:setting '_remove' omap key 2024-09-18T16:01:09.877 INFO:tasks.workunit.client.0.smithi047.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-09-18T16:01:10.537 INFO:tasks.workunit.client.0.smithi047.stdout:Remove successful 2024-09-18T16:01:10.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:113: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.1 --file td/pg-split-merge/1.1 2024-09-18T16:01:11.840 INFO:tasks.workunit.client.0.smithi047.stdout:get_pg_num_history pg_num_history pg_num_history(e20 pg_nums {1={7=2,19=1}} deleted_pools ) 2024-09-18T16:01:11.840 INFO:tasks.workunit.client.0.smithi047.stdout:Importing pgid 1.1 2024-09-18T16:01:11.841 INFO:tasks.workunit.client.0.smithi047.stdout:pool 1 pg_num 2 -> 1 2024-09-18T16:01:11.841 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:806a8ff0:::benchmark_data_smithi047_474001_object14:head# 2024-09-18T16:01:11.841 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:11.957 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:827184ed:::benchmark_data_smithi047_474001_object202:head# 2024-09-18T16:01:11.957 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:11.999 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:83cb4be8:::benchmark_data_smithi047_474001_object189:head# 2024-09-18T16:01:11.999 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:12.058 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:83d7b3d6:::benchmark_data_smithi047_474001_object427:head# 2024-09-18T16:01:12.058 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:12.125 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:83e6a7ba:::benchmark_data_smithi047_474001_object326:head# 2024-09-18T16:01:12.125 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:12.217 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:83e714c1:::benchmark_data_smithi047_474001_object384:head# 2024-09-18T16:01:12.217 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:12.275 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:84fc4a07:::benchmark_data_smithi047_474001_object235:head# 2024-09-18T16:01:12.275 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:12.326 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8635a002:::benchmark_data_smithi047_474001_object8:head# 2024-09-18T16:01:12.326 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:12.359 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8659e9ad:::benchmark_data_smithi047_474001_object111:head# 2024-09-18T16:01:12.359 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:12.392 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:865f4a69:::benchmark_data_smithi047_474001_object312:head# 2024-09-18T16:01:12.392 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:12.426 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:87201b49:::benchmark_data_smithi047_474001_object185:head# 2024-09-18T16:01:12.426 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:12.459 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:889f60a0:::benchmark_data_smithi047_474001_object109:head# 2024-09-18T16:01:12.459 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:12.526 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8904e269:::benchmark_data_smithi047_474001_object228:head# 2024-09-18T16:01:12.526 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:12.576 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:89ec4c33:::benchmark_data_smithi047_474001_object133:head# 2024-09-18T16:01:12.577 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:12.627 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8a2258fa:::benchmark_data_smithi047_474001_object119:head# 2024-09-18T16:01:12.627 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:12.660 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8aba1a3a:::benchmark_data_smithi047_474001_object291:head# 2024-09-18T16:01:12.660 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:12.685 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8cb0b5a1:::benchmark_data_smithi047_474001_object3:head# 2024-09-18T16:01:12.685 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:12.710 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8d65845a:::benchmark_data_smithi047_474001_object324:head# 2024-09-18T16:01:12.711 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:12.735 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8e21472c:::benchmark_data_smithi047_474001_object110:head# 2024-09-18T16:01:12.736 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:12.769 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8efa467b:::benchmark_data_smithi047_474001_object247:head# 2024-09-18T16:01:12.769 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:12.794 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8f162f27:::benchmark_data_smithi047_474001_object222:head# 2024-09-18T16:01:12.794 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:12.819 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8f2ab4de:::benchmark_data_smithi047_474001_object112:head# 2024-09-18T16:01:12.819 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:12.853 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8f3016de:::benchmark_data_smithi047_474001_object337:head# 2024-09-18T16:01:12.853 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:12.886 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:906470c5:::benchmark_data_smithi047_474001_object108:head# 2024-09-18T16:01:12.886 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:12.970 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:90a988ef:::benchmark_data_smithi047_474001_object391:head# 2024-09-18T16:01:12.970 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:12.995 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:91461d4c:::benchmark_data_smithi047_474001_object252:head# 2024-09-18T16:01:12.995 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:13.020 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:92438533:::benchmark_data_smithi047_474001_object17:head# 2024-09-18T16:01:13.020 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:13.045 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9256e853:::benchmark_data_smithi047_474001_object314:head# 2024-09-18T16:01:13.045 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:13.070 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:928bb93a:::benchmark_data_smithi047_474001_object51:head# 2024-09-18T16:01:13.071 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:13.096 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:92b0a2c6:::benchmark_data_smithi047_474001_object250:head# 2024-09-18T16:01:13.096 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:13.121 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9324d622:::benchmark_data_smithi047_474001_object95:head# 2024-09-18T16:01:13.121 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:13.146 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:939c1aa8:::benchmark_data_smithi047_474001_object382:head# 2024-09-18T16:01:13.146 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:13.171 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:945bc203:::benchmark_data_smithi047_474001_object177:head# 2024-09-18T16:01:13.171 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:13.196 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:94707a46:::benchmark_data_smithi047_474001_object138:head# 2024-09-18T16:01:13.196 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:13.221 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:94f79b31:::benchmark_data_smithi047_474001_object253:head# 2024-09-18T16:01:13.221 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:13.246 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9502e979:::benchmark_data_smithi047_474001_object50:head# 2024-09-18T16:01:13.246 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:13.271 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:953c3a4c:::benchmark_data_smithi047_474001_object74:head# 2024-09-18T16:01:13.271 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:13.296 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:95edf5ad:::benchmark_data_smithi047_474001_object345:head# 2024-09-18T16:01:13.296 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:13.338 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9612cb9d:::benchmark_data_smithi047_474001_object59:head# 2024-09-18T16:01:13.338 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:13.372 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9670fdab:::benchmark_data_smithi047_474001_object116:head# 2024-09-18T16:01:13.372 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:13.413 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:978a53fd:::benchmark_data_smithi047_474001_object358:head# 2024-09-18T16:01:13.414 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:13.447 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:986d3d20:::benchmark_data_smithi047_474001_object38:head# 2024-09-18T16:01:13.447 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:13.480 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9a0edcb5:::benchmark_data_smithi047_474001_object89:head# 2024-09-18T16:01:13.481 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:13.514 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9bef3576:::benchmark_data_smithi047_474001_object180:head# 2024-09-18T16:01:13.514 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:13.547 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9dafb813:::benchmark_data_smithi047_474001_object226:head# 2024-09-18T16:01:13.547 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:13.581 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9dc7b4a6:::benchmark_data_smithi047_474001_object346:head# 2024-09-18T16:01:13.581 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:13.614 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9de9d04a:::benchmark_data_smithi047_474001_object338:head# 2024-09-18T16:01:13.614 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:13.648 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9e17b853:::benchmark_data_smithi047_474001_object124:head# 2024-09-18T16:01:13.648 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:13.681 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9e534a18:::benchmark_data_smithi047_474001_object70:head# 2024-09-18T16:01:13.681 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:13.715 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9e5ea7f3:::benchmark_data_smithi047_474001_object348:head# 2024-09-18T16:01:13.715 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:13.748 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a0254336:::benchmark_data_smithi047_474001_object188:head# 2024-09-18T16:01:13.748 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:13.781 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a0c154fe:::benchmark_data_smithi047_474001_object383:head# 2024-09-18T16:01:13.781 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:13.815 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a1981567:::benchmark_data_smithi047_474001_object414:head# 2024-09-18T16:01:13.815 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:13.848 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a23c3345:::benchmark_data_smithi047_474001_object408:head# 2024-09-18T16:01:13.848 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:13.882 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a244fe5e:::benchmark_data_smithi047_474001_object307:head# 2024-09-18T16:01:13.882 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:13.915 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a256b83b:::benchmark_data_smithi047_474001_object417:head# 2024-09-18T16:01:13.915 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:13.949 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a331eaa2:::benchmark_data_smithi047_474001_object103:head# 2024-09-18T16:01:13.949 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:13.982 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a36fdbde:::benchmark_data_smithi047_474001_object125:head# 2024-09-18T16:01:13.982 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:14.016 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a388d7f6:::benchmark_data_smithi047_474001_object46:head# 2024-09-18T16:01:14.016 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:14.049 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a473749a:::benchmark_data_smithi047_474001_object396:head# 2024-09-18T16:01:14.049 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:14.083 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a546fc38:::benchmark_data_smithi047_474001_object344:head# 2024-09-18T16:01:14.083 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:14.141 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a56684c3:::benchmark_data_smithi047_474001_object73:head# 2024-09-18T16:01:14.141 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:14.367 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a6a637ca:::benchmark_data_smithi047_474001_object352:head# 2024-09-18T16:01:14.367 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:14.392 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a6ac7233:::benchmark_data_smithi047_474001_object273:head# 2024-09-18T16:01:14.392 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:14.417 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a6be61c0:::benchmark_data_smithi047_474001_object212:head# 2024-09-18T16:01:14.417 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:14.450 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a6d03d77:::benchmark_data_smithi047_474001_object429:head# 2024-09-18T16:01:14.450 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:14.509 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a7029afc:::benchmark_data_smithi047_474001_object57:head# 2024-09-18T16:01:14.509 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:14.559 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a87abb60:::benchmark_data_smithi047_474001_object28:head# 2024-09-18T16:01:14.559 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:14.626 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a99c9566:::benchmark_data_smithi047_474001_object285:head# 2024-09-18T16:01:14.626 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:14.652 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a9ee7f09:::benchmark_data_smithi047_474001_object60:head# 2024-09-18T16:01:14.652 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:14.676 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:aaf069cf:::benchmark_data_smithi047_474001_object426:head# 2024-09-18T16:01:14.677 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:14.702 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ac9a95a5:::benchmark_data_smithi047_474001_object327:head# 2024-09-18T16:01:14.702 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:14.735 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ad75c20a:::benchmark_data_smithi047_474001_object322:head# 2024-09-18T16:01:14.735 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:14.769 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:add97b11:::benchmark_data_smithi047_474001_object120:head# 2024-09-18T16:01:14.769 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:14.802 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ae2bc6da:::benchmark_data_smithi047_474001_object374:head# 2024-09-18T16:01:14.803 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:14.836 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ae32eaa9:::benchmark_data_smithi047_474001_object217:head# 2024-09-18T16:01:14.836 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:14.869 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ae5f08a5:::benchmark_data_smithi047_474001_object156:head# 2024-09-18T16:01:14.869 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:14.903 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ae90731a:::benchmark_data_smithi047_474001_object275:head# 2024-09-18T16:01:14.903 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:14.936 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:aeb96a61:::benchmark_data_smithi047_474001_object35:head# 2024-09-18T16:01:14.936 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:14.969 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:af8861c2:::benchmark_data_smithi047_474001_object19:head# 2024-09-18T16:01:14.969 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:15.003 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:afa79d73:::benchmark_data_smithi047_474001_object297:head# 2024-09-18T16:01:15.003 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:15.036 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b06f5ff6:::benchmark_data_smithi047_474001_object96:head# 2024-09-18T16:01:15.036 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:15.078 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b09fcb0b:::benchmark_data_smithi047_474001_object330:head# 2024-09-18T16:01:15.078 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:15.112 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b0f4cb26:::benchmark_data_smithi047_474001_object262:head# 2024-09-18T16:01:15.112 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:15.145 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b1589877:::benchmark_data_smithi047_474001_object332:head# 2024-09-18T16:01:15.145 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:15.179 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b1dd1fef:::benchmark_data_smithi047_474001_object134:head# 2024-09-18T16:01:15.179 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:15.212 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b3f3163f:::benchmark_data_smithi047_474001_object341:head# 2024-09-18T16:01:15.212 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:15.245 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b5265e20:::benchmark_data_smithi047_474001_object159:head# 2024-09-18T16:01:15.245 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:15.279 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b533e7e6:::benchmark_data_smithi047_474001_object78:head# 2024-09-18T16:01:15.279 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:15.313 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b585c96f:::benchmark_data_smithi047_474001_object422:head# 2024-09-18T16:01:15.313 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:15.346 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b5e588a2:::benchmark_data_smithi047_474001_object30:head# 2024-09-18T16:01:15.346 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:15.371 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b606ef85:::benchmark_data_smithi047_474001_object12:head# 2024-09-18T16:01:15.371 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:15.396 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b61ac151:::benchmark_data_smithi047_474001_object361:head# 2024-09-18T16:01:15.396 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:15.430 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b707ad78:::benchmark_data_smithi047_474001_object33:head# 2024-09-18T16:01:15.430 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:15.463 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b72090f3:::benchmark_data_smithi047_474001_object225:head# 2024-09-18T16:01:15.463 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:15.488 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b74f9085:::benchmark_data_smithi047_474001_object224:head# 2024-09-18T16:01:15.488 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:15.522 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b854e709:::benchmark_data_smithi047_474001_object279:head# 2024-09-18T16:01:15.522 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:15.555 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b8615e71:::benchmark_last_metadata:head# 2024-09-18T16:01:15.555 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:15.588 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b88427b9:::benchmark_data_smithi047_474001_object204:head# 2024-09-18T16:01:15.589 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:15.622 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b8a01441:::benchmark_data_smithi047_474001_object99:head# 2024-09-18T16:01:15.622 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:15.664 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b8d632ef:::benchmark_data_smithi047_474001_object49:head# 2024-09-18T16:01:15.664 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:15.705 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b9a03ab3:::benchmark_data_smithi047_474001_object41:head# 2024-09-18T16:01:15.706 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:15.739 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ba233a46:::benchmark_data_smithi047_474001_object368:head# 2024-09-18T16:01:15.739 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:15.772 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:bb9dcf18:::benchmark_data_smithi047_474001_object306:head# 2024-09-18T16:01:15.772 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:15.806 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:bc2e5e68:::benchmark_data_smithi047_474001_object24:head# 2024-09-18T16:01:15.806 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:15.840 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:bd291308:::benchmark_data_smithi047_474001_object316:head# 2024-09-18T16:01:15.840 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:15.873 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:bd68e671:::benchmark_data_smithi047_474001_object44:head# 2024-09-18T16:01:15.873 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:15.906 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:bdd1c304:::benchmark_data_smithi047_474001_object237:head# 2024-09-18T16:01:15.906 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:15.940 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:be3f8098:::benchmark_data_smithi047_474001_object423:head# 2024-09-18T16:01:15.940 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:15.982 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:be77543b:::benchmark_data_smithi047_474001_object77:head# 2024-09-18T16:01:15.982 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:16.023 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:be86e7ec:::benchmark_data_smithi047_474001_object122:head# 2024-09-18T16:01:16.023 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:16.057 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:bed16cbf:::benchmark_data_smithi047_474001_object137:head# 2024-09-18T16:01:16.057 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:16.110 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:bf89fe68:::benchmark_data_smithi047_474001_object54:head# 2024-09-18T16:01:16.110 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:16.157 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c08db5a2:::benchmark_data_smithi047_474001_object236:head# 2024-09-18T16:01:16.157 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:16.207 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c18ab8fc:::benchmark_data_smithi047_474001_object131:head# 2024-09-18T16:01:16.207 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:16.241 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c1fcb644:::benchmark_data_smithi047_474001_object158:head# 2024-09-18T16:01:16.241 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:16.274 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c24a7c46:::benchmark_data_smithi047_474001_object282:head# 2024-09-18T16:01:16.274 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:16.308 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c313dcbc:::benchmark_data_smithi047_474001_object321:head# 2024-09-18T16:01:16.308 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:16.341 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c35a5205:::benchmark_data_smithi047_474001_object303:head# 2024-09-18T16:01:16.341 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:16.375 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c3d61783:::benchmark_data_smithi047_474001_object288:head# 2024-09-18T16:01:16.375 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:16.408 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c3fe223d:::benchmark_data_smithi047_474001_object356:head# 2024-09-18T16:01:16.408 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:16.442 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c4b79889:::benchmark_data_smithi047_474001_object325:head# 2024-09-18T16:01:16.442 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:16.475 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c60c9d19:::benchmark_data_smithi047_474001_object146:head# 2024-09-18T16:01:16.476 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:16.542 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c6123b6f:::benchmark_data_smithi047_474001_object55:head# 2024-09-18T16:01:16.542 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:16.592 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c6869eae:::benchmark_data_smithi047_474001_object407:head# 2024-09-18T16:01:16.592 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:16.651 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c849831c:::benchmark_data_smithi047_474001_object118:head# 2024-09-18T16:01:16.651 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:16.676 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c9eb3ea0:::benchmark_data_smithi047_474001_object87:head# 2024-09-18T16:01:16.676 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:16.701 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ca4e33a3:::benchmark_data_smithi047_474001_object283:head# 2024-09-18T16:01:16.701 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:16.726 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cb6747e8:::benchmark_data_smithi047_474001_object357:head# 2024-09-18T16:01:16.726 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:16.760 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cb84b5d4:::benchmark_data_smithi047_474001_object173:head# 2024-09-18T16:01:16.760 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:16.785 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cbdaedd2:::benchmark_data_smithi047_474001_object179:head# 2024-09-18T16:01:16.785 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:16.818 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cd271f33:::benchmark_data_smithi047_474001_object166:head# 2024-09-18T16:01:16.818 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:16.852 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cd28a341:::benchmark_data_smithi047_474001_object150:head# 2024-09-18T16:01:16.852 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:16.885 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cd4d6810:::benchmark_data_smithi047_474001_object363:head# 2024-09-18T16:01:16.886 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:16.919 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cd5cc2dc:::benchmark_data_smithi047_474001_object88:head# 2024-09-18T16:01:16.919 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:16.952 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cd77e455:::benchmark_data_smithi047_474001_object404:head# 2024-09-18T16:01:16.952 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:16.986 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cdedbf36:::benchmark_data_smithi047_474001_object155:head# 2024-09-18T16:01:16.986 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:17.019 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d007de6f:::benchmark_data_smithi047_474001_object151:head# 2024-09-18T16:01:17.019 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:17.053 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d275e6f0:::benchmark_data_smithi047_474001_object419:head# 2024-09-18T16:01:17.053 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:17.086 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d295be0f:::benchmark_data_smithi047_474001_object167:head# 2024-09-18T16:01:17.086 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:17.119 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d2992f10:::benchmark_data_smithi047_474001_object175:head# 2024-09-18T16:01:17.120 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:17.153 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d2cdad48:::benchmark_data_smithi047_474001_object90:head# 2024-09-18T16:01:17.153 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:17.195 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d43afad7:::benchmark_data_smithi047_474001_object376:head# 2024-09-18T16:01:17.195 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:17.228 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d6762583:::benchmark_data_smithi047_474001_object15:head# 2024-09-18T16:01:17.228 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:17.262 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d6d0e63b:::benchmark_data_smithi047_474001_object117:head# 2024-09-18T16:01:17.262 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:17.295 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d74454f2:::benchmark_data_smithi047_474001_object191:head# 2024-09-18T16:01:17.295 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:17.329 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d7ba857c:::benchmark_data_smithi047_474001_object160:head# 2024-09-18T16:01:17.329 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:17.362 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d7ca34da:::benchmark_data_smithi047_474001_object40:head# 2024-09-18T16:01:17.362 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:17.396 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d8141741:::benchmark_data_smithi047_474001_object287:head# 2024-09-18T16:01:17.396 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:17.429 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:da6a8dc2:::benchmark_data_smithi047_474001_object18:head# 2024-09-18T16:01:17.429 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:17.462 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:da6c2c4d:::benchmark_data_smithi047_474001_object31:head# 2024-09-18T16:01:17.463 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:17.496 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:db2f6939:::benchmark_data_smithi047_474001_object400:head# 2024-09-18T16:01:17.496 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:17.529 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:dc4893ce:::benchmark_data_smithi047_474001_object232:head# 2024-09-18T16:01:17.530 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:17.563 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:dc6daefe:::benchmark_data_smithi047_474001_object276:head# 2024-09-18T16:01:17.563 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:17.596 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:dcdad940:::benchmark_data_smithi047_474001_object182:head# 2024-09-18T16:01:17.596 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:17.630 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:dd4ec68f:::benchmark_data_smithi047_474001_object56:head# 2024-09-18T16:01:17.630 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:17.663 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ddc4bb3d:::benchmark_data_smithi047_474001_object199:head# 2024-09-18T16:01:17.663 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:17.697 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ddece86f:::benchmark_data_smithi047_474001_object43:head# 2024-09-18T16:01:17.697 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:17.730 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:df65649c:::benchmark_data_smithi047_474001_object176:head# 2024-09-18T16:01:17.730 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:17.764 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:dfa0b551:::benchmark_data_smithi047_474001_object421:head# 2024-09-18T16:01:17.764 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:17.797 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e07f15b6:::benchmark_data_smithi047_474001_object10:head# 2024-09-18T16:01:17.797 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:17.831 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e1757a69:::benchmark_data_smithi047_474001_object366:head# 2024-09-18T16:01:17.831 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:17.864 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e1859c12:::benchmark_data_smithi047_474001_object102:head# 2024-09-18T16:01:17.864 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:17.897 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e1d3a554:::benchmark_data_smithi047_474001_object369:head# 2024-09-18T16:01:17.898 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:17.931 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e437b5d2:::benchmark_data_smithi047_474001_object377:head# 2024-09-18T16:01:17.931 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:17.964 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e445afb5:::benchmark_data_smithi047_474001_object355:head# 2024-09-18T16:01:17.964 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:17.998 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e61a1975:::benchmark_data_smithi047_474001_object296:head# 2024-09-18T16:01:17.998 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:18.031 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e65db1db:::benchmark_data_smithi047_474001_object329:head# 2024-09-18T16:01:18.032 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:18.065 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e65e637a:::benchmark_data_smithi047_474001_object424:head# 2024-09-18T16:01:18.065 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:18.110 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e6b693e8:::benchmark_data_smithi047_474001_object308:head# 2024-09-18T16:01:18.110 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:18.157 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e6d85c1d:::benchmark_data_smithi047_474001_object220:head# 2024-09-18T16:01:18.157 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:18.207 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e6f3dc67:::benchmark_data_smithi047_474001_object293:head# 2024-09-18T16:01:18.207 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:18.241 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e6f460e2:::benchmark_data_smithi047_474001_object130:head# 2024-09-18T16:01:18.241 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:18.282 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e6f5814f:::benchmark_data_smithi047_474001_object64:head# 2024-09-18T16:01:18.282 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:18.315 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e7a2c4e1:::benchmark_data_smithi047_474001_object94:head# 2024-09-18T16:01:18.316 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:18.349 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e945e46a:::benchmark_data_smithi047_474001_object171:head# 2024-09-18T16:01:18.349 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:18.382 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e9b3c7f8:::benchmark_data_smithi047_474001_object181:head# 2024-09-18T16:01:18.382 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:18.416 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ea0dab4b:::benchmark_data_smithi047_474001_object406:head# 2024-09-18T16:01:18.416 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:18.449 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ea0dd8c2:::benchmark_data_smithi047_474001_object381:head# 2024-09-18T16:01:18.449 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:18.493 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ea20ab79:::benchmark_data_smithi047_474001_object178:head# 2024-09-18T16:01:18.493 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:18.541 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ea95d8c2:::benchmark_data_smithi047_474001_object238:head# 2024-09-18T16:01:18.541 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:18.600 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ecaeb389:::benchmark_data_smithi047_474001_object135:head# 2024-09-18T16:01:18.600 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:18.634 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ed907482:::benchmark_data_smithi047_474001_object34:head# 2024-09-18T16:01:18.634 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:18.667 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ee898c6a:::benchmark_data_smithi047_474001_object184:head# 2024-09-18T16:01:18.667 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:18.701 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:eeceac30:::benchmark_data_smithi047_474001_object97:head# 2024-09-18T16:01:18.701 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:18.734 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:eee17d95:::benchmark_data_smithi047_474001_object187:head# 2024-09-18T16:01:18.734 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:18.767 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:eefc7324:::benchmark_data_smithi047_474001_object91:head# 2024-09-18T16:01:18.768 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:18.801 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ef61b0b9:::benchmark_data_smithi047_474001_object428:head# 2024-09-18T16:01:18.801 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:18.834 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:efbee6f8:::benchmark_data_smithi047_474001_object211:head# 2024-09-18T16:01:18.835 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:18.868 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:eff64798:::benchmark_data_smithi047_474001_object93:head# 2024-09-18T16:01:18.868 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:18.893 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f09ab14b:::benchmark_data_smithi047_474001_object339:head# 2024-09-18T16:01:18.893 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:18.918 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f09c2c79:::benchmark_data_smithi047_474001_object258:head# 2024-09-18T16:01:18.918 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:18.943 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f0a2b2e0:::benchmark_data_smithi047_474001_object234:head# 2024-09-18T16:01:18.943 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:18.977 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f0e21778:::benchmark_data_smithi047_474001_object210:head# 2024-09-18T16:01:18.977 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:19.002 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f12591b2:::benchmark_data_smithi047_474001_object142:head# 2024-09-18T16:01:19.002 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:19.027 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f15f0e45:::benchmark_data_smithi047_474001_object353:head# 2024-09-18T16:01:19.027 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:19.052 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f17effda:::benchmark_data_smithi047_474001_object121:head# 2024-09-18T16:01:19.052 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:19.077 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f188e48b:::benchmark_data_smithi047_474001_object82:head# 2024-09-18T16:01:19.077 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:19.102 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f286bf95:::benchmark_data_smithi047_474001_object115:head# 2024-09-18T16:01:19.102 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:19.128 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f400c3e0:::benchmark_data_smithi047_474001_object305:head# 2024-09-18T16:01:19.128 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:19.153 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f40d5282:::benchmark_data_smithi047_474001_object48:head# 2024-09-18T16:01:19.153 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:19.186 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f4d9af26:::benchmark_data_smithi047_474001_object311:head# 2024-09-18T16:01:19.186 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:19.219 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f50aa0e5:::benchmark_data_smithi047_474001_object65:head# 2024-09-18T16:01:19.220 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:19.261 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f58a2ed4:::benchmark_data_smithi047_474001_object393:head# 2024-09-18T16:01:19.262 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:19.320 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f61edbb2:::benchmark_data_smithi047_474001_object140:head# 2024-09-18T16:01:19.320 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:19.353 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f6596924:::benchmark_data_smithi047_474001_object256:head# 2024-09-18T16:01:19.353 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:19.387 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f68bfbc5:::benchmark_data_smithi047_474001_object295:head# 2024-09-18T16:01:19.387 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:19.420 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f7031fa9:::benchmark_data_smithi047_474001_object318:head# 2024-09-18T16:01:19.421 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:19.454 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f8576353:::benchmark_data_smithi047_474001_object336:head# 2024-09-18T16:01:19.454 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:19.487 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f8e14a8e:::benchmark_data_smithi047_474001_object63:head# 2024-09-18T16:01:19.487 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:19.521 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f9e01ed3:::benchmark_data_smithi047_474001_object20:head# 2024-09-18T16:01:19.521 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:19.554 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fa676313:::benchmark_data_smithi047_474001_object268:head# 2024-09-18T16:01:19.554 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:19.588 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fb2704e8:::benchmark_data_smithi047_474001_object294:head# 2024-09-18T16:01:19.588 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:19.621 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fb557b5c:::benchmark_data_smithi047_474001_object25:head# 2024-09-18T16:01:19.621 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:19.654 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fba861de:::benchmark_data_smithi047_474001_object233:head# 2024-09-18T16:01:19.655 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:19.688 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fc48c5f5:::benchmark_data_smithi047_474001_object128:head# 2024-09-18T16:01:19.688 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:19.721 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fcc00bd9:::benchmark_data_smithi047_474001_object267:head# 2024-09-18T16:01:19.722 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:19.755 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fcc267e2:::benchmark_data_smithi047_474001_object420:head# 2024-09-18T16:01:19.755 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:19.788 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fcd59592:::benchmark_data_smithi047_474001_object387:head# 2024-09-18T16:01:19.788 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:19.822 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fcff1581:::benchmark_data_smithi047_474001_object229:head# 2024-09-18T16:01:19.822 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:19.855 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fd5c7c49:::benchmark_data_smithi047_474001_object375:head# 2024-09-18T16:01:19.855 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:19.889 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fdd0ff05:::benchmark_data_smithi047_474001_object67:head# 2024-09-18T16:01:19.889 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:19.922 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fdf729e9:::benchmark_data_smithi047_474001_object230:head# 2024-09-18T16:01:19.922 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:19.955 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fe6c9d0e:::benchmark_data_smithi047_474001_object52:head# 2024-09-18T16:01:19.956 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:19.989 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fe938a07:::benchmark_data_smithi047_474001_object207:head# 2024-09-18T16:01:19.989 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:20.022 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fea2446c:::benchmark_data_smithi047_474001_object105:head# 2024-09-18T16:01:20.023 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:20.056 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fee86c8a:::benchmark_data_smithi047_474001_object412:head# 2024-09-18T16:01:20.056 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:20.090 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ff03a3e6:::benchmark_data_smithi047_474001_object37:head# 2024-09-18T16:01:20.090 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:20.148 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ff8f6396:::benchmark_data_smithi047_474001_object114:head# 2024-09-18T16:01:20.148 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:20.198 INFO:tasks.workunit.client.0.smithi047.stdout:write_pg epoch 9 info 1.1( v 9'229 (0'0,9'229] local-lis/les=7/8 n=229 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-09-18T16:01:20.232 INFO:tasks.workunit.client.0.smithi047.stdout:Import successful 2024-09-18T16:01:20.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:114: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0 2024-09-18T16:01:21.945 INFO:tasks.workunit.client.0.smithi047.stdout:get_pg_num_history pg_num_history pg_num_history(e20 pg_nums {1={7=2,19=1}} deleted_pools ) 2024-09-18T16:01:21.945 INFO:tasks.workunit.client.0.smithi047.stdout:Importing pgid 1.0 2024-09-18T16:01:21.945 INFO:tasks.workunit.client.0.smithi047.stdout:pool 1 pg_num 2 -> 1 2024-09-18T16:01:21.945 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0118b086:::benchmark_data_smithi047_474001_object299:head# 2024-09-18T16:01:21.945 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:22.053 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:03a83bbd:::benchmark_data_smithi047_474001_object340:head# 2024-09-18T16:01:22.053 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:22.111 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:03ef44ea:::benchmark_data_smithi047_474001_object136:head# 2024-09-18T16:01:22.111 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:22.205 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:04ce55e6:::benchmark_data_smithi047_474001_object104:head# 2024-09-18T16:01:22.206 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:22.264 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:04e7bb72:::benchmark_data_smithi047_474001_object259:head# 2024-09-18T16:01:22.264 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:22.323 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0522b134:::benchmark_data_smithi047_474001_object390:head# 2024-09-18T16:01:22.323 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:22.381 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:065742a4:::benchmark_data_smithi047_474001_object240:head# 2024-09-18T16:01:22.381 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:22.431 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0667ed83:::benchmark_data_smithi047_474001_object29:head# 2024-09-18T16:01:22.431 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:22.532 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:067cfdc0:::benchmark_data_smithi047_474001_object373:head# 2024-09-18T16:01:22.532 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:22.624 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:06873111:::benchmark_data_smithi047_474001_object270:head# 2024-09-18T16:01:22.624 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:22.699 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:06fbd7c2:::benchmark_data_smithi047_474001_object127:head# 2024-09-18T16:01:22.699 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:22.758 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:07ded5f1:::benchmark_data_smithi047_474001_object266:head# 2024-09-18T16:01:22.758 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:22.816 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0846b0e1:::benchmark_data_smithi047_474001_object255:head# 2024-09-18T16:01:22.816 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:22.875 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0a425810:::benchmark_data_smithi047_474001_object410:head# 2024-09-18T16:01:22.875 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:22.934 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0b99baba:::benchmark_data_smithi047_474001_object215:head# 2024-09-18T16:01:22.934 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:22.992 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0cfa8c38:::benchmark_data_smithi047_474001_object163:head# 2024-09-18T16:01:22.992 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:23.052 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0e30d732:::benchmark_data_smithi047_474001_object402:head# 2024-09-18T16:01:23.052 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:23.109 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0e508c23:::benchmark_data_smithi047_474001_object196:head# 2024-09-18T16:01:23.110 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:23.143 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0e642663:::benchmark_data_smithi047_474001_object80:head# 2024-09-18T16:01:23.143 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:23.176 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0e743ace:::benchmark_data_smithi047_474001_object21:head# 2024-09-18T16:01:23.176 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:23.210 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0eaf1576:::benchmark_data_smithi047_474001_object172:head# 2024-09-18T16:01:23.210 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:23.243 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0fcd5078:::benchmark_data_smithi047_474001_object364:head# 2024-09-18T16:01:23.243 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:23.277 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0fe45a41:::benchmark_data_smithi047_474001_object333:head# 2024-09-18T16:01:23.277 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:23.310 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:104ccd46:::benchmark_data_smithi047_474001_object365:head# 2024-09-18T16:01:23.310 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:23.343 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:10c24a87:::benchmark_data_smithi047_474001_object141:head# 2024-09-18T16:01:23.343 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:23.377 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:11014d3d:::benchmark_data_smithi047_474001_object281:head# 2024-09-18T16:01:23.377 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:23.410 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1161e93e:::benchmark_data_smithi047_474001_object278:head# 2024-09-18T16:01:23.410 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:23.444 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:129bbd1b:::benchmark_data_smithi047_474001_object389:head# 2024-09-18T16:01:23.444 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:23.477 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:131e86da:::benchmark_data_smithi047_474001_object203:head# 2024-09-18T16:01:23.477 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:23.502 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1444d7df:::benchmark_data_smithi047_474001_object271:head# 2024-09-18T16:01:23.502 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:23.527 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:14ce8c64:::benchmark_data_smithi047_474001_object246:head# 2024-09-18T16:01:23.528 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:23.553 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:15241e69:::benchmark_data_smithi047_474001_object148:head# 2024-09-18T16:01:23.553 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:23.578 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1593a397:::benchmark_data_smithi047_474001_object347:head# 2024-09-18T16:01:23.578 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:23.603 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:17f3626e:::benchmark_data_smithi047_474001_object42:head# 2024-09-18T16:01:23.603 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:23.628 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:180c940a:::benchmark_data_smithi047_474001_object372:head# 2024-09-18T16:01:23.628 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:23.653 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:183bb03e:::benchmark_data_smithi047_474001_object323:head# 2024-09-18T16:01:23.653 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:23.678 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:18747883:::benchmark_data_smithi047_474001_object192:head# 2024-09-18T16:01:23.678 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:23.712 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1a19ff87:::benchmark_data_smithi047_474001_object2:head# 2024-09-18T16:01:23.712 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:23.737 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1a2c0aea:::benchmark_data_smithi047_474001_object274:head# 2024-09-18T16:01:23.737 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:23.770 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1bd588e8:::benchmark_data_smithi047_474001_object320:head# 2024-09-18T16:01:23.770 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:23.804 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1cc6e811:::benchmark_data_smithi047_474001_object149:head# 2024-09-18T16:01:23.804 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:23.837 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1ce6bbbf:::benchmark_data_smithi047_474001_object164:head# 2024-09-18T16:01:23.837 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:23.871 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1d83d665:::benchmark_data_smithi047_474001_object153:head# 2024-09-18T16:01:23.871 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:23.937 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1d849551:::benchmark_data_smithi047_474001_object395:head# 2024-09-18T16:01:23.938 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:23.962 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1df00925:::benchmark_data_smithi047_474001_object86:head# 2024-09-18T16:01:23.962 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:23.988 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1ea58aa2:::benchmark_data_smithi047_474001_object265:head# 2024-09-18T16:01:23.988 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:24.013 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:20be7217:::benchmark_data_smithi047_474001_object145:head# 2024-09-18T16:01:24.013 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:24.038 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:21c5c01f:::benchmark_data_smithi047_474001_object69:head# 2024-09-18T16:01:24.038 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:24.071 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:221f844d:::benchmark_data_smithi047_474001_object350:head# 2024-09-18T16:01:24.072 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:24.117 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:22d3be20:::benchmark_data_smithi047_474001_object315:head# 2024-09-18T16:01:24.117 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:24.164 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2334c1fe:::benchmark_data_smithi047_474001_object84:head# 2024-09-18T16:01:24.164 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:24.197 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:24393629:::benchmark_data_smithi047_474001_object239:head# 2024-09-18T16:01:24.197 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:24.230 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:25482445:::benchmark_data_smithi047_474001_object53:head# 2024-09-18T16:01:24.231 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:24.264 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2555ae90:::benchmark_data_smithi047_474001_object370:head# 2024-09-18T16:01:24.264 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:24.298 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:26a1c85e:::benchmark_data_smithi047_474001_object32:head# 2024-09-18T16:01:24.298 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:24.331 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:26b7b87f:::benchmark_data_smithi047_474001_object107:head# 2024-09-18T16:01:24.331 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:24.364 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:27190702:::benchmark_data_smithi047_474001_object218:head# 2024-09-18T16:01:24.365 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:24.411 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:27f9a01e:::benchmark_data_smithi047_474001_object349:head# 2024-09-18T16:01:24.411 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:24.440 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:284c9f3a:::benchmark_data_smithi047_474001_object254:head# 2024-09-18T16:01:24.440 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:24.484 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2856b42f:::benchmark_data_smithi047_474001_object81:head# 2024-09-18T16:01:24.484 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:24.540 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:29494f84:::benchmark_data_smithi047_474001_object248:head# 2024-09-18T16:01:24.540 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:24.599 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2a2243e8:::benchmark_data_smithi047_474001_object1:head# 2024-09-18T16:01:24.599 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:24.659 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2abea481:::benchmark_data_smithi047_474001_object23:head# 2024-09-18T16:01:24.659 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:24.716 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2ced8b80:::benchmark_data_smithi047_474001_object401:head# 2024-09-18T16:01:24.716 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:24.750 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2e2d9e2b:::benchmark_data_smithi047_474001_object367:head# 2024-09-18T16:01:24.750 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:24.816 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2e9ea224:::benchmark_data_smithi047_474001_object106:head# 2024-09-18T16:01:24.816 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:24.858 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2fb8dbb3:::benchmark_data_smithi047_474001_object170:head# 2024-09-18T16:01:24.858 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:24.917 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:300fdc07:::benchmark_data_smithi047_474001_object227:head# 2024-09-18T16:01:24.917 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:24.951 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3078741e:::benchmark_data_smithi047_474001_object139:head# 2024-09-18T16:01:24.951 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:24.984 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:31669bfe:::benchmark_data_smithi047_474001_object319:head# 2024-09-18T16:01:24.984 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:25.017 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:321ae169:::benchmark_data_smithi047_474001_object244:head# 2024-09-18T16:01:25.018 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:25.051 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:32ff3ac8:::benchmark_data_smithi047_474001_object261:head# 2024-09-18T16:01:25.051 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:25.084 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:33134995:::benchmark_data_smithi047_474001_object310:head# 2024-09-18T16:01:25.085 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:25.118 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:33b1fdf8:::benchmark_data_smithi047_474001_object161:head# 2024-09-18T16:01:25.118 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:25.151 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:33b93a31:::benchmark_data_smithi047_474001_object403:head# 2024-09-18T16:01:25.151 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:25.185 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:33fa48ba:::benchmark_data_smithi047_474001_object289:head# 2024-09-18T16:01:25.185 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:25.218 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:34348ae4:::benchmark_data_smithi047_474001_object6:head# 2024-09-18T16:01:25.218 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:25.252 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:34e0f433:::benchmark_data_smithi047_474001_object342:head# 2024-09-18T16:01:25.252 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:25.285 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:355eddbc:::benchmark_data_smithi047_474001_object198:head# 2024-09-18T16:01:25.285 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:25.318 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:35841d04:::benchmark_data_smithi047_474001_object186:head# 2024-09-18T16:01:25.319 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:25.352 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:35c32413:::benchmark_data_smithi047_474001_object413:head# 2024-09-18T16:01:25.352 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:25.385 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:35e67788:::benchmark_data_smithi047_474001_object388:head# 2024-09-18T16:01:25.385 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:25.419 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3613c729:::benchmark_data_smithi047_474001_object380:head# 2024-09-18T16:01:25.419 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:25.452 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:37ccfdd9:::benchmark_data_smithi047_474001_object219:head# 2024-09-18T16:01:25.453 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:25.486 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:380af478:::benchmark_data_smithi047_474001_object359:head# 2024-09-18T16:01:25.486 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:25.519 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3812ffca:::benchmark_data_smithi047_474001_object286:head# 2024-09-18T16:01:25.519 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:25.553 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:381ae33c:::benchmark_data_smithi047_474001_object152:head# 2024-09-18T16:01:25.553 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:25.586 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3994ed2b:::benchmark_data_smithi047_474001_object162:head# 2024-09-18T16:01:25.586 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:25.619 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3ad1e94f:::benchmark_data_smithi047_474001_object26:head# 2024-09-18T16:01:25.620 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:25.653 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3b2460a7:::benchmark_data_smithi047_474001_object418:head# 2024-09-18T16:01:25.653 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:25.686 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3b70d910:::benchmark_data_smithi047_474001_object157:head# 2024-09-18T16:01:25.687 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:25.737 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3b976a95:::benchmark_data_smithi047_474001_object392:head# 2024-09-18T16:01:25.737 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:25.762 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3c01f944:::benchmark_data_smithi047_474001_object85:head# 2024-09-18T16:01:25.762 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:25.787 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3c08c862:::benchmark_data_smithi047_474001_object209:head# 2024-09-18T16:01:25.787 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:25.812 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3c3b4221:::benchmark_data_smithi047_474001_object302:head# 2024-09-18T16:01:25.812 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:25.837 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3e2bc2d1:::benchmark_data_smithi047_474001_object290:head# 2024-09-18T16:01:25.837 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:25.862 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3f0792f2:::benchmark_data_smithi047_474001_object205:head# 2024-09-18T16:01:25.862 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:25.887 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3f0bbc7e:::benchmark_data_smithi047_474001_object214:head# 2024-09-18T16:01:25.888 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:25.912 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:40e84920:::benchmark_data_smithi047_474001_object201:head# 2024-09-18T16:01:25.913 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:25.938 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:410710e5:::benchmark_data_smithi047_474001_object360:head# 2024-09-18T16:01:25.938 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:25.971 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4286a502:::benchmark_data_smithi047_474001_object280:head# 2024-09-18T16:01:25.971 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:26.005 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:429b03e9:::benchmark_data_smithi047_474001_object249:head# 2024-09-18T16:01:26.005 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:26.030 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:43c8723a:::benchmark_data_smithi047_474001_object4:head# 2024-09-18T16:01:26.030 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:26.055 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:43c8e265:::benchmark_data_smithi047_474001_object208:head# 2024-09-18T16:01:26.055 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:26.089 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:43d378b4:::benchmark_data_smithi047_474001_object425:head# 2024-09-18T16:01:26.089 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:26.147 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:43fb4408:::benchmark_data_smithi047_474001_object399:head# 2024-09-18T16:01:26.147 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:26.189 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:44ab53e5:::benchmark_data_smithi047_474001_object27:head# 2024-09-18T16:01:26.189 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:26.222 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4515f62e:::benchmark_data_smithi047_474001_object143:head# 2024-09-18T16:01:26.223 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:26.256 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:457b93bc:::benchmark_data_smithi047_474001_object300:head# 2024-09-18T16:01:26.256 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:26.289 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:457e7435:::benchmark_data_smithi047_474001_object231:head# 2024-09-18T16:01:26.290 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:26.323 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:468b56fb:::benchmark_data_smithi047_474001_object242:head# 2024-09-18T16:01:26.323 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:26.356 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:47454808:::benchmark_data_smithi047_474001_object394:head# 2024-09-18T16:01:26.357 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:26.390 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4757d830:::benchmark_data_smithi047_474001_object147:head# 2024-09-18T16:01:26.390 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:26.423 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4790fa42:::benchmark_data_smithi047_474001_object16:head# 2024-09-18T16:01:26.423 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:26.457 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:481edef6:::benchmark_data_smithi047_474001_object129:head# 2024-09-18T16:01:26.457 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:26.515 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:498127df:::benchmark_data_smithi047_474001_object409:head# 2024-09-18T16:01:26.516 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:26.570 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4a94b2bf:::benchmark_data_smithi047_474001_object331:head# 2024-09-18T16:01:26.570 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:26.633 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4b0f17cd:::benchmark_data_smithi047_474001_object194:head# 2024-09-18T16:01:26.633 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:26.666 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4ccceada:::benchmark_data_smithi047_474001_object213:head# 2024-09-18T16:01:26.666 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:26.700 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4d145d82:::benchmark_data_smithi047_474001_object206:head# 2024-09-18T16:01:26.700 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:26.733 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4e67dcba:::benchmark_data_smithi047_474001_object144:head# 2024-09-18T16:01:26.733 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:26.766 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4e6bc7a4:::benchmark_data_smithi047_474001_object165:head# 2024-09-18T16:01:26.767 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:26.800 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4e7fe7b2:::benchmark_data_smithi047_474001_object431:head# 2024-09-18T16:01:26.800 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:26.833 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4f5e76df:::benchmark_data_smithi047_474001_object415:head# 2024-09-18T16:01:26.833 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:26.867 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5030ebeb:::benchmark_data_smithi047_474001_object335:head# 2024-09-18T16:01:26.867 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:26.900 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:505987dd:::benchmark_data_smithi047_474001_object313:head# 2024-09-18T16:01:26.900 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:26.934 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:50a1999f:::benchmark_data_smithi047_474001_object190:head# 2024-09-18T16:01:26.934 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:26.967 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:51905d78:::benchmark_data_smithi047_474001_object371:head# 2024-09-18T16:01:26.967 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:27.000 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:53516fa7:::benchmark_data_smithi047_474001_object47:head# 2024-09-18T16:01:27.001 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:27.059 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:53a75224:::benchmark_data_smithi047_474001_object277:head# 2024-09-18T16:01:27.059 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:27.093 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:53bc5374:::benchmark_data_smithi047_474001_object304:head# 2024-09-18T16:01:27.093 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:27.143 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:54037397:::benchmark_data_smithi047_474001_object168:head# 2024-09-18T16:01:27.143 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:27.176 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:54e93f24:::benchmark_data_smithi047_474001_object351:head# 2024-09-18T16:01:27.176 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:27.210 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5549c508:::benchmark_data_smithi047_474001_object154:head# 2024-09-18T16:01:27.210 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:27.243 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:55a8f195:::benchmark_data_smithi047_474001_object183:head# 2024-09-18T16:01:27.243 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:27.277 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:58bb18eb:::benchmark_data_smithi047_474001_object92:head# 2024-09-18T16:01:27.277 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:27.310 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:58e3824a:::benchmark_data_smithi047_474001_object22:head# 2024-09-18T16:01:27.310 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:27.344 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:58eacbbf:::benchmark_data_smithi047_474001_object100:head# 2024-09-18T16:01:27.344 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:27.377 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:590d7cd0:::benchmark_data_smithi047_474001_object328:head# 2024-09-18T16:01:27.377 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:27.410 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:59701efb:::benchmark_data_smithi047_474001_object243:head# 2024-09-18T16:01:27.410 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:27.444 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:59852438:::benchmark_data_smithi047_474001_object83:head# 2024-09-18T16:01:27.444 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:27.477 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5a123740:::benchmark_data_smithi047_474001_object36:head# 2024-09-18T16:01:27.478 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:27.511 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5b73d6c3:::benchmark_data_smithi047_474001_object284:head# 2024-09-18T16:01:27.511 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:27.544 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5c8e3731:::benchmark_data_smithi047_474001_object132:head# 2024-09-18T16:01:27.544 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:27.578 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5caa7c9f:::benchmark_data_smithi047_474001_object416:head# 2024-09-18T16:01:27.578 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:27.611 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5cf5ea91:::benchmark_data_smithi047_474001_object397:head# 2024-09-18T16:01:27.611 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:27.644 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5d1b58ad:::benchmark_data_smithi047_474001_object269:head# 2024-09-18T16:01:27.645 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:27.678 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5e669bb0:::benchmark_data_smithi047_474001_object405:head# 2024-09-18T16:01:27.678 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:27.712 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5eac6d5a:::benchmark_data_smithi047_474001_object411:head# 2024-09-18T16:01:27.712 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:27.745 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5ef5c1d9:::benchmark_data_smithi047_474001_object385:head# 2024-09-18T16:01:27.745 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:27.778 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5f7398fd:::benchmark_data_smithi047_474001_object68:head# 2024-09-18T16:01:27.779 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:27.812 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5f8ba50b:::benchmark_data_smithi047_474001_object174:head# 2024-09-18T16:01:27.812 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:27.845 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5fb38f21:::benchmark_data_smithi047_474001_object75:head# 2024-09-18T16:01:27.845 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:27.879 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:60a5ff1a:::benchmark_data_smithi047_474001_object298:head# 2024-09-18T16:01:27.879 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:27.912 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6159ad07:::benchmark_data_smithi047_474001_object221:head# 2024-09-18T16:01:27.912 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:27.946 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:62273e63:::benchmark_data_smithi047_474001_object62:head# 2024-09-18T16:01:27.946 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:27.979 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:63cf18a2:::benchmark_data_smithi047_474001_object272:head# 2024-09-18T16:01:27.979 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:28.013 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:63f415e6:::benchmark_data_smithi047_474001_object58:head# 2024-09-18T16:01:28.013 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:28.046 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:64ff073d:::benchmark_data_smithi047_474001_object362:head# 2024-09-18T16:01:28.046 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:28.080 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:657bccc4:::benchmark_data_smithi047_474001_object169:head# 2024-09-18T16:01:28.080 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:28.113 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:660836d4:::benchmark_data_smithi047_474001_object39:head# 2024-09-18T16:01:28.113 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:28.163 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:66682ebd:::benchmark_data_smithi047_474001_object101:head# 2024-09-18T16:01:28.163 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:28.222 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:66e730e2:::benchmark_data_smithi047_474001_object61:head# 2024-09-18T16:01:28.222 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:28.247 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:66fbc3ec:::benchmark_data_smithi047_474001_object71:head# 2024-09-18T16:01:28.247 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:28.272 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6717adea:::benchmark_data_smithi047_474001_object317:head# 2024-09-18T16:01:28.272 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:28.297 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:68107ec1:::benchmark_data_smithi047_474001_object195:head# 2024-09-18T16:01:28.298 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:28.322 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6812f8cb:::benchmark_data_smithi047_474001_object264:head# 2024-09-18T16:01:28.323 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:28.347 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:68f519f4:::benchmark_data_smithi047_474001_object343:head# 2024-09-18T16:01:28.348 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:28.373 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6a39a8d4:::benchmark_data_smithi047_474001_object66:head# 2024-09-18T16:01:28.373 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:28.398 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6ab23d26:::benchmark_data_smithi047_474001_object9:head# 2024-09-18T16:01:28.398 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:28.423 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6b25d55a:::benchmark_data_smithi047_474001_object72:head# 2024-09-18T16:01:28.423 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:28.448 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6b5fc221:::benchmark_data_smithi047_474001_object334:head# 2024-09-18T16:01:28.448 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:28.490 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6c2d825c:::benchmark_data_smithi047_474001_object5:head# 2024-09-18T16:01:28.490 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:28.544 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6caa7eaf:::benchmark_data_smithi047_474001_object263:head# 2024-09-18T16:01:28.544 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:28.599 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6dfd6f4e:::benchmark_data_smithi047_474001_object251:head# 2024-09-18T16:01:28.599 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:28.632 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6e9248a9:::benchmark_data_smithi047_474001_object245:head# 2024-09-18T16:01:28.633 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:28.666 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6f58cdd1:::benchmark_data_smithi047_474001_object430:head# 2024-09-18T16:01:28.666 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:28.699 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:71084a82:::benchmark_data_smithi047_474001_object398:head# 2024-09-18T16:01:28.699 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:28.733 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:722633c0:::benchmark_data_smithi047_474001_object7:head# 2024-09-18T16:01:28.733 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:28.766 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:73078ec4:::benchmark_data_smithi047_474001_object200:head# 2024-09-18T16:01:28.766 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:28.799 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7319939a:::benchmark_data_smithi047_474001_object378:head# 2024-09-18T16:01:28.800 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:28.833 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:732bbbc2:::benchmark_data_smithi047_474001_object113:head# 2024-09-18T16:01:28.833 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:28.866 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:73694cfd:::benchmark_data_smithi047_474001_object257:head# 2024-09-18T16:01:28.867 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:28.900 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:739a0e00:::benchmark_data_smithi047_474001_object11:head# 2024-09-18T16:01:28.900 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:28.933 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:748a50de:::benchmark_data_smithi047_474001_object223:head# 2024-09-18T16:01:28.933 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:28.967 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:74cbfe52:::benchmark_data_smithi047_474001_object309:head# 2024-09-18T16:01:28.967 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:29.033 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:751b68bd:::benchmark_data_smithi047_474001_object45:head# 2024-09-18T16:01:29.033 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:29.067 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:76583154:::benchmark_data_smithi047_474001_object13:head# 2024-09-18T16:01:29.067 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:29.101 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7669c90e:::benchmark_data_smithi047_474001_object216:head# 2024-09-18T16:01:29.101 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:29.134 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:76dc48ec:::benchmark_data_smithi047_474001_object79:head# 2024-09-18T16:01:29.134 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:29.168 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:77226950:::benchmark_data_smithi047_474001_object386:head# 2024-09-18T16:01:29.168 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:29.201 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:77a33037:::benchmark_data_smithi047_474001_object197:head# 2024-09-18T16:01:29.201 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:29.234 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:77f7974a:::benchmark_data_smithi047_474001_object123:head# 2024-09-18T16:01:29.234 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:29.293 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:78d90d4e:::benchmark_data_smithi047_474001_object301:head# 2024-09-18T16:01:29.293 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:29.326 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7b36e97b:::benchmark_data_smithi047_474001_object292:head# 2024-09-18T16:01:29.326 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:29.360 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7c8233bd:::benchmark_data_smithi047_474001_object76:head# 2024-09-18T16:01:29.360 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:29.393 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7cce9f64:::benchmark_data_smithi047_474001_object0:head# 2024-09-18T16:01:29.393 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:29.427 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7d8e31eb:::benchmark_data_smithi047_474001_object193:head# 2024-09-18T16:01:29.427 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:29.460 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7dc806f2:::benchmark_data_smithi047_474001_object126:head# 2024-09-18T16:01:29.460 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:29.494 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7dfaf25d:::benchmark_data_smithi047_474001_object260:head# 2024-09-18T16:01:29.494 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:29.538 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7e00f5f7:::benchmark_data_smithi047_474001_object98:head# 2024-09-18T16:01:29.538 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:29.569 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7f679bd9:::benchmark_data_smithi047_474001_object379:head# 2024-09-18T16:01:29.569 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:29.602 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7fa61220:::benchmark_data_smithi047_474001_object354:head# 2024-09-18T16:01:29.602 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:29.636 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7fb32424:::benchmark_data_smithi047_474001_object241:head# 2024-09-18T16:01:29.636 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:01:29.670 INFO:tasks.workunit.client.0.smithi047.stdout:write_pg epoch 9 info 1.0( v 9'204 (0'0,9'204] local-lis/les=7/8 n=204 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-09-18T16:01:29.703 INFO:tasks.workunit.client.0.smithi047.stdout:Import successful 2024-09-18T16:01:30.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:115: TEST_import_after_merge_and_gap: activate_osd td/pg-split-merge 0 2024-09-18T16:01:30.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-09-18T16:01:30.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T16:01:30.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T16:01:30.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T16:01:30.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/0 2024-09-18T16:01:30.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1a671e43-8866-44a5-abe3-c1e1f4ff765b --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-18T16:01:30.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:01:30.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:01:30.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:01:30.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-09-18T16:01:30.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-09-18T16:01:30.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T16:01:30.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T16:01:30.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-09-18T16:01:30.226 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T16:01:30.226 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:01:30.226 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:01:30.226 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:01:30.226 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:01:30.226 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.463129 2024-09-18T16:01:30.226 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.463129/$cluster-$name.asok' 2024-09-18T16:01:30.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.463129/$cluster-$name.asok' 2024-09-18T16:01:30.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:01:30.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-09-18T16:01:30.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-09-18T16:01:30.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:01:30.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:01:30.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:01:30.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:01:30.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T16:01:30.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T16:01:30.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/0 2024-09-18T16:01:30.228 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T16:01:30.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T16:01:30.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1a671e43-8866-44a5-abe3-c1e1f4ff765b --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.463129/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:01:30.229 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/0/whoami 2024-09-18T16:01:30.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T16:01:30.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T16:01:30.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T16:01:30.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T16:01:30.253 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:01:30.252+0000 7f4953010440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:01:30.255 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:01:30.255+0000 7f4953010440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:01:30.257 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:01:30.256+0000 7f4953010440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:01:30.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T16:01:30.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:01:30.602 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T16:01:30.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:01:30.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:01:30.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:01:30.603 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:01:30.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:01:30.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:01:30.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:01:30.774 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:01:30.773+0000 7f4953010440 -1 Falling back to public interface 2024-09-18T16:01:31.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:01:32.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:01:32.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:01:32.053 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T16:01:32.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:01:32.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:01:32.054 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:01:32.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:01:33.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:01:33.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:01:33.441 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T16:01:33.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:01:33.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:01:33.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:01:33.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:01:34.082 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:01:34.081+0000 7f4953010440 -1 osd.0 20 log_to_monitors true 2024-09-18T16:01:34.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:01:34.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:01:34.812 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:01:34.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:01:34.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:01:34.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:01:35.221 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 23 up_thru 23 down_at 21 last_clean_interval [12,20) [v2:127.0.0.1:6802/476800,v1:127.0.0.1:6803/476800] [v2:127.0.0.1:6804/476800,v1:127.0.0.1:6805/476800] exists,up 1b6f27f6-2f00-44c7-8d93-1f3fa5ab4594 2024-09-18T16:01:35.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:01:35.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:01:35.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:01:35.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:117: TEST_import_after_merge_and_gap: wait_for_clean 2024-09-18T16:01:35.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:01:35.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:01:35.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:01:35.223 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:01:35.223 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:01:35.223 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:01:35.223 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:01:35.224 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:01:35.224 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:01:35.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T16:01:35.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:01:35.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:01:35.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:01:35.312 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:01:35.312 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:01:35.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T16:01:35.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:01:35.679 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:01:35.679 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:01:35.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=98784247811 2024-09-18T16:01:35.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 98784247811 2024-09-18T16:01:35.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-98784247811' 2024-09-18T16:01:35.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:01:35.789 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-98784247811 2024-09-18T16:01:35.789 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:01:35.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:01:35.790 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-98784247811 2024-09-18T16:01:35.790 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:01:35.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=98784247811 2024-09-18T16:01:35.791 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 98784247811 2024-09-18T16:01:35.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 98784247811' 2024-09-18T16:01:35.792 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:01:36.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 98784247811 -lt 98784247811 2024-09-18T16:01:36.191 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:01:36.192 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:01:36.192 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:01:36.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:01:36.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:01:36.721 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:01:36.721 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:01:36.721 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:01:36.722 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:01:36.722 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:01:36.722 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:01:37.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:01:37.101 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:01:37.101 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:01:37.101 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:01:37.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:01:37.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:01:37.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:01:37.604 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: seq 1 50 2024-09-18T16:01:37.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:01:37.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:01:38.279 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:01:38.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:01:39.048 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:01:39.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:01:39.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:01:39.877 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:01:39.888 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:01:40.688 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:01:40.699 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:01:40.699 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:01:41.425 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:01:41.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:01:42.271 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:01:42.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:01:42.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:01:43.024 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:01:43.035 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:01:43.752 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:01:43.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:01:43.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:01:44.505 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:01:44.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:01:45.435 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:01:45.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:01:45.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:01:46.264 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:01:46.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:01:47.017 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:01:47.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:01:47.028 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:01:47.762 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:01:47.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:01:48.524 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:01:48.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:01:48.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:01:49.285 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:01:49.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:01:50.014 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:01:50.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:01:50.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:01:50.733 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:01:50.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:01:51.503 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:01:51.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:01:51.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:01:52.282 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:01:52.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:01:53.044 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:01:53.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:01:53.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:01:53.764 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:01:53.775 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:01:54.618 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:01:54.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:01:54.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:01:55.320 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:01:55.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:01:56.099 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:01:56.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:01:56.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:01:56.793 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:01:56.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:01:57.555 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:01:57.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:01:57.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:01:58.392 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:01:58.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:01:59.086 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:01:59.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:01:59.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:01:59.697 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:01:59.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:00.450 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:00.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:02:00.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:02:01.204 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:02:01.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:01.941 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:01.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:02:01.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:02:02.752 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:02:02.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:03.464 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:03.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:02:03.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:02:04.318 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:02:04.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:05.164 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:05.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:02:05.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:02:05.892 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:02:05.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:06.620 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:06.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:02:06.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:02:07.373 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:02:07.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:08.077 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:08.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:02:08.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:02:08.771 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:02:08.782 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:09.474 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:09.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:02:09.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:02:10.252 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:02:10.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:11.038 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:11.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:02:11.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:02:11.800 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:02:11.811 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:12.570 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:12.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:02:12.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:02:13.398 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:02:13.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:14.152 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:14.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:02:14.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:02:14.880 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:02:14.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:15.749 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:15.760 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:02:15.761 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:02:16.545 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:02:16.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:17.340 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:17.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:02:17.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:02:17.975 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:02:17.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:18.745 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:18.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:02:18.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:02:19.497 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:02:19.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:20.266 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:20.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:02:20.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:02:20.928 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:02:20.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:21.681 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:21.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:02:21.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:02:22.401 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:02:22.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:23.078 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:23.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:02:23.090 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:02:23.809 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:02:23.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:24.529 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:24.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:02:24.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:02:25.417 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:02:25.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:26.246 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:26.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:02:26.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:02:26.991 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:02:27.001 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:27.694 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:27.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:02:27.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:02:28.456 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:02:28.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:29.210 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:29.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:02:29.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:02:29.938 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:02:29.949 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:30.783 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:30.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:02:30.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:02:31.562 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:02:31.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:32.323 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:32.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:02:32.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:02:33.035 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:02:33.046 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:33.789 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:33.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:02:33.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:02:34.534 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:02:34.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:35.287 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:35.298 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:02:35.299 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:02:36.032 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:02:36.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:36.752 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:36.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:02:36.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:02:37.505 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:02:37.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:38.243 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:38.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:02:38.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:02:38.979 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:02:38.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:39.733 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:39.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:02:39.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:02:40.495 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:02:40.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:41.290 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:41.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:02:41.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:02:42.052 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:02:42.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:42.755 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:42.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:02:42.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:02:43.516 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:02:43.527 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:44.355 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:44.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:02:44.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:02:45.158 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:02:45.169 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:45.928 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:45.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:02:45.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:02:46.849 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:02:46.860 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:47.536 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:47.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:02:47.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:02:48.356 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:02:48.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:49.034 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:49.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:02:49.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:02:49.796 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:02:49.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:50.516 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:50.527 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:02:50.527 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:02:51.278 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:02:51.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:52.082 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:52.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-18T16:02:52.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:02:52.759 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:02:52.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:02:53.513 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:02:53.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:127: TEST_import_after_merge_and_gap: rados -p foo bench 1 write -b 1024 --no-cleanup 2024-09-18T16:02:53.552 INFO:tasks.workunit.client.0.smithi047.stdout:hints = 1 2024-09-18T16:02:53.552 INFO:tasks.workunit.client.0.smithi047.stdout:Maintaining 16 concurrent writes of 1024 bytes to objects of size 1024 for up to 1 seconds or 0 objects 2024-09-18T16:02:53.552 INFO:tasks.workunit.client.0.smithi047.stdout:Object prefix: benchmark_data_smithi047_479866 2024-09-18T16:02:53.552 INFO:tasks.workunit.client.0.smithi047.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T16:02:53.552 INFO:tasks.workunit.client.0.smithi047.stdout: 0 0 0 0 0 0 - 0 2024-09-18T16:02:54.552 INFO:tasks.workunit.client.0.smithi047.stdout: 1 16 177 161 0.157206 0.157227 0.0752401 0.0895475 2024-09-18T16:02:55.552 INFO:tasks.workunit.client.0.smithi047.stdout:Total time run: 1.08222 2024-09-18T16:02:55.552 INFO:tasks.workunit.client.0.smithi047.stdout:Total writes made: 177 2024-09-18T16:02:55.552 INFO:tasks.workunit.client.0.smithi047.stdout:Write size: 1024 2024-09-18T16:02:55.552 INFO:tasks.workunit.client.0.smithi047.stdout:Object size: 1024 2024-09-18T16:02:55.553 INFO:tasks.workunit.client.0.smithi047.stdout:Bandwidth (MB/sec): 0.159719 2024-09-18T16:02:55.553 INFO:tasks.workunit.client.0.smithi047.stdout:Stddev Bandwidth: 0 2024-09-18T16:02:55.553 INFO:tasks.workunit.client.0.smithi047.stdout:Max bandwidth (MB/sec): 0.157227 2024-09-18T16:02:55.553 INFO:tasks.workunit.client.0.smithi047.stdout:Min bandwidth (MB/sec): 0.157227 2024-09-18T16:02:55.553 INFO:tasks.workunit.client.0.smithi047.stdout:Average IOPS: 163 2024-09-18T16:02:55.553 INFO:tasks.workunit.client.0.smithi047.stdout:Stddev IOPS: 0 2024-09-18T16:02:55.553 INFO:tasks.workunit.client.0.smithi047.stdout:Max IOPS: 161 2024-09-18T16:02:55.553 INFO:tasks.workunit.client.0.smithi047.stdout:Min IOPS: 161 2024-09-18T16:02:55.553 INFO:tasks.workunit.client.0.smithi047.stdout:Average Latency(s): 0.0928763 2024-09-18T16:02:55.553 INFO:tasks.workunit.client.0.smithi047.stdout:Stddev Latency(s): 0.0271667 2024-09-18T16:02:55.554 INFO:tasks.workunit.client.0.smithi047.stdout:Max latency(s): 0.144906 2024-09-18T16:02:55.554 INFO:tasks.workunit.client.0.smithi047.stdout:Min latency(s): 0.0665435 2024-09-18T16:02:55.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:128: TEST_import_after_merge_and_gap: wait_for_clean 2024-09-18T16:02:55.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:02:55.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:02:55.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:02:55.589 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:02:55.590 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:02:55.590 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:02:55.590 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:02:55.590 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:02:55.590 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:02:55.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T16:02:55.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:02:55.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:02:55.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:02:55.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:02:55.690 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:02:56.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T16:02:56.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:02:56.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:02:56.061 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:02:56.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=98784247829 2024-09-18T16:02:56.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 98784247829 2024-09-18T16:02:56.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-98784247829' 2024-09-18T16:02:56.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:02:56.171 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-98784247829 2024-09-18T16:02:56.172 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:02:56.172 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:02:56.173 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-98784247829 2024-09-18T16:02:56.173 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:02:56.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=98784247829 2024-09-18T16:02:56.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 98784247829' 2024-09-18T16:02:56.174 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 98784247829 2024-09-18T16:02:56.174 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:02:56.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 98784247827 -lt 98784247829 2024-09-18T16:02:56.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:02:57.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:02:57.548 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:02:57.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 98784247827 -lt 98784247829 2024-09-18T16:02:57.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:02:58.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-18T16:02:58.969 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:02:59.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 98784247829 -lt 98784247829 2024-09-18T16:02:59.387 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:02:59.387 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:02:59.388 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:02:59.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:02:59.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:02:59.896 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:02:59.896 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:02:59.896 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:02:59.896 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:02:59.896 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:02:59.896 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:03:00.261 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:03:00.261 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:03:00.261 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:03:00.261 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:03:00.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:03:00.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:03:00.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:03:00.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:129: TEST_import_after_merge_and_gap: ceph tell osd.0 send_beacon 2024-09-18T16:03:00.876 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:130: TEST_import_after_merge_and_gap: sleep 5 2024-09-18T16:03:05.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:131: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-18T16:03:06.510 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is set 2024-09-18T16:03:06.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:132: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-18T16:03:07.289 INFO:tasks.workunit.client.0.smithi047.stderr:nodown is unset 2024-09-18T16:03:07.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:133: TEST_import_after_merge_and_gap: sleep 5 2024-09-18T16:03:12.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:135: TEST_import_after_merge_and_gap: kill_daemons td/pg-split-merge TERM osd.0 2024-09-18T16:03:12.303 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:03:12.303 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:03:12.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:03:12.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:03:12.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:03:12.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:03:12.610 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:138: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.1 --file td/pg-split-merge/1.1 2024-09-18T16:03:14.046 INFO:tasks.workunit.client.0.smithi047.stdout:get_pg_num_history pg_num_history pg_num_history(e228 pg_nums {1={19=1}} deleted_pools ) 2024-09-18T16:03:14.046 INFO:tasks.workunit.client.0.smithi047.stdout:Importing pgid 1.1 2024-09-18T16:03:14.046 INFO:tasks.workunit.client.0.smithi047.stderr:PG export's map 9 is older than OSD's oldest_map 121 2024-09-18T16:03:14.046 INFO:tasks.workunit.client.0.smithi047.stderr: pass --force to proceed anyway (with incomplete PastIntervals) 2024-09-18T16:03:14.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:140: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op export-remove --pgid 1.0 --force --file td/pg-split-merge/1.0.later 2024-09-18T16:03:15.858 INFO:tasks.workunit.client.0.smithi047.stderr:Exporting 1.0 info 1.0( v 224'407 (9'229,224'407] local-lis/les=23/24 n=610 ec=7/7 lis/c=23/23 les/c/f=24/24/0 sis=23) 2024-09-18T16:03:15.871 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:008f3a82:::benchmark_data_smithi047_479866_object170:head# 2024-09-18T16:03:15.893 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0118b086:::benchmark_data_smithi047_474001_object299:head# 2024-09-18T16:03:15.900 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:02a07fce:::benchmark_data_smithi047_479866_object166:head# 2024-09-18T16:03:15.908 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:03a83bbd:::benchmark_data_smithi047_474001_object340:head# 2024-09-18T16:03:15.908 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:03ef44ea:::benchmark_data_smithi047_474001_object136:head# 2024-09-18T16:03:15.921 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:04ce55e6:::benchmark_data_smithi047_474001_object104:head# 2024-09-18T16:03:15.933 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:04e7bb72:::benchmark_data_smithi047_474001_object259:head# 2024-09-18T16:03:15.933 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0522b134:::benchmark_data_smithi047_474001_object390:head# 2024-09-18T16:03:15.933 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0538e625:::benchmark_data_smithi047_479866_object35:head# 2024-09-18T16:03:15.937 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0621c7be:::benchmark_data_smithi047_479866_object70:head# 2024-09-18T16:03:15.940 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:065742a4:::benchmark_data_smithi047_474001_object240:head# 2024-09-18T16:03:15.941 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0667ed83:::benchmark_data_smithi047_474001_object29:head# 2024-09-18T16:03:15.941 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:067cfdc0:::benchmark_data_smithi047_474001_object373:head# 2024-09-18T16:03:15.941 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:06873111:::benchmark_data_smithi047_474001_object270:head# 2024-09-18T16:03:15.942 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:06fbd7c2:::benchmark_data_smithi047_474001_object127:head# 2024-09-18T16:03:15.942 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:07ded5f1:::benchmark_data_smithi047_474001_object266:head# 2024-09-18T16:03:15.942 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0846b0e1:::benchmark_data_smithi047_474001_object255:head# 2024-09-18T16:03:15.943 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:084dae8d:::benchmark_data_smithi047_479866_object46:head# 2024-09-18T16:03:15.943 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:088043d9:::benchmark_data_smithi047_479866_object133:head# 2024-09-18T16:03:15.944 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0a425810:::benchmark_data_smithi047_474001_object410:head# 2024-09-18T16:03:15.944 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0b99baba:::benchmark_data_smithi047_474001_object215:head# 2024-09-18T16:03:15.944 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0cbc1619:::benchmark_data_smithi047_479866_object92:head# 2024-09-18T16:03:15.945 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0cfa8c38:::benchmark_data_smithi047_474001_object163:head# 2024-09-18T16:03:15.945 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0cffcd84:::benchmark_data_smithi047_479866_object21:head# 2024-09-18T16:03:15.954 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0e30d732:::benchmark_data_smithi047_474001_object402:head# 2024-09-18T16:03:15.965 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0e508c23:::benchmark_data_smithi047_474001_object196:head# 2024-09-18T16:03:15.966 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0e642663:::benchmark_data_smithi047_474001_object80:head# 2024-09-18T16:03:15.966 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0e743ace:::benchmark_data_smithi047_474001_object21:head# 2024-09-18T16:03:15.966 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0eaf1576:::benchmark_data_smithi047_474001_object172:head# 2024-09-18T16:03:15.967 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0f5d34c4:::benchmark_data_smithi047_479866_object5:head# 2024-09-18T16:03:15.968 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0fcd5078:::benchmark_data_smithi047_474001_object364:head# 2024-09-18T16:03:15.968 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0fe45a41:::benchmark_data_smithi047_474001_object333:head# 2024-09-18T16:03:15.969 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:104ccd46:::benchmark_data_smithi047_474001_object365:head# 2024-09-18T16:03:15.969 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:10c24a87:::benchmark_data_smithi047_474001_object141:head# 2024-09-18T16:03:15.970 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:11014d3d:::benchmark_data_smithi047_474001_object281:head# 2024-09-18T16:03:15.970 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1161e93e:::benchmark_data_smithi047_474001_object278:head# 2024-09-18T16:03:15.970 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:129bbd1b:::benchmark_data_smithi047_474001_object389:head# 2024-09-18T16:03:15.971 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:12b8132b:::benchmark_data_smithi047_479866_object40:head# 2024-09-18T16:03:15.971 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:131e86da:::benchmark_data_smithi047_474001_object203:head# 2024-09-18T16:03:15.972 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1444d7df:::benchmark_data_smithi047_474001_object271:head# 2024-09-18T16:03:15.972 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:14ce8c64:::benchmark_data_smithi047_474001_object246:head# 2024-09-18T16:03:15.973 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:15241e69:::benchmark_data_smithi047_474001_object148:head# 2024-09-18T16:03:15.973 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:153afee9:::benchmark_data_smithi047_479866_object104:head# 2024-09-18T16:03:15.974 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:158af36b:::benchmark_data_smithi047_479866_object30:head# 2024-09-18T16:03:15.974 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1593a397:::benchmark_data_smithi047_474001_object347:head# 2024-09-18T16:03:15.975 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:173ef99d:::benchmark_data_smithi047_479866_object137:head# 2024-09-18T16:03:15.975 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:175d5f0f:::benchmark_data_smithi047_479866_object48:head# 2024-09-18T16:03:15.976 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:17f3626e:::benchmark_data_smithi047_474001_object42:head# 2024-09-18T16:03:15.977 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:180c940a:::benchmark_data_smithi047_474001_object372:head# 2024-09-18T16:03:15.977 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:183bb03e:::benchmark_data_smithi047_474001_object323:head# 2024-09-18T16:03:15.978 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:18417460:::benchmark_data_smithi047_479866_object60:head# 2024-09-18T16:03:15.978 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:18747883:::benchmark_data_smithi047_474001_object192:head# 2024-09-18T16:03:15.978 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:189ef644:::benchmark_data_smithi047_479866_object39:head# 2024-09-18T16:03:15.979 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1a19ff87:::benchmark_data_smithi047_474001_object2:head# 2024-09-18T16:03:15.979 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1a2c0aea:::benchmark_data_smithi047_474001_object274:head# 2024-09-18T16:03:15.979 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1b28af72:::benchmark_data_smithi047_479866_object139:head# 2024-09-18T16:03:15.980 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1bd588e8:::benchmark_data_smithi047_474001_object320:head# 2024-09-18T16:03:15.980 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1c1035dd:::benchmark_data_smithi047_479866_object161:head# 2024-09-18T16:03:15.980 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1c648bcf:::benchmark_data_smithi047_479866_object138:head# 2024-09-18T16:03:15.981 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1cc6e811:::benchmark_data_smithi047_474001_object149:head# 2024-09-18T16:03:15.981 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1ce6bbbf:::benchmark_data_smithi047_474001_object164:head# 2024-09-18T16:03:15.981 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1d83d665:::benchmark_data_smithi047_474001_object153:head# 2024-09-18T16:03:15.982 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1d849551:::benchmark_data_smithi047_474001_object395:head# 2024-09-18T16:03:15.982 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1df00925:::benchmark_data_smithi047_474001_object86:head# 2024-09-18T16:03:15.983 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1e14a4fe:::benchmark_data_smithi047_479866_object115:head# 2024-09-18T16:03:15.984 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1ea58aa2:::benchmark_data_smithi047_474001_object265:head# 2024-09-18T16:03:15.984 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1fa7c67c:::benchmark_data_smithi047_479866_object134:head# 2024-09-18T16:03:15.984 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:20724c42:::benchmark_data_smithi047_479866_object64:head# 2024-09-18T16:03:15.985 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:20be7217:::benchmark_data_smithi047_474001_object145:head# 2024-09-18T16:03:15.986 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:21551d46:::benchmark_data_smithi047_479866_object1:head# 2024-09-18T16:03:15.986 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:21c5c01f:::benchmark_data_smithi047_474001_object69:head# 2024-09-18T16:03:15.986 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:221f844d:::benchmark_data_smithi047_474001_object350:head# 2024-09-18T16:03:15.987 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:223dcfbd:::benchmark_data_smithi047_479866_object94:head# 2024-09-18T16:03:15.987 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:22d3be20:::benchmark_data_smithi047_474001_object315:head# 2024-09-18T16:03:15.988 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2334c1fe:::benchmark_data_smithi047_474001_object84:head# 2024-09-18T16:03:15.988 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:23ea4327:::benchmark_data_smithi047_479866_object148:head# 2024-09-18T16:03:15.989 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:24393629:::benchmark_data_smithi047_474001_object239:head# 2024-09-18T16:03:15.989 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:25482445:::benchmark_data_smithi047_474001_object53:head# 2024-09-18T16:03:15.990 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2555ae90:::benchmark_data_smithi047_474001_object370:head# 2024-09-18T16:03:15.990 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:26a1c85e:::benchmark_data_smithi047_474001_object32:head# 2024-09-18T16:03:15.990 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:26a53c7b:::benchmark_data_smithi047_479866_object98:head# 2024-09-18T16:03:15.991 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:26b7b87f:::benchmark_data_smithi047_474001_object107:head# 2024-09-18T16:03:15.991 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:26d7a228:::benchmark_data_smithi047_479866_object12:head# 2024-09-18T16:03:15.992 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:27190702:::benchmark_data_smithi047_474001_object218:head# 2024-09-18T16:03:15.993 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:27f9a01e:::benchmark_data_smithi047_474001_object349:head# 2024-09-18T16:03:15.993 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:284c9f3a:::benchmark_data_smithi047_474001_object254:head# 2024-09-18T16:03:15.993 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2856b42f:::benchmark_data_smithi047_474001_object81:head# 2024-09-18T16:03:15.994 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:28ca5326:::benchmark_data_smithi047_479866_object27:head# 2024-09-18T16:03:15.994 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:29494f84:::benchmark_data_smithi047_474001_object248:head# 2024-09-18T16:03:15.994 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2a1b326f:::benchmark_data_smithi047_479866_object151:head# 2024-09-18T16:03:15.995 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2a2243e8:::benchmark_data_smithi047_474001_object1:head# 2024-09-18T16:03:15.995 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2abea481:::benchmark_data_smithi047_474001_object23:head# 2024-09-18T16:03:15.995 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2b1d853f:::benchmark_data_smithi047_479866_object163:head# 2024-09-18T16:03:15.996 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2bcedee4:::benchmark_data_smithi047_479866_object86:head# 2024-09-18T16:03:15.996 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2c9f9183:::benchmark_data_smithi047_479866_object130:head# 2024-09-18T16:03:15.997 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2ced8b80:::benchmark_data_smithi047_474001_object401:head# 2024-09-18T16:03:15.997 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2e2d9e2b:::benchmark_data_smithi047_474001_object367:head# 2024-09-18T16:03:15.997 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2e9ea224:::benchmark_data_smithi047_474001_object106:head# 2024-09-18T16:03:15.997 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2fb8dbb3:::benchmark_data_smithi047_474001_object170:head# 2024-09-18T16:03:15.998 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:300fdc07:::benchmark_data_smithi047_474001_object227:head# 2024-09-18T16:03:15.998 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3078741e:::benchmark_data_smithi047_474001_object139:head# 2024-09-18T16:03:15.998 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:30d72d7d:::benchmark_data_smithi047_479866_object82:head# 2024-09-18T16:03:16.000 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:31669bfe:::benchmark_data_smithi047_474001_object319:head# 2024-09-18T16:03:16.000 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:321ae169:::benchmark_data_smithi047_474001_object244:head# 2024-09-18T16:03:16.000 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:32ff3ac8:::benchmark_data_smithi047_474001_object261:head# 2024-09-18T16:03:16.001 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:33134995:::benchmark_data_smithi047_474001_object310:head# 2024-09-18T16:03:16.001 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:338c48e8:::benchmark_data_smithi047_479866_object168:head# 2024-09-18T16:03:16.002 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:33933c4f:::benchmark_data_smithi047_479866_object22:head# 2024-09-18T16:03:16.003 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:33b1fdf8:::benchmark_data_smithi047_474001_object161:head# 2024-09-18T16:03:16.003 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:33b93a31:::benchmark_data_smithi047_474001_object403:head# 2024-09-18T16:03:16.003 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:33d9125e:::benchmark_data_smithi047_479866_object85:head# 2024-09-18T16:03:16.003 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:33fa48ba:::benchmark_data_smithi047_474001_object289:head# 2024-09-18T16:03:16.004 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:34348ae4:::benchmark_data_smithi047_474001_object6:head# 2024-09-18T16:03:16.004 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:348ab036:::benchmark_data_smithi047_479866_object129:head# 2024-09-18T16:03:16.005 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:34e0f433:::benchmark_data_smithi047_474001_object342:head# 2024-09-18T16:03:16.006 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:355eddbc:::benchmark_data_smithi047_474001_object198:head# 2024-09-18T16:03:16.006 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:35841d04:::benchmark_data_smithi047_474001_object186:head# 2024-09-18T16:03:16.006 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:35c32413:::benchmark_data_smithi047_474001_object413:head# 2024-09-18T16:03:16.006 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:35d09d62:::benchmark_data_smithi047_479866_object3:head# 2024-09-18T16:03:16.006 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:35db8d7b:::benchmark_data_smithi047_479866_object118:head# 2024-09-18T16:03:16.007 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:35e67788:::benchmark_data_smithi047_474001_object388:head# 2024-09-18T16:03:16.007 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:35fd02f5:::benchmark_data_smithi047_479866_object36:head# 2024-09-18T16:03:16.008 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3613c729:::benchmark_data_smithi047_474001_object380:head# 2024-09-18T16:03:16.008 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:365d6d16:::benchmark_data_smithi047_479866_object171:head# 2024-09-18T16:03:16.009 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:36f643e7:::benchmark_data_smithi047_479866_object113:head# 2024-09-18T16:03:16.009 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:37ccfdd9:::benchmark_data_smithi047_474001_object219:head# 2024-09-18T16:03:16.010 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:380af478:::benchmark_data_smithi047_474001_object359:head# 2024-09-18T16:03:16.010 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3812ffca:::benchmark_data_smithi047_474001_object286:head# 2024-09-18T16:03:16.011 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:381ae33c:::benchmark_data_smithi047_474001_object152:head# 2024-09-18T16:03:16.011 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3994ed2b:::benchmark_data_smithi047_474001_object162:head# 2024-09-18T16:03:16.012 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3ad1e94f:::benchmark_data_smithi047_474001_object26:head# 2024-09-18T16:03:16.012 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3b2460a7:::benchmark_data_smithi047_474001_object418:head# 2024-09-18T16:03:16.012 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3b70d910:::benchmark_data_smithi047_474001_object157:head# 2024-09-18T16:03:16.012 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3b94819a:::benchmark_data_smithi047_479866_object159:head# 2024-09-18T16:03:16.013 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3b976a95:::benchmark_data_smithi047_474001_object392:head# 2024-09-18T16:03:16.013 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3c01f944:::benchmark_data_smithi047_474001_object85:head# 2024-09-18T16:03:16.014 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3c08c862:::benchmark_data_smithi047_474001_object209:head# 2024-09-18T16:03:16.014 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3c3b4221:::benchmark_data_smithi047_474001_object302:head# 2024-09-18T16:03:16.014 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3c65ea78:::benchmark_data_smithi047_479866_object88:head# 2024-09-18T16:03:16.015 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3cc8b88e:::benchmark_data_smithi047_479866_object100:head# 2024-09-18T16:03:16.015 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3d6d3489:::benchmark_data_smithi047_479866_object23:head# 2024-09-18T16:03:16.015 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3dfa0593:::benchmark_data_smithi047_479866_object120:head# 2024-09-18T16:03:16.016 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3e01dac2:::benchmark_data_smithi047_479866_object25:head# 2024-09-18T16:03:16.016 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3e2bc2d1:::benchmark_data_smithi047_474001_object290:head# 2024-09-18T16:03:16.017 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3e83fb4d:::benchmark_data_smithi047_479866_object140:head# 2024-09-18T16:03:16.017 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3f0792f2:::benchmark_data_smithi047_474001_object205:head# 2024-09-18T16:03:16.018 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3f0bbc7e:::benchmark_data_smithi047_474001_object214:head# 2024-09-18T16:03:16.018 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:40e84920:::benchmark_data_smithi047_474001_object201:head# 2024-09-18T16:03:16.018 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:410710e5:::benchmark_data_smithi047_474001_object360:head# 2024-09-18T16:03:16.020 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4286a502:::benchmark_data_smithi047_474001_object280:head# 2024-09-18T16:03:16.020 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:429b03e9:::benchmark_data_smithi047_474001_object249:head# 2024-09-18T16:03:16.020 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:43c8723a:::benchmark_data_smithi047_474001_object4:head# 2024-09-18T16:03:16.020 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:43c8e265:::benchmark_data_smithi047_474001_object208:head# 2024-09-18T16:03:16.020 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:43d378b4:::benchmark_data_smithi047_474001_object425:head# 2024-09-18T16:03:16.020 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:43f9c3e1:::benchmark_data_smithi047_479866_object45:head# 2024-09-18T16:03:16.021 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:43fb4408:::benchmark_data_smithi047_474001_object399:head# 2024-09-18T16:03:16.021 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:44ab53e5:::benchmark_data_smithi047_474001_object27:head# 2024-09-18T16:03:16.021 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4515f62e:::benchmark_data_smithi047_474001_object143:head# 2024-09-18T16:03:16.022 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:457b93bc:::benchmark_data_smithi047_474001_object300:head# 2024-09-18T16:03:16.022 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:457e7435:::benchmark_data_smithi047_474001_object231:head# 2024-09-18T16:03:16.022 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4594ea0a:::benchmark_data_smithi047_479866_object87:head# 2024-09-18T16:03:16.023 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:45a3456f:::benchmark_data_smithi047_479866_object74:head# 2024-09-18T16:03:16.024 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:45f35e08:::benchmark_data_smithi047_479866_object99:head# 2024-09-18T16:03:16.024 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4640e213:::benchmark_data_smithi047_479866_object66:head# 2024-09-18T16:03:16.026 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:468b56fb:::benchmark_data_smithi047_474001_object242:head# 2024-09-18T16:03:16.026 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:47454808:::benchmark_data_smithi047_474001_object394:head# 2024-09-18T16:03:16.026 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4757d830:::benchmark_data_smithi047_474001_object147:head# 2024-09-18T16:03:16.026 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4790fa42:::benchmark_data_smithi047_474001_object16:head# 2024-09-18T16:03:16.027 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:481edef6:::benchmark_data_smithi047_474001_object129:head# 2024-09-18T16:03:16.027 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:498127df:::benchmark_data_smithi047_474001_object409:head# 2024-09-18T16:03:16.027 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4a94b2bf:::benchmark_data_smithi047_474001_object331:head# 2024-09-18T16:03:16.028 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4b0f17cd:::benchmark_data_smithi047_474001_object194:head# 2024-09-18T16:03:16.028 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4be7a76e:::benchmark_data_smithi047_479866_object108:head# 2024-09-18T16:03:16.028 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4ccceada:::benchmark_data_smithi047_474001_object213:head# 2024-09-18T16:03:16.028 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4ce89ee7:::benchmark_data_smithi047_479866_object26:head# 2024-09-18T16:03:16.029 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4d145d82:::benchmark_data_smithi047_474001_object206:head# 2024-09-18T16:03:16.029 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4dead740:::benchmark_data_smithi047_479866_object41:head# 2024-09-18T16:03:16.029 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4e67dcba:::benchmark_data_smithi047_474001_object144:head# 2024-09-18T16:03:16.030 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4e6bc7a4:::benchmark_data_smithi047_474001_object165:head# 2024-09-18T16:03:16.030 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4e75155f:::benchmark_data_smithi047_479866_object29:head# 2024-09-18T16:03:16.031 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4e7fe7b2:::benchmark_data_smithi047_474001_object431:head# 2024-09-18T16:03:16.031 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4f5e76df:::benchmark_data_smithi047_474001_object415:head# 2024-09-18T16:03:16.031 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5030ebeb:::benchmark_data_smithi047_474001_object335:head# 2024-09-18T16:03:16.032 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:505987dd:::benchmark_data_smithi047_474001_object313:head# 2024-09-18T16:03:16.032 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:50a1999f:::benchmark_data_smithi047_474001_object190:head# 2024-09-18T16:03:16.032 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:510b9f73:::benchmark_data_smithi047_479866_object125:head# 2024-09-18T16:03:16.033 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:51905d78:::benchmark_data_smithi047_474001_object371:head# 2024-09-18T16:03:16.033 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5275f435:::benchmark_data_smithi047_479866_object145:head# 2024-09-18T16:03:16.033 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5295c252:::benchmark_data_smithi047_479866_object91:head# 2024-09-18T16:03:16.034 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:53516fa7:::benchmark_data_smithi047_474001_object47:head# 2024-09-18T16:03:16.034 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:53a75224:::benchmark_data_smithi047_474001_object277:head# 2024-09-18T16:03:16.036 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:53bc5374:::benchmark_data_smithi047_474001_object304:head# 2024-09-18T16:03:16.037 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:54037397:::benchmark_data_smithi047_474001_object168:head# 2024-09-18T16:03:16.037 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:54e93f24:::benchmark_data_smithi047_474001_object351:head# 2024-09-18T16:03:16.037 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5549c508:::benchmark_data_smithi047_474001_object154:head# 2024-09-18T16:03:16.037 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:55a8f195:::benchmark_data_smithi047_474001_object183:head# 2024-09-18T16:03:16.037 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:58bb18eb:::benchmark_data_smithi047_474001_object92:head# 2024-09-18T16:03:16.037 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:58e3824a:::benchmark_data_smithi047_474001_object22:head# 2024-09-18T16:03:16.037 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:58eacbbf:::benchmark_data_smithi047_474001_object100:head# 2024-09-18T16:03:16.037 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:590d7cd0:::benchmark_data_smithi047_474001_object328:head# 2024-09-18T16:03:16.037 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:595ccf9f:::benchmark_data_smithi047_479866_object20:head# 2024-09-18T16:03:16.038 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:59701efb:::benchmark_data_smithi047_474001_object243:head# 2024-09-18T16:03:16.038 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:59852438:::benchmark_data_smithi047_474001_object83:head# 2024-09-18T16:03:16.038 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5a123740:::benchmark_data_smithi047_474001_object36:head# 2024-09-18T16:03:16.039 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5b73d6c3:::benchmark_data_smithi047_474001_object284:head# 2024-09-18T16:03:16.039 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5be37116:::benchmark_data_smithi047_479866_object33:head# 2024-09-18T16:03:16.039 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5c8e3731:::benchmark_data_smithi047_474001_object132:head# 2024-09-18T16:03:16.040 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5caa7c9f:::benchmark_data_smithi047_474001_object416:head# 2024-09-18T16:03:16.040 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5cf5ea91:::benchmark_data_smithi047_474001_object397:head# 2024-09-18T16:03:16.040 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5d1b58ad:::benchmark_data_smithi047_474001_object269:head# 2024-09-18T16:03:16.041 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5da527aa:::benchmark_data_smithi047_479866_object8:head# 2024-09-18T16:03:16.041 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5e669bb0:::benchmark_data_smithi047_474001_object405:head# 2024-09-18T16:03:16.042 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5eac6d5a:::benchmark_data_smithi047_474001_object411:head# 2024-09-18T16:03:16.043 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5ef5c1d9:::benchmark_data_smithi047_474001_object385:head# 2024-09-18T16:03:16.043 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5f7398fd:::benchmark_data_smithi047_474001_object68:head# 2024-09-18T16:03:16.043 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5f8ba50b:::benchmark_data_smithi047_474001_object174:head# 2024-09-18T16:03:16.044 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5fb38f21:::benchmark_data_smithi047_474001_object75:head# 2024-09-18T16:03:16.044 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5fc0bd7a:::benchmark_data_smithi047_479866_object4:head# 2024-09-18T16:03:16.046 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:60a5ff1a:::benchmark_data_smithi047_474001_object298:head# 2024-09-18T16:03:16.046 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:60e704db:::benchmark_data_smithi047_479866_object31:head# 2024-09-18T16:03:16.046 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6159ad07:::benchmark_data_smithi047_474001_object221:head# 2024-09-18T16:03:16.053 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:62273e63:::benchmark_data_smithi047_474001_object62:head# 2024-09-18T16:03:16.054 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:63cf18a2:::benchmark_data_smithi047_474001_object272:head# 2024-09-18T16:03:16.054 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:63f415e6:::benchmark_data_smithi047_474001_object58:head# 2024-09-18T16:03:16.054 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:64b93fc8:::benchmark_data_smithi047_479866_object150:head# 2024-09-18T16:03:16.054 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:64ff073d:::benchmark_data_smithi047_474001_object362:head# 2024-09-18T16:03:16.054 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:657bccc4:::benchmark_data_smithi047_474001_object169:head# 2024-09-18T16:03:16.054 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:660836d4:::benchmark_data_smithi047_474001_object39:head# 2024-09-18T16:03:16.055 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:66682ebd:::benchmark_data_smithi047_474001_object101:head# 2024-09-18T16:03:16.055 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:66e730e2:::benchmark_data_smithi047_474001_object61:head# 2024-09-18T16:03:16.055 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:66fbc3ec:::benchmark_data_smithi047_474001_object71:head# 2024-09-18T16:03:16.055 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6717adea:::benchmark_data_smithi047_474001_object317:head# 2024-09-18T16:03:16.055 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:67bf8aa9:::benchmark_data_smithi047_479866_object160:head# 2024-09-18T16:03:16.055 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:67e02299:::benchmark_data_smithi047_479866_object67:head# 2024-09-18T16:03:16.056 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:68016fab:::benchmark_data_smithi047_479866_object127:head# 2024-09-18T16:03:16.056 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:68107ec1:::benchmark_data_smithi047_474001_object195:head# 2024-09-18T16:03:16.056 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6812f8cb:::benchmark_data_smithi047_474001_object264:head# 2024-09-18T16:03:16.056 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:68f519f4:::benchmark_data_smithi047_474001_object343:head# 2024-09-18T16:03:16.056 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:694b8617:::benchmark_data_smithi047_479866_object121:head# 2024-09-18T16:03:16.063 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6a39a8d4:::benchmark_data_smithi047_474001_object66:head# 2024-09-18T16:03:16.063 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6ab23d26:::benchmark_data_smithi047_474001_object9:head# 2024-09-18T16:03:16.063 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6aeb612f:::benchmark_data_smithi047_479866_object43:head# 2024-09-18T16:03:16.063 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6b25d55a:::benchmark_data_smithi047_474001_object72:head# 2024-09-18T16:03:16.063 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6b5fc221:::benchmark_data_smithi047_474001_object334:head# 2024-09-18T16:03:16.064 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6c2d825c:::benchmark_data_smithi047_474001_object5:head# 2024-09-18T16:03:16.064 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6caa7eaf:::benchmark_data_smithi047_474001_object263:head# 2024-09-18T16:03:16.064 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6df9d633:::benchmark_data_smithi047_479866_object37:head# 2024-09-18T16:03:16.064 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6dfd6f4e:::benchmark_data_smithi047_474001_object251:head# 2024-09-18T16:03:16.064 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6e9248a9:::benchmark_data_smithi047_474001_object245:head# 2024-09-18T16:03:16.064 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6f38583d:::benchmark_data_smithi047_479866_object47:head# 2024-09-18T16:03:16.064 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6f58cdd1:::benchmark_data_smithi047_474001_object430:head# 2024-09-18T16:03:16.064 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6f9d3f5a:::benchmark_data_smithi047_479866_object73:head# 2024-09-18T16:03:16.064 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6fb45753:::benchmark_data_smithi047_479866_object103:head# 2024-09-18T16:03:16.064 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:71084a82:::benchmark_data_smithi047_474001_object398:head# 2024-09-18T16:03:16.065 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:712abb08:::benchmark_data_smithi047_479866_object13:head# 2024-09-18T16:03:16.065 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:712fda89:::benchmark_data_smithi047_479866_object141:head# 2024-09-18T16:03:16.065 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:715472aa:::benchmark_data_smithi047_479866_object32:head# 2024-09-18T16:03:16.065 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:722633c0:::benchmark_data_smithi047_474001_object7:head# 2024-09-18T16:03:16.065 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:73078ec4:::benchmark_data_smithi047_474001_object200:head# 2024-09-18T16:03:16.065 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7319939a:::benchmark_data_smithi047_474001_object378:head# 2024-09-18T16:03:16.065 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:732bbbc2:::benchmark_data_smithi047_474001_object113:head# 2024-09-18T16:03:16.065 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:73694cfd:::benchmark_data_smithi047_474001_object257:head# 2024-09-18T16:03:16.065 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:739a0e00:::benchmark_data_smithi047_474001_object11:head# 2024-09-18T16:03:16.065 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:74060ef1:::benchmark_data_smithi047_479866_object19:head# 2024-09-18T16:03:16.067 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:748a50de:::benchmark_data_smithi047_474001_object223:head# 2024-09-18T16:03:16.067 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:74cbfe52:::benchmark_data_smithi047_474001_object309:head# 2024-09-18T16:03:16.067 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:751b68bd:::benchmark_data_smithi047_474001_object45:head# 2024-09-18T16:03:16.067 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7634e380:::benchmark_data_smithi047_479866_object112:head# 2024-09-18T16:03:16.067 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:76583154:::benchmark_data_smithi047_474001_object13:head# 2024-09-18T16:03:16.067 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7669c90e:::benchmark_data_smithi047_474001_object216:head# 2024-09-18T16:03:16.067 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:76dc48ec:::benchmark_data_smithi047_474001_object79:head# 2024-09-18T16:03:16.068 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:77226950:::benchmark_data_smithi047_474001_object386:head# 2024-09-18T16:03:16.068 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:77a33037:::benchmark_data_smithi047_474001_object197:head# 2024-09-18T16:03:16.069 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:77f7974a:::benchmark_data_smithi047_474001_object123:head# 2024-09-18T16:03:16.069 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:78d90d4e:::benchmark_data_smithi047_474001_object301:head# 2024-09-18T16:03:16.069 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:78f80541:::benchmark_data_smithi047_479866_object173:head# 2024-09-18T16:03:16.070 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7ab906db:::benchmark_data_smithi047_479866_object131:head# 2024-09-18T16:03:16.071 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7b36e97b:::benchmark_data_smithi047_474001_object292:head# 2024-09-18T16:03:16.071 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7bde396b:::benchmark_data_smithi047_479866_object38:head# 2024-09-18T16:03:16.072 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7c6e25b9:::benchmark_data_smithi047_479866_object0:head# 2024-09-18T16:03:16.075 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7c70570e:::benchmark_data_smithi047_479866_object105:head# 2024-09-18T16:03:16.075 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7c8233bd:::benchmark_data_smithi047_474001_object76:head# 2024-09-18T16:03:16.075 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7cce9f64:::benchmark_data_smithi047_474001_object0:head# 2024-09-18T16:03:16.075 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7d8e31eb:::benchmark_data_smithi047_474001_object193:head# 2024-09-18T16:03:16.075 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7dc806f2:::benchmark_data_smithi047_474001_object126:head# 2024-09-18T16:03:16.075 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7dfaf25d:::benchmark_data_smithi047_474001_object260:head# 2024-09-18T16:03:16.075 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7e00f5f7:::benchmark_data_smithi047_474001_object98:head# 2024-09-18T16:03:16.075 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7f679bd9:::benchmark_data_smithi047_474001_object379:head# 2024-09-18T16:03:16.075 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7fa61220:::benchmark_data_smithi047_474001_object354:head# 2024-09-18T16:03:16.075 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7fb32424:::benchmark_data_smithi047_474001_object241:head# 2024-09-18T16:03:16.075 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:806a8ff0:::benchmark_data_smithi047_474001_object14:head# 2024-09-18T16:03:16.076 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8114a0ce:::benchmark_data_smithi047_479866_object84:head# 2024-09-18T16:03:16.076 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:811d44c1:::benchmark_data_smithi047_479866_object110:head# 2024-09-18T16:03:16.076 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:81d272b1:::benchmark_data_smithi047_479866_object78:head# 2024-09-18T16:03:16.077 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:81f3f33d:::benchmark_data_smithi047_479866_object119:head# 2024-09-18T16:03:16.078 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:827184ed:::benchmark_data_smithi047_474001_object202:head# 2024-09-18T16:03:16.078 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:829e9bf2:::benchmark_data_smithi047_479866_object135:head# 2024-09-18T16:03:16.078 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:83cb4be8:::benchmark_data_smithi047_474001_object189:head# 2024-09-18T16:03:16.079 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:83d7b3d6:::benchmark_data_smithi047_474001_object427:head# 2024-09-18T16:03:16.079 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:83e6a7ba:::benchmark_data_smithi047_474001_object326:head# 2024-09-18T16:03:16.079 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:83e714c1:::benchmark_data_smithi047_474001_object384:head# 2024-09-18T16:03:16.079 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:84fc4a07:::benchmark_data_smithi047_474001_object235:head# 2024-09-18T16:03:16.080 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:859c4c12:::benchmark_data_smithi047_479866_object54:head# 2024-09-18T16:03:16.081 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:85afe96e:::benchmark_data_smithi047_479866_object71:head# 2024-09-18T16:03:16.082 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8635a002:::benchmark_data_smithi047_474001_object8:head# 2024-09-18T16:03:16.082 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8659e9ad:::benchmark_data_smithi047_474001_object111:head# 2024-09-18T16:03:16.082 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:865f4a69:::benchmark_data_smithi047_474001_object312:head# 2024-09-18T16:03:16.082 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:87201b49:::benchmark_data_smithi047_474001_object185:head# 2024-09-18T16:03:16.082 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8727bd0b:::benchmark_data_smithi047_479866_object107:head# 2024-09-18T16:03:16.082 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:87f935e7:::benchmark_data_smithi047_479866_object62:head# 2024-09-18T16:03:16.083 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:889f60a0:::benchmark_data_smithi047_474001_object109:head# 2024-09-18T16:03:16.083 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8904e269:::benchmark_data_smithi047_474001_object228:head# 2024-09-18T16:03:16.083 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:89ec4c33:::benchmark_data_smithi047_474001_object133:head# 2024-09-18T16:03:16.083 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8a2258fa:::benchmark_data_smithi047_474001_object119:head# 2024-09-18T16:03:16.083 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8a67fb29:::benchmark_data_smithi047_479866_object136:head# 2024-09-18T16:03:16.085 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8aba1a3a:::benchmark_data_smithi047_474001_object291:head# 2024-09-18T16:03:16.085 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8cb0b5a1:::benchmark_data_smithi047_474001_object3:head# 2024-09-18T16:03:16.085 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8d65845a:::benchmark_data_smithi047_474001_object324:head# 2024-09-18T16:03:16.085 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8d83eb55:::benchmark_data_smithi047_479866_object142:head# 2024-09-18T16:03:16.085 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8e21472c:::benchmark_data_smithi047_474001_object110:head# 2024-09-18T16:03:16.085 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8e8844e6:::benchmark_data_smithi047_479866_object172:head# 2024-09-18T16:03:16.088 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8efa467b:::benchmark_data_smithi047_474001_object247:head# 2024-09-18T16:03:16.088 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8f162f27:::benchmark_data_smithi047_474001_object222:head# 2024-09-18T16:03:16.088 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8f2ab4de:::benchmark_data_smithi047_474001_object112:head# 2024-09-18T16:03:16.088 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8f3016de:::benchmark_data_smithi047_474001_object337:head# 2024-09-18T16:03:16.088 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8faff132:::benchmark_data_smithi047_479866_object167:head# 2024-09-18T16:03:16.089 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:906470c5:::benchmark_data_smithi047_474001_object108:head# 2024-09-18T16:03:16.089 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:90a887d3:::benchmark_data_smithi047_479866_object175:head# 2024-09-18T16:03:16.089 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:90a988ef:::benchmark_data_smithi047_474001_object391:head# 2024-09-18T16:03:16.089 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:90fba3e5:::benchmark_data_smithi047_479866_object111:head# 2024-09-18T16:03:16.101 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:91461d4c:::benchmark_data_smithi047_474001_object252:head# 2024-09-18T16:03:16.102 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:92438533:::benchmark_data_smithi047_474001_object17:head# 2024-09-18T16:03:16.102 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9256e853:::benchmark_data_smithi047_474001_object314:head# 2024-09-18T16:03:16.102 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:928bb93a:::benchmark_data_smithi047_474001_object51:head# 2024-09-18T16:03:16.102 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:92b0a2c6:::benchmark_data_smithi047_474001_object250:head# 2024-09-18T16:03:16.102 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9324d622:::benchmark_data_smithi047_474001_object95:head# 2024-09-18T16:03:16.102 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:939c1aa8:::benchmark_data_smithi047_474001_object382:head# 2024-09-18T16:03:16.102 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:94353a0c:::benchmark_data_smithi047_479866_object169:head# 2024-09-18T16:03:16.102 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:945bc203:::benchmark_data_smithi047_474001_object177:head# 2024-09-18T16:03:16.102 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:94707a46:::benchmark_data_smithi047_474001_object138:head# 2024-09-18T16:03:16.102 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:94f79b31:::benchmark_data_smithi047_474001_object253:head# 2024-09-18T16:03:16.103 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9502e979:::benchmark_data_smithi047_474001_object50:head# 2024-09-18T16:03:16.103 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:953c3a4c:::benchmark_data_smithi047_474001_object74:head# 2024-09-18T16:03:16.103 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:95921a67:::benchmark_data_smithi047_479866_object174:head# 2024-09-18T16:03:16.103 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:95e04f4b:::benchmark_data_smithi047_479866_object72:head# 2024-09-18T16:03:16.103 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:95edf5ad:::benchmark_data_smithi047_474001_object345:head# 2024-09-18T16:03:16.103 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9612cb9d:::benchmark_data_smithi047_474001_object59:head# 2024-09-18T16:03:16.103 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9670fdab:::benchmark_data_smithi047_474001_object116:head# 2024-09-18T16:03:16.103 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:978a53fd:::benchmark_data_smithi047_474001_object358:head# 2024-09-18T16:03:16.103 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:986d3d20:::benchmark_data_smithi047_474001_object38:head# 2024-09-18T16:03:16.103 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:989e240c:::benchmark_data_smithi047_479866_object77:head# 2024-09-18T16:03:16.103 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:99a611fc:::benchmark_data_smithi047_479866_object158:head# 2024-09-18T16:03:16.103 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9a0edcb5:::benchmark_data_smithi047_474001_object89:head# 2024-09-18T16:03:16.103 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9b79df93:::benchmark_data_smithi047_479866_object132:head# 2024-09-18T16:03:16.104 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9bef3576:::benchmark_data_smithi047_474001_object180:head# 2024-09-18T16:03:16.104 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9dafb813:::benchmark_data_smithi047_474001_object226:head# 2024-09-18T16:03:16.104 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9dc7b4a6:::benchmark_data_smithi047_474001_object346:head# 2024-09-18T16:03:16.104 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9de9d04a:::benchmark_data_smithi047_474001_object338:head# 2024-09-18T16:03:16.104 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9e17b853:::benchmark_data_smithi047_474001_object124:head# 2024-09-18T16:03:16.104 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9e534a18:::benchmark_data_smithi047_474001_object70:head# 2024-09-18T16:03:16.104 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9e5ea7f3:::benchmark_data_smithi047_474001_object348:head# 2024-09-18T16:03:16.104 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a0254336:::benchmark_data_smithi047_474001_object188:head# 2024-09-18T16:03:16.104 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a0c154fe:::benchmark_data_smithi047_474001_object383:head# 2024-09-18T16:03:16.104 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a16d06ab:::benchmark_data_smithi047_479866_object76:head# 2024-09-18T16:03:16.104 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a1981567:::benchmark_data_smithi047_474001_object414:head# 2024-09-18T16:03:16.105 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a23c3345:::benchmark_data_smithi047_474001_object408:head# 2024-09-18T16:03:16.105 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a244fe5e:::benchmark_data_smithi047_474001_object307:head# 2024-09-18T16:03:16.105 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a256b83b:::benchmark_data_smithi047_474001_object417:head# 2024-09-18T16:03:16.105 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a331eaa2:::benchmark_data_smithi047_474001_object103:head# 2024-09-18T16:03:16.105 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a3674a4c:::benchmark_data_smithi047_479866_object96:head# 2024-09-18T16:03:16.105 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a36fdbde:::benchmark_data_smithi047_474001_object125:head# 2024-09-18T16:03:16.105 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a388d7f6:::benchmark_data_smithi047_474001_object46:head# 2024-09-18T16:03:16.105 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a473749a:::benchmark_data_smithi047_474001_object396:head# 2024-09-18T16:03:16.105 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a49be1f1:::benchmark_data_smithi047_479866_object44:head# 2024-09-18T16:03:16.105 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a53e9cf3:::benchmark_data_smithi047_479866_object53:head# 2024-09-18T16:03:16.105 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a546fc38:::benchmark_data_smithi047_474001_object344:head# 2024-09-18T16:03:16.105 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a56684c3:::benchmark_data_smithi047_474001_object73:head# 2024-09-18T16:03:16.113 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a6a637ca:::benchmark_data_smithi047_474001_object352:head# 2024-09-18T16:03:16.113 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a6ac7233:::benchmark_data_smithi047_474001_object273:head# 2024-09-18T16:03:16.113 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a6be61c0:::benchmark_data_smithi047_474001_object212:head# 2024-09-18T16:03:16.113 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a6d03d77:::benchmark_data_smithi047_474001_object429:head# 2024-09-18T16:03:16.113 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a7029afc:::benchmark_data_smithi047_474001_object57:head# 2024-09-18T16:03:16.113 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a75b0434:::benchmark_data_smithi047_479866_object95:head# 2024-09-18T16:03:16.113 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a87abb60:::benchmark_data_smithi047_474001_object28:head# 2024-09-18T16:03:16.114 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a99c9566:::benchmark_data_smithi047_474001_object285:head# 2024-09-18T16:03:16.114 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a9ee7f09:::benchmark_data_smithi047_474001_object60:head# 2024-09-18T16:03:16.114 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:aaf069cf:::benchmark_data_smithi047_474001_object426:head# 2024-09-18T16:03:16.114 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ab22f0d1:::benchmark_data_smithi047_479866_object63:head# 2024-09-18T16:03:16.114 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ac9a95a5:::benchmark_data_smithi047_474001_object327:head# 2024-09-18T16:03:16.114 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ad75c20a:::benchmark_data_smithi047_474001_object322:head# 2024-09-18T16:03:16.114 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:add97b11:::benchmark_data_smithi047_474001_object120:head# 2024-09-18T16:03:16.114 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ae2bc6da:::benchmark_data_smithi047_474001_object374:head# 2024-09-18T16:03:16.114 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ae32eaa9:::benchmark_data_smithi047_474001_object217:head# 2024-09-18T16:03:16.114 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ae5f08a5:::benchmark_data_smithi047_474001_object156:head# 2024-09-18T16:03:16.114 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ae90731a:::benchmark_data_smithi047_474001_object275:head# 2024-09-18T16:03:16.114 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:aeb96a61:::benchmark_data_smithi047_474001_object35:head# 2024-09-18T16:03:16.114 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:af3252fa:::benchmark_data_smithi047_479866_object56:head# 2024-09-18T16:03:16.114 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:af8861c2:::benchmark_data_smithi047_474001_object19:head# 2024-09-18T16:03:16.115 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:afa325d5:::benchmark_data_smithi047_479866_object155:head# 2024-09-18T16:03:16.115 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:afa79d73:::benchmark_data_smithi047_474001_object297:head# 2024-09-18T16:03:16.115 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b06f5ff6:::benchmark_data_smithi047_474001_object96:head# 2024-09-18T16:03:16.125 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b09fcb0b:::benchmark_data_smithi047_474001_object330:head# 2024-09-18T16:03:16.125 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b0d19e8f:::benchmark_data_smithi047_479866_object116:head# 2024-09-18T16:03:16.133 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b0f4cb26:::benchmark_data_smithi047_474001_object262:head# 2024-09-18T16:03:16.135 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b0faad12:::benchmark_data_smithi047_479866_object89:head# 2024-09-18T16:03:16.150 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b1589877:::benchmark_data_smithi047_474001_object332:head# 2024-09-18T16:03:16.158 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b1c4832b:::benchmark_data_smithi047_479866_object157:head# 2024-09-18T16:03:16.158 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b1dd1fef:::benchmark_data_smithi047_474001_object134:head# 2024-09-18T16:03:16.159 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b2ada191:::benchmark_data_smithi047_479866_object59:head# 2024-09-18T16:03:16.159 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b3f3163f:::benchmark_data_smithi047_474001_object341:head# 2024-09-18T16:03:16.159 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b5265e20:::benchmark_data_smithi047_474001_object159:head# 2024-09-18T16:03:16.160 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b533e7e6:::benchmark_data_smithi047_474001_object78:head# 2024-09-18T16:03:16.170 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b539a2e0:::benchmark_data_smithi047_479866_object149:head# 2024-09-18T16:03:16.171 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b585c96f:::benchmark_data_smithi047_474001_object422:head# 2024-09-18T16:03:16.171 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b5d2794c:::benchmark_data_smithi047_479866_object10:head# 2024-09-18T16:03:16.172 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b5e588a2:::benchmark_data_smithi047_474001_object30:head# 2024-09-18T16:03:16.173 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b606ef85:::benchmark_data_smithi047_474001_object12:head# 2024-09-18T16:03:16.173 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b61ac151:::benchmark_data_smithi047_474001_object361:head# 2024-09-18T16:03:16.173 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b69470e5:::benchmark_data_smithi047_479866_object126:head# 2024-09-18T16:03:16.174 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b6f6d98b:::benchmark_data_smithi047_479866_object143:head# 2024-09-18T16:03:16.174 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b707ad78:::benchmark_data_smithi047_474001_object33:head# 2024-09-18T16:03:16.174 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b72090f3:::benchmark_data_smithi047_474001_object225:head# 2024-09-18T16:03:16.175 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b74f9085:::benchmark_data_smithi047_474001_object224:head# 2024-09-18T16:03:16.175 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b7d7a44f:::benchmark_data_smithi047_479866_object6:head# 2024-09-18T16:03:16.176 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b809a3ef:::benchmark_data_smithi047_479866_object11:head# 2024-09-18T16:03:16.176 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b81c2890:::benchmark_data_smithi047_479866_object83:head# 2024-09-18T16:03:16.177 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b854e709:::benchmark_data_smithi047_474001_object279:head# 2024-09-18T16:03:16.177 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b8615e71:::benchmark_last_metadata:head# 2024-09-18T16:03:16.177 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b88427b9:::benchmark_data_smithi047_474001_object204:head# 2024-09-18T16:03:16.178 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b8a01441:::benchmark_data_smithi047_474001_object99:head# 2024-09-18T16:03:16.178 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b8d632ef:::benchmark_data_smithi047_474001_object49:head# 2024-09-18T16:03:16.179 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b9a03ab3:::benchmark_data_smithi047_474001_object41:head# 2024-09-18T16:03:16.179 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ba233a46:::benchmark_data_smithi047_474001_object368:head# 2024-09-18T16:03:16.179 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ba80b0e7:::benchmark_data_smithi047_479866_object65:head# 2024-09-18T16:03:16.181 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:bb9dcf18:::benchmark_data_smithi047_474001_object306:head# 2024-09-18T16:03:16.181 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:bc2e5e68:::benchmark_data_smithi047_474001_object24:head# 2024-09-18T16:03:16.181 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:bc5f16d1:::benchmark_data_smithi047_479866_object16:head# 2024-09-18T16:03:16.182 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:bcb5235a:::benchmark_data_smithi047_479866_object146:head# 2024-09-18T16:03:16.182 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:bd291308:::benchmark_data_smithi047_474001_object316:head# 2024-09-18T16:03:16.182 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:bd68e671:::benchmark_data_smithi047_474001_object44:head# 2024-09-18T16:03:16.183 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:bdd1c304:::benchmark_data_smithi047_474001_object237:head# 2024-09-18T16:03:16.183 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:bdff383c:::benchmark_data_smithi047_479866_object7:head# 2024-09-18T16:03:16.185 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:be3f8098:::benchmark_data_smithi047_474001_object423:head# 2024-09-18T16:03:16.185 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:be77543b:::benchmark_data_smithi047_474001_object77:head# 2024-09-18T16:03:16.185 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:be86e7ec:::benchmark_data_smithi047_474001_object122:head# 2024-09-18T16:03:16.185 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:bed16cbf:::benchmark_data_smithi047_474001_object137:head# 2024-09-18T16:03:16.185 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:bf89fe68:::benchmark_data_smithi047_474001_object54:head# 2024-09-18T16:03:16.186 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:bfe9f633:::benchmark_data_smithi047_479866_object2:head# 2024-09-18T16:03:16.186 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c08db5a2:::benchmark_data_smithi047_474001_object236:head# 2024-09-18T16:03:16.187 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c18ab8fc:::benchmark_data_smithi047_474001_object131:head# 2024-09-18T16:03:16.187 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c1fcb644:::benchmark_data_smithi047_474001_object158:head# 2024-09-18T16:03:16.188 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c24a7c46:::benchmark_data_smithi047_474001_object282:head# 2024-09-18T16:03:16.188 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c313dcbc:::benchmark_data_smithi047_474001_object321:head# 2024-09-18T16:03:16.188 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c31fafef:::benchmark_data_smithi047_479866_object69:head# 2024-09-18T16:03:16.189 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c35a5205:::benchmark_data_smithi047_474001_object303:head# 2024-09-18T16:03:16.190 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c3d61783:::benchmark_data_smithi047_474001_object288:head# 2024-09-18T16:03:16.190 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c3fe223d:::benchmark_data_smithi047_474001_object356:head# 2024-09-18T16:03:16.190 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c4b79889:::benchmark_data_smithi047_474001_object325:head# 2024-09-18T16:03:16.190 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c4c4398f:::benchmark_data_smithi047_479866_object147:head# 2024-09-18T16:03:16.191 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c598be3a:::benchmark_data_smithi047_479866_object51:head# 2024-09-18T16:03:16.192 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c60c9d19:::benchmark_data_smithi047_474001_object146:head# 2024-09-18T16:03:16.192 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c6123b6f:::benchmark_data_smithi047_474001_object55:head# 2024-09-18T16:03:16.192 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c6869eae:::benchmark_data_smithi047_474001_object407:head# 2024-09-18T16:03:16.193 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c79249b8:::benchmark_data_smithi047_479866_object164:head# 2024-09-18T16:03:16.193 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c849831c:::benchmark_data_smithi047_474001_object118:head# 2024-09-18T16:03:16.194 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c960d8c2:::benchmark_data_smithi047_479866_object93:head# 2024-09-18T16:03:16.195 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c9eb3ea0:::benchmark_data_smithi047_474001_object87:head# 2024-09-18T16:03:16.195 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ca4e33a3:::benchmark_data_smithi047_474001_object283:head# 2024-09-18T16:03:16.195 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ca6f5c73:::benchmark_data_smithi047_479866_object52:head# 2024-09-18T16:03:16.196 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:cb18044d:::benchmark_data_smithi047_479866_object165:head# 2024-09-18T16:03:16.197 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:cb2f2b17:::benchmark_data_smithi047_479866_object114:head# 2024-09-18T16:03:16.198 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:cb6747e8:::benchmark_data_smithi047_474001_object357:head# 2024-09-18T16:03:16.198 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:cb84b5d4:::benchmark_data_smithi047_474001_object173:head# 2024-09-18T16:03:16.199 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:cbdaedd2:::benchmark_data_smithi047_474001_object179:head# 2024-09-18T16:03:16.199 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:cbef6232:::benchmark_data_smithi047_479866_object42:head# 2024-09-18T16:03:16.200 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:cc244efe:::benchmark_data_smithi047_479866_object97:head# 2024-09-18T16:03:16.200 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:cc26e7a6:::benchmark_data_smithi047_479866_object49:head# 2024-09-18T16:03:16.201 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:cc8b0a99:::benchmark_data_smithi047_479866_object144:head# 2024-09-18T16:03:16.201 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:cd271f33:::benchmark_data_smithi047_474001_object166:head# 2024-09-18T16:03:16.202 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:cd28a341:::benchmark_data_smithi047_474001_object150:head# 2024-09-18T16:03:16.202 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:cd4d6810:::benchmark_data_smithi047_474001_object363:head# 2024-09-18T16:03:16.202 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:cd5cc2dc:::benchmark_data_smithi047_474001_object88:head# 2024-09-18T16:03:16.203 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:cd77e455:::benchmark_data_smithi047_474001_object404:head# 2024-09-18T16:03:16.203 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:cdedbf36:::benchmark_data_smithi047_474001_object155:head# 2024-09-18T16:03:16.203 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ceb22576:::benchmark_data_smithi047_479866_object79:head# 2024-09-18T16:03:16.205 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d007de6f:::benchmark_data_smithi047_474001_object151:head# 2024-09-18T16:03:16.205 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d00fd5c1:::benchmark_data_smithi047_479866_object55:head# 2024-09-18T16:03:16.206 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d02aa9a9:::benchmark_data_smithi047_479866_object123:head# 2024-09-18T16:03:16.206 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d275e6f0:::benchmark_data_smithi047_474001_object419:head# 2024-09-18T16:03:16.206 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d28bdb82:::benchmark_data_smithi047_479866_object154:head# 2024-09-18T16:03:16.207 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d295be0f:::benchmark_data_smithi047_474001_object167:head# 2024-09-18T16:03:16.207 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d2992f10:::benchmark_data_smithi047_474001_object175:head# 2024-09-18T16:03:16.207 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d2cdad48:::benchmark_data_smithi047_474001_object90:head# 2024-09-18T16:03:16.208 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d43afad7:::benchmark_data_smithi047_474001_object376:head# 2024-09-18T16:03:16.208 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d564afb6:::benchmark_data_smithi047_479866_object117:head# 2024-09-18T16:03:16.209 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d63d4966:::benchmark_data_smithi047_479866_object162:head# 2024-09-18T16:03:16.209 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d6762583:::benchmark_data_smithi047_474001_object15:head# 2024-09-18T16:03:16.210 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d6d0e63b:::benchmark_data_smithi047_474001_object117:head# 2024-09-18T16:03:16.210 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d74454f2:::benchmark_data_smithi047_474001_object191:head# 2024-09-18T16:03:16.210 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d793612f:::benchmark_data_smithi047_479866_object80:head# 2024-09-18T16:03:16.211 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d7ba857c:::benchmark_data_smithi047_474001_object160:head# 2024-09-18T16:03:16.212 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d7ca34da:::benchmark_data_smithi047_474001_object40:head# 2024-09-18T16:03:16.212 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d7f93f31:::benchmark_data_smithi047_479866_object153:head# 2024-09-18T16:03:16.213 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d8141741:::benchmark_data_smithi047_474001_object287:head# 2024-09-18T16:03:16.213 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:da6a8dc2:::benchmark_data_smithi047_474001_object18:head# 2024-09-18T16:03:16.213 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:da6c2c4d:::benchmark_data_smithi047_474001_object31:head# 2024-09-18T16:03:16.214 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:db2849da:::benchmark_data_smithi047_479866_object24:head# 2024-09-18T16:03:16.214 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:db2f6939:::benchmark_data_smithi047_474001_object400:head# 2024-09-18T16:03:16.215 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:dbe821c7:::benchmark_data_smithi047_479866_object57:head# 2024-09-18T16:03:16.215 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:dc4893ce:::benchmark_data_smithi047_474001_object232:head# 2024-09-18T16:03:16.216 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:dc6daefe:::benchmark_data_smithi047_474001_object276:head# 2024-09-18T16:03:16.216 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:dcdad940:::benchmark_data_smithi047_474001_object182:head# 2024-09-18T16:03:16.216 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:dcef5ff5:::benchmark_data_smithi047_479866_object102:head# 2024-09-18T16:03:16.217 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:dd4ec68f:::benchmark_data_smithi047_474001_object56:head# 2024-09-18T16:03:16.217 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ddc4bb3d:::benchmark_data_smithi047_474001_object199:head# 2024-09-18T16:03:16.217 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ddece86f:::benchmark_data_smithi047_474001_object43:head# 2024-09-18T16:03:16.218 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:de8db7e0:::benchmark_data_smithi047_479866_object34:head# 2024-09-18T16:03:16.218 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ded47609:::benchmark_data_smithi047_479866_object109:head# 2024-09-18T16:03:16.219 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:df65649c:::benchmark_data_smithi047_474001_object176:head# 2024-09-18T16:03:16.219 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:dfa0b551:::benchmark_data_smithi047_474001_object421:head# 2024-09-18T16:03:16.219 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e07f15b6:::benchmark_data_smithi047_474001_object10:head# 2024-09-18T16:03:16.220 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e1757a69:::benchmark_data_smithi047_474001_object366:head# 2024-09-18T16:03:16.220 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e1859c12:::benchmark_data_smithi047_474001_object102:head# 2024-09-18T16:03:16.221 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e1d3a554:::benchmark_data_smithi047_474001_object369:head# 2024-09-18T16:03:16.221 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e437b5d2:::benchmark_data_smithi047_474001_object377:head# 2024-09-18T16:03:16.221 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e445afb5:::benchmark_data_smithi047_474001_object355:head# 2024-09-18T16:03:16.221 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e45ac62c:::benchmark_data_smithi047_479866_object128:head# 2024-09-18T16:03:16.222 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e61a1975:::benchmark_data_smithi047_474001_object296:head# 2024-09-18T16:03:16.222 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e61d1049:::benchmark_data_smithi047_479866_object28:head# 2024-09-18T16:03:16.223 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e65db1db:::benchmark_data_smithi047_474001_object329:head# 2024-09-18T16:03:16.223 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e65e637a:::benchmark_data_smithi047_474001_object424:head# 2024-09-18T16:03:16.224 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e69ffe03:::benchmark_data_smithi047_479866_object156:head# 2024-09-18T16:03:16.224 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e6b693e8:::benchmark_data_smithi047_474001_object308:head# 2024-09-18T16:03:16.224 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e6d85c1d:::benchmark_data_smithi047_474001_object220:head# 2024-09-18T16:03:16.225 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e6f3dc67:::benchmark_data_smithi047_474001_object293:head# 2024-09-18T16:03:16.225 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e6f460e2:::benchmark_data_smithi047_474001_object130:head# 2024-09-18T16:03:16.225 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e6f5814f:::benchmark_data_smithi047_474001_object64:head# 2024-09-18T16:03:16.226 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e7a2c4e1:::benchmark_data_smithi047_474001_object94:head# 2024-09-18T16:03:16.226 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e8b890a6:::benchmark_data_smithi047_479866_object176:head# 2024-09-18T16:03:16.227 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e8de693f:::benchmark_data_smithi047_479866_object122:head# 2024-09-18T16:03:16.228 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e9129935:::benchmark_data_smithi047_479866_object17:head# 2024-09-18T16:03:16.229 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e921a909:::benchmark_data_smithi047_479866_object101:head# 2024-09-18T16:03:16.229 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e945e46a:::benchmark_data_smithi047_474001_object171:head# 2024-09-18T16:03:16.229 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e9b2d046:::benchmark_data_smithi047_479866_object18:head# 2024-09-18T16:03:16.230 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e9b3c7f8:::benchmark_data_smithi047_474001_object181:head# 2024-09-18T16:03:16.230 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e9bb35fb:::benchmark_data_smithi047_479866_object58:head# 2024-09-18T16:03:16.231 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e9bd9230:::benchmark_data_smithi047_479866_object15:head# 2024-09-18T16:03:16.231 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ea0dab4b:::benchmark_data_smithi047_474001_object406:head# 2024-09-18T16:03:16.232 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ea0dd8c2:::benchmark_data_smithi047_474001_object381:head# 2024-09-18T16:03:16.232 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ea20ab79:::benchmark_data_smithi047_474001_object178:head# 2024-09-18T16:03:16.233 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ea95d8c2:::benchmark_data_smithi047_474001_object238:head# 2024-09-18T16:03:16.233 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ecaeb389:::benchmark_data_smithi047_474001_object135:head# 2024-09-18T16:03:16.233 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ed1fe358:::benchmark_data_smithi047_479866_object106:head# 2024-09-18T16:03:16.234 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ed907482:::benchmark_data_smithi047_474001_object34:head# 2024-09-18T16:03:16.235 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ee898c6a:::benchmark_data_smithi047_474001_object184:head# 2024-09-18T16:03:16.235 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:eeceac30:::benchmark_data_smithi047_474001_object97:head# 2024-09-18T16:03:16.235 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:eee17d95:::benchmark_data_smithi047_474001_object187:head# 2024-09-18T16:03:16.235 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:eefc7324:::benchmark_data_smithi047_474001_object91:head# 2024-09-18T16:03:16.239 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ef61b0b9:::benchmark_data_smithi047_474001_object428:head# 2024-09-18T16:03:16.239 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:efbee6f8:::benchmark_data_smithi047_474001_object211:head# 2024-09-18T16:03:16.239 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:eff64798:::benchmark_data_smithi047_474001_object93:head# 2024-09-18T16:03:16.239 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f09ab14b:::benchmark_data_smithi047_474001_object339:head# 2024-09-18T16:03:16.239 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f09c2c79:::benchmark_data_smithi047_474001_object258:head# 2024-09-18T16:03:16.240 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f0a2b2e0:::benchmark_data_smithi047_474001_object234:head# 2024-09-18T16:03:16.240 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f0e21778:::benchmark_data_smithi047_474001_object210:head# 2024-09-18T16:03:16.240 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f12591b2:::benchmark_data_smithi047_474001_object142:head# 2024-09-18T16:03:16.240 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f15f0e45:::benchmark_data_smithi047_474001_object353:head# 2024-09-18T16:03:16.240 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f17effda:::benchmark_data_smithi047_474001_object121:head# 2024-09-18T16:03:16.240 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f188e48b:::benchmark_data_smithi047_474001_object82:head# 2024-09-18T16:03:16.240 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f286bf95:::benchmark_data_smithi047_474001_object115:head# 2024-09-18T16:03:16.240 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f400c3e0:::benchmark_data_smithi047_474001_object305:head# 2024-09-18T16:03:16.240 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f40d5282:::benchmark_data_smithi047_474001_object48:head# 2024-09-18T16:03:16.240 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f4d9af26:::benchmark_data_smithi047_474001_object311:head# 2024-09-18T16:03:16.240 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f50aa0e5:::benchmark_data_smithi047_474001_object65:head# 2024-09-18T16:03:16.240 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f58a2ed4:::benchmark_data_smithi047_474001_object393:head# 2024-09-18T16:03:16.240 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f5cd98c8:::benchmark_data_smithi047_479866_object14:head# 2024-09-18T16:03:16.241 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f61edbb2:::benchmark_data_smithi047_474001_object140:head# 2024-09-18T16:03:16.242 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f62ec792:::benchmark_data_smithi047_479866_object81:head# 2024-09-18T16:03:16.242 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f6596924:::benchmark_data_smithi047_474001_object256:head# 2024-09-18T16:03:16.242 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f68bfbc5:::benchmark_data_smithi047_474001_object295:head# 2024-09-18T16:03:16.242 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f7031fa9:::benchmark_data_smithi047_474001_object318:head# 2024-09-18T16:03:16.242 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f8576353:::benchmark_data_smithi047_474001_object336:head# 2024-09-18T16:03:16.242 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f87e20c7:::benchmark_data_smithi047_479866_object68:head# 2024-09-18T16:03:16.242 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f8e14a8e:::benchmark_data_smithi047_474001_object63:head# 2024-09-18T16:03:16.243 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f9e01ed3:::benchmark_data_smithi047_474001_object20:head# 2024-09-18T16:03:16.243 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fa676313:::benchmark_data_smithi047_474001_object268:head# 2024-09-18T16:03:16.243 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fb2704e8:::benchmark_data_smithi047_474001_object294:head# 2024-09-18T16:03:16.244 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fb557b5c:::benchmark_data_smithi047_474001_object25:head# 2024-09-18T16:03:16.245 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fba861de:::benchmark_data_smithi047_474001_object233:head# 2024-09-18T16:03:16.245 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fbc5f8c9:::benchmark_data_smithi047_479866_object75:head# 2024-09-18T16:03:16.245 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fc267536:::benchmark_data_smithi047_479866_object124:head# 2024-09-18T16:03:16.245 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fc48c5f5:::benchmark_data_smithi047_474001_object128:head# 2024-09-18T16:03:16.246 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fcc00bd9:::benchmark_data_smithi047_474001_object267:head# 2024-09-18T16:03:16.246 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fcc267e2:::benchmark_data_smithi047_474001_object420:head# 2024-09-18T16:03:16.246 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fcd59592:::benchmark_data_smithi047_474001_object387:head# 2024-09-18T16:03:16.247 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fcff1581:::benchmark_data_smithi047_474001_object229:head# 2024-09-18T16:03:16.247 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fd5c7c49:::benchmark_data_smithi047_474001_object375:head# 2024-09-18T16:03:16.247 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fdc8029d:::benchmark_data_smithi047_479866_object50:head# 2024-09-18T16:03:16.248 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fdd0ff05:::benchmark_data_smithi047_474001_object67:head# 2024-09-18T16:03:16.249 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fdf729e9:::benchmark_data_smithi047_474001_object230:head# 2024-09-18T16:03:16.249 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fe6bbe76:::benchmark_data_smithi047_479866_object61:head# 2024-09-18T16:03:16.249 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fe6c9d0e:::benchmark_data_smithi047_474001_object52:head# 2024-09-18T16:03:16.250 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fe938a07:::benchmark_data_smithi047_474001_object207:head# 2024-09-18T16:03:16.250 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fe96169b:::benchmark_data_smithi047_479866_object9:head# 2024-09-18T16:03:16.251 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fea2446c:::benchmark_data_smithi047_474001_object105:head# 2024-09-18T16:03:16.251 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:feb4dfe4:::benchmark_data_smithi047_479866_object152:head# 2024-09-18T16:03:16.252 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fee86c8a:::benchmark_data_smithi047_474001_object412:head# 2024-09-18T16:03:16.252 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ff03a3e6:::benchmark_data_smithi047_474001_object37:head# 2024-09-18T16:03:16.253 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ff401af6:::benchmark_data_smithi047_479866_object90:head# 2024-09-18T16:03:16.253 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ff8f6396:::benchmark_data_smithi047_474001_object114:head# 2024-09-18T16:03:16.254 INFO:tasks.workunit.client.0.smithi047.stdout: marking collection for removal 2024-09-18T16:03:16.254 INFO:tasks.workunit.client.0.smithi047.stderr:Export successful 2024-09-18T16:03:16.254 INFO:tasks.workunit.client.0.smithi047.stdout:setting '_remove' omap key 2024-09-18T16:03:16.254 INFO:tasks.workunit.client.0.smithi047.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-09-18T16:03:16.928 INFO:tasks.workunit.client.0.smithi047.stderr:Remove successful 2024-09-18T16:03:17.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:143: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.1 --file td/pg-split-merge/1.1 2024-09-18T16:03:18.832 INFO:tasks.workunit.client.0.smithi047.stdout:get_pg_num_history pg_num_history pg_num_history(e228 pg_nums {1={19=1}} deleted_pools ) 2024-09-18T16:03:18.832 INFO:tasks.workunit.client.0.smithi047.stdout:Importing pgid 1.1 2024-09-18T16:03:18.833 INFO:tasks.workunit.client.0.smithi047.stderr:PG export's map 9 is older than OSD's oldest_map 121 2024-09-18T16:03:18.833 INFO:tasks.workunit.client.0.smithi047.stderr: pass --force to proceed anyway (with incomplete PastIntervals) 2024-09-18T16:03:19.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:144: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0 2024-09-18T16:03:20.581 INFO:tasks.workunit.client.0.smithi047.stdout:get_pg_num_history pg_num_history pg_num_history(e228 pg_nums {1={19=1}} deleted_pools ) 2024-09-18T16:03:20.581 INFO:tasks.workunit.client.0.smithi047.stdout:Importing pgid 1.0 2024-09-18T16:03:20.581 INFO:tasks.workunit.client.0.smithi047.stderr:PG export's map 9 is older than OSD's oldest_map 121 2024-09-18T16:03:20.581 INFO:tasks.workunit.client.0.smithi047.stderr: pass --force to proceed anyway (with incomplete PastIntervals) 2024-09-18T16:03:21.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:147: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.1 --file td/pg-split-merge/1.1 --force 2024-09-18T16:03:22.323 INFO:tasks.workunit.client.0.smithi047.stdout:get_pg_num_history pg_num_history pg_num_history(e228 pg_nums {1={19=1}} deleted_pools ) 2024-09-18T16:03:22.323 INFO:tasks.workunit.client.0.smithi047.stdout:Importing pgid 1.1 2024-09-18T16:03:22.324 INFO:tasks.workunit.client.0.smithi047.stdout:pool 1 pg_num 2 -> 1 2024-09-18T16:03:22.324 INFO:tasks.workunit.client.0.smithi047.stderr:PG export's map 9 is older than OSD's oldest_map 121 2024-09-18T16:03:22.324 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:806a8ff0:::benchmark_data_smithi047_474001_object14:head# 2024-09-18T16:03:22.325 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:22.378 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:827184ed:::benchmark_data_smithi047_474001_object202:head# 2024-09-18T16:03:22.379 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:22.429 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:83cb4be8:::benchmark_data_smithi047_474001_object189:head# 2024-09-18T16:03:22.429 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:22.479 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:83d7b3d6:::benchmark_data_smithi047_474001_object427:head# 2024-09-18T16:03:22.479 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:22.582 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:83e6a7ba:::benchmark_data_smithi047_474001_object326:head# 2024-09-18T16:03:22.582 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:22.656 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:83e714c1:::benchmark_data_smithi047_474001_object384:head# 2024-09-18T16:03:22.656 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:22.698 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:84fc4a07:::benchmark_data_smithi047_474001_object235:head# 2024-09-18T16:03:22.698 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:22.739 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8635a002:::benchmark_data_smithi047_474001_object8:head# 2024-09-18T16:03:22.739 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:22.781 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8659e9ad:::benchmark_data_smithi047_474001_object111:head# 2024-09-18T16:03:22.781 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:22.832 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:865f4a69:::benchmark_data_smithi047_474001_object312:head# 2024-09-18T16:03:22.832 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:22.899 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:87201b49:::benchmark_data_smithi047_474001_object185:head# 2024-09-18T16:03:22.899 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:22.949 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:889f60a0:::benchmark_data_smithi047_474001_object109:head# 2024-09-18T16:03:22.949 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:22.999 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8904e269:::benchmark_data_smithi047_474001_object228:head# 2024-09-18T16:03:22.999 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:23.049 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:89ec4c33:::benchmark_data_smithi047_474001_object133:head# 2024-09-18T16:03:23.049 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:23.100 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8a2258fa:::benchmark_data_smithi047_474001_object119:head# 2024-09-18T16:03:23.100 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:23.150 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8aba1a3a:::benchmark_data_smithi047_474001_object291:head# 2024-09-18T16:03:23.150 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:23.200 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8cb0b5a1:::benchmark_data_smithi047_474001_object3:head# 2024-09-18T16:03:23.200 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:23.242 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8d65845a:::benchmark_data_smithi047_474001_object324:head# 2024-09-18T16:03:23.242 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:23.292 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8e21472c:::benchmark_data_smithi047_474001_object110:head# 2024-09-18T16:03:23.292 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:23.342 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8efa467b:::benchmark_data_smithi047_474001_object247:head# 2024-09-18T16:03:23.343 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:23.393 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8f162f27:::benchmark_data_smithi047_474001_object222:head# 2024-09-18T16:03:23.393 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:23.435 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8f2ab4de:::benchmark_data_smithi047_474001_object112:head# 2024-09-18T16:03:23.435 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:23.485 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8f3016de:::benchmark_data_smithi047_474001_object337:head# 2024-09-18T16:03:23.485 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:23.527 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:906470c5:::benchmark_data_smithi047_474001_object108:head# 2024-09-18T16:03:23.527 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:23.577 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:90a988ef:::benchmark_data_smithi047_474001_object391:head# 2024-09-18T16:03:23.578 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:23.627 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:91461d4c:::benchmark_data_smithi047_474001_object252:head# 2024-09-18T16:03:23.627 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:23.677 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:92438533:::benchmark_data_smithi047_474001_object17:head# 2024-09-18T16:03:23.677 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:23.728 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9256e853:::benchmark_data_smithi047_474001_object314:head# 2024-09-18T16:03:23.728 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:23.778 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:928bb93a:::benchmark_data_smithi047_474001_object51:head# 2024-09-18T16:03:23.778 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:23.828 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:92b0a2c6:::benchmark_data_smithi047_474001_object250:head# 2024-09-18T16:03:23.828 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:23.878 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9324d622:::benchmark_data_smithi047_474001_object95:head# 2024-09-18T16:03:23.879 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:23.929 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:939c1aa8:::benchmark_data_smithi047_474001_object382:head# 2024-09-18T16:03:23.929 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:23.979 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:945bc203:::benchmark_data_smithi047_474001_object177:head# 2024-09-18T16:03:23.979 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:24.029 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:94707a46:::benchmark_data_smithi047_474001_object138:head# 2024-09-18T16:03:24.029 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:24.079 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:94f79b31:::benchmark_data_smithi047_474001_object253:head# 2024-09-18T16:03:24.080 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:24.139 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9502e979:::benchmark_data_smithi047_474001_object50:head# 2024-09-18T16:03:24.139 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:24.230 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:953c3a4c:::benchmark_data_smithi047_474001_object74:head# 2024-09-18T16:03:24.231 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:24.281 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:95edf5ad:::benchmark_data_smithi047_474001_object345:head# 2024-09-18T16:03:24.281 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:24.323 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9612cb9d:::benchmark_data_smithi047_474001_object59:head# 2024-09-18T16:03:24.323 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:24.365 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9670fdab:::benchmark_data_smithi047_474001_object116:head# 2024-09-18T16:03:24.365 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:24.406 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:978a53fd:::benchmark_data_smithi047_474001_object358:head# 2024-09-18T16:03:24.407 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:24.448 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:986d3d20:::benchmark_data_smithi047_474001_object38:head# 2024-09-18T16:03:24.448 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:24.507 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9a0edcb5:::benchmark_data_smithi047_474001_object89:head# 2024-09-18T16:03:24.507 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:24.599 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9bef3576:::benchmark_data_smithi047_474001_object180:head# 2024-09-18T16:03:24.599 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:24.683 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9dafb813:::benchmark_data_smithi047_474001_object226:head# 2024-09-18T16:03:24.683 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:24.742 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9dc7b4a6:::benchmark_data_smithi047_474001_object346:head# 2024-09-18T16:03:24.742 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:24.792 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9de9d04a:::benchmark_data_smithi047_474001_object338:head# 2024-09-18T16:03:24.792 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:24.842 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9e17b853:::benchmark_data_smithi047_474001_object124:head# 2024-09-18T16:03:24.842 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:24.892 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9e534a18:::benchmark_data_smithi047_474001_object70:head# 2024-09-18T16:03:24.892 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:24.942 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9e5ea7f3:::benchmark_data_smithi047_474001_object348:head# 2024-09-18T16:03:24.943 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:25.009 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a0254336:::benchmark_data_smithi047_474001_object188:head# 2024-09-18T16:03:25.010 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:25.093 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a0c154fe:::benchmark_data_smithi047_474001_object383:head# 2024-09-18T16:03:25.093 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:25.189 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a1981567:::benchmark_data_smithi047_474001_object414:head# 2024-09-18T16:03:25.189 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:25.269 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a23c3345:::benchmark_data_smithi047_474001_object408:head# 2024-09-18T16:03:25.269 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:25.345 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a244fe5e:::benchmark_data_smithi047_474001_object307:head# 2024-09-18T16:03:25.345 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:25.395 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a256b83b:::benchmark_data_smithi047_474001_object417:head# 2024-09-18T16:03:25.395 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:25.445 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a331eaa2:::benchmark_data_smithi047_474001_object103:head# 2024-09-18T16:03:25.445 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:25.495 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a36fdbde:::benchmark_data_smithi047_474001_object125:head# 2024-09-18T16:03:25.496 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:25.546 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a388d7f6:::benchmark_data_smithi047_474001_object46:head# 2024-09-18T16:03:25.546 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:25.596 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a473749a:::benchmark_data_smithi047_474001_object396:head# 2024-09-18T16:03:25.596 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:25.646 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a546fc38:::benchmark_data_smithi047_474001_object344:head# 2024-09-18T16:03:25.646 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:25.696 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a56684c3:::benchmark_data_smithi047_474001_object73:head# 2024-09-18T16:03:25.697 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:25.755 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a6a637ca:::benchmark_data_smithi047_474001_object352:head# 2024-09-18T16:03:25.756 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:25.822 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a6ac7233:::benchmark_data_smithi047_474001_object273:head# 2024-09-18T16:03:25.823 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:25.873 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a6be61c0:::benchmark_data_smithi047_474001_object212:head# 2024-09-18T16:03:25.873 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:25.923 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a6d03d77:::benchmark_data_smithi047_474001_object429:head# 2024-09-18T16:03:25.923 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:25.973 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a7029afc:::benchmark_data_smithi047_474001_object57:head# 2024-09-18T16:03:25.973 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:26.015 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a87abb60:::benchmark_data_smithi047_474001_object28:head# 2024-09-18T16:03:26.015 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:26.057 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a99c9566:::benchmark_data_smithi047_474001_object285:head# 2024-09-18T16:03:26.057 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:26.099 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a9ee7f09:::benchmark_data_smithi047_474001_object60:head# 2024-09-18T16:03:26.099 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:26.166 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:aaf069cf:::benchmark_data_smithi047_474001_object426:head# 2024-09-18T16:03:26.166 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:26.233 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ac9a95a5:::benchmark_data_smithi047_474001_object327:head# 2024-09-18T16:03:26.233 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:26.283 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ad75c20a:::benchmark_data_smithi047_474001_object322:head# 2024-09-18T16:03:26.283 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:26.333 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:add97b11:::benchmark_data_smithi047_474001_object120:head# 2024-09-18T16:03:26.334 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:26.384 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ae2bc6da:::benchmark_data_smithi047_474001_object374:head# 2024-09-18T16:03:26.384 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:26.434 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ae32eaa9:::benchmark_data_smithi047_474001_object217:head# 2024-09-18T16:03:26.434 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:26.484 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ae5f08a5:::benchmark_data_smithi047_474001_object156:head# 2024-09-18T16:03:26.484 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:26.584 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ae90731a:::benchmark_data_smithi047_474001_object275:head# 2024-09-18T16:03:26.585 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:26.685 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:aeb96a61:::benchmark_data_smithi047_474001_object35:head# 2024-09-18T16:03:26.685 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:26.735 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:af8861c2:::benchmark_data_smithi047_474001_object19:head# 2024-09-18T16:03:26.736 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:26.785 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:afa79d73:::benchmark_data_smithi047_474001_object297:head# 2024-09-18T16:03:26.786 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:26.827 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b06f5ff6:::benchmark_data_smithi047_474001_object96:head# 2024-09-18T16:03:26.828 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:26.878 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b09fcb0b:::benchmark_data_smithi047_474001_object330:head# 2024-09-18T16:03:26.878 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:26.920 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b0f4cb26:::benchmark_data_smithi047_474001_object262:head# 2024-09-18T16:03:26.920 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:26.970 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b1589877:::benchmark_data_smithi047_474001_object332:head# 2024-09-18T16:03:26.970 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:27.020 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b1dd1fef:::benchmark_data_smithi047_474001_object134:head# 2024-09-18T16:03:27.020 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:27.070 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b3f3163f:::benchmark_data_smithi047_474001_object341:head# 2024-09-18T16:03:27.070 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:27.121 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b5265e20:::benchmark_data_smithi047_474001_object159:head# 2024-09-18T16:03:27.121 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:27.171 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b533e7e6:::benchmark_data_smithi047_474001_object78:head# 2024-09-18T16:03:27.171 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:27.221 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b585c96f:::benchmark_data_smithi047_474001_object422:head# 2024-09-18T16:03:27.222 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:27.299 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b5e588a2:::benchmark_data_smithi047_474001_object30:head# 2024-09-18T16:03:27.299 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:27.372 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b606ef85:::benchmark_data_smithi047_474001_object12:head# 2024-09-18T16:03:27.372 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:27.422 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b61ac151:::benchmark_data_smithi047_474001_object361:head# 2024-09-18T16:03:27.422 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:27.472 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b707ad78:::benchmark_data_smithi047_474001_object33:head# 2024-09-18T16:03:27.473 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:27.514 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b72090f3:::benchmark_data_smithi047_474001_object225:head# 2024-09-18T16:03:27.515 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:27.565 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b74f9085:::benchmark_data_smithi047_474001_object224:head# 2024-09-18T16:03:27.565 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:27.615 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b854e709:::benchmark_data_smithi047_474001_object279:head# 2024-09-18T16:03:27.615 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:27.657 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b8615e71:::benchmark_last_metadata:head# 2024-09-18T16:03:27.657 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:27.699 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b88427b9:::benchmark_data_smithi047_474001_object204:head# 2024-09-18T16:03:27.699 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:27.741 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b8a01441:::benchmark_data_smithi047_474001_object99:head# 2024-09-18T16:03:27.741 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:27.783 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b8d632ef:::benchmark_data_smithi047_474001_object49:head# 2024-09-18T16:03:27.783 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:27.825 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b9a03ab3:::benchmark_data_smithi047_474001_object41:head# 2024-09-18T16:03:27.825 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:27.866 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ba233a46:::benchmark_data_smithi047_474001_object368:head# 2024-09-18T16:03:27.866 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:27.917 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:bb9dcf18:::benchmark_data_smithi047_474001_object306:head# 2024-09-18T16:03:27.917 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:27.967 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:bc2e5e68:::benchmark_data_smithi047_474001_object24:head# 2024-09-18T16:03:27.967 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:28.017 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:bd291308:::benchmark_data_smithi047_474001_object316:head# 2024-09-18T16:03:28.017 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:28.067 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:bd68e671:::benchmark_data_smithi047_474001_object44:head# 2024-09-18T16:03:28.067 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:28.118 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:bdd1c304:::benchmark_data_smithi047_474001_object237:head# 2024-09-18T16:03:28.118 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:28.201 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:be3f8098:::benchmark_data_smithi047_474001_object423:head# 2024-09-18T16:03:28.202 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:28.252 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:be77543b:::benchmark_data_smithi047_474001_object77:head# 2024-09-18T16:03:28.252 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:28.302 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:be86e7ec:::benchmark_data_smithi047_474001_object122:head# 2024-09-18T16:03:28.302 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:28.352 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:bed16cbf:::benchmark_data_smithi047_474001_object137:head# 2024-09-18T16:03:28.352 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:28.394 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:bf89fe68:::benchmark_data_smithi047_474001_object54:head# 2024-09-18T16:03:28.395 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:28.436 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c08db5a2:::benchmark_data_smithi047_474001_object236:head# 2024-09-18T16:03:28.436 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:28.486 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c18ab8fc:::benchmark_data_smithi047_474001_object131:head# 2024-09-18T16:03:28.487 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:28.587 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c1fcb644:::benchmark_data_smithi047_474001_object158:head# 2024-09-18T16:03:28.587 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:28.679 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c24a7c46:::benchmark_data_smithi047_474001_object282:head# 2024-09-18T16:03:28.680 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:28.729 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c313dcbc:::benchmark_data_smithi047_474001_object321:head# 2024-09-18T16:03:28.729 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:28.780 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c35a5205:::benchmark_data_smithi047_474001_object303:head# 2024-09-18T16:03:28.780 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:28.830 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c3d61783:::benchmark_data_smithi047_474001_object288:head# 2024-09-18T16:03:28.830 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:28.889 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c3fe223d:::benchmark_data_smithi047_474001_object356:head# 2024-09-18T16:03:28.889 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:28.939 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c4b79889:::benchmark_data_smithi047_474001_object325:head# 2024-09-18T16:03:28.939 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:28.989 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c60c9d19:::benchmark_data_smithi047_474001_object146:head# 2024-09-18T16:03:28.989 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:29.039 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c6123b6f:::benchmark_data_smithi047_474001_object55:head# 2024-09-18T16:03:29.039 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:29.090 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c6869eae:::benchmark_data_smithi047_474001_object407:head# 2024-09-18T16:03:29.090 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:29.132 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c849831c:::benchmark_data_smithi047_474001_object118:head# 2024-09-18T16:03:29.132 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:29.182 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c9eb3ea0:::benchmark_data_smithi047_474001_object87:head# 2024-09-18T16:03:29.182 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:29.224 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ca4e33a3:::benchmark_data_smithi047_474001_object283:head# 2024-09-18T16:03:29.224 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:29.274 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cb6747e8:::benchmark_data_smithi047_474001_object357:head# 2024-09-18T16:03:29.274 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:29.316 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cb84b5d4:::benchmark_data_smithi047_474001_object173:head# 2024-09-18T16:03:29.316 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:29.358 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cbdaedd2:::benchmark_data_smithi047_474001_object179:head# 2024-09-18T16:03:29.358 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:29.400 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cd271f33:::benchmark_data_smithi047_474001_object166:head# 2024-09-18T16:03:29.400 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:29.442 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cd28a341:::benchmark_data_smithi047_474001_object150:head# 2024-09-18T16:03:29.442 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:29.484 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cd4d6810:::benchmark_data_smithi047_474001_object363:head# 2024-09-18T16:03:29.484 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:29.534 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cd5cc2dc:::benchmark_data_smithi047_474001_object88:head# 2024-09-18T16:03:29.534 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:29.584 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cd77e455:::benchmark_data_smithi047_474001_object404:head# 2024-09-18T16:03:29.584 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:29.634 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cdedbf36:::benchmark_data_smithi047_474001_object155:head# 2024-09-18T16:03:29.634 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:29.685 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d007de6f:::benchmark_data_smithi047_474001_object151:head# 2024-09-18T16:03:29.685 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:29.735 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d275e6f0:::benchmark_data_smithi047_474001_object419:head# 2024-09-18T16:03:29.735 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:29.785 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d295be0f:::benchmark_data_smithi047_474001_object167:head# 2024-09-18T16:03:29.785 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:29.827 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d2992f10:::benchmark_data_smithi047_474001_object175:head# 2024-09-18T16:03:29.827 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:29.869 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d2cdad48:::benchmark_data_smithi047_474001_object90:head# 2024-09-18T16:03:29.869 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:29.911 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d43afad7:::benchmark_data_smithi047_474001_object376:head# 2024-09-18T16:03:29.911 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:29.953 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d6762583:::benchmark_data_smithi047_474001_object15:head# 2024-09-18T16:03:29.953 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:29.994 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d6d0e63b:::benchmark_data_smithi047_474001_object117:head# 2024-09-18T16:03:29.995 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:30.098 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d74454f2:::benchmark_data_smithi047_474001_object191:head# 2024-09-18T16:03:30.098 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:30.187 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d7ba857c:::benchmark_data_smithi047_474001_object160:head# 2024-09-18T16:03:30.187 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:30.296 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d7ca34da:::benchmark_data_smithi047_474001_object40:head# 2024-09-18T16:03:30.297 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:30.397 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d8141741:::benchmark_data_smithi047_474001_object287:head# 2024-09-18T16:03:30.397 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:30.447 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:da6a8dc2:::benchmark_data_smithi047_474001_object18:head# 2024-09-18T16:03:30.448 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:30.506 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:da6c2c4d:::benchmark_data_smithi047_474001_object31:head# 2024-09-18T16:03:30.506 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:30.608 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:db2f6939:::benchmark_data_smithi047_474001_object400:head# 2024-09-18T16:03:30.608 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:30.690 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:dc4893ce:::benchmark_data_smithi047_474001_object232:head# 2024-09-18T16:03:30.690 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:30.732 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:dc6daefe:::benchmark_data_smithi047_474001_object276:head# 2024-09-18T16:03:30.732 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:30.774 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:dcdad940:::benchmark_data_smithi047_474001_object182:head# 2024-09-18T16:03:30.774 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:30.824 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:dd4ec68f:::benchmark_data_smithi047_474001_object56:head# 2024-09-18T16:03:30.825 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:30.866 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ddc4bb3d:::benchmark_data_smithi047_474001_object199:head# 2024-09-18T16:03:30.867 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:30.908 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ddece86f:::benchmark_data_smithi047_474001_object43:head# 2024-09-18T16:03:30.909 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:30.950 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:df65649c:::benchmark_data_smithi047_474001_object176:head# 2024-09-18T16:03:30.951 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:30.992 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:dfa0b551:::benchmark_data_smithi047_474001_object421:head# 2024-09-18T16:03:30.992 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:31.034 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e07f15b6:::benchmark_data_smithi047_474001_object10:head# 2024-09-18T16:03:31.034 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:31.084 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e1757a69:::benchmark_data_smithi047_474001_object366:head# 2024-09-18T16:03:31.085 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:31.135 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e1859c12:::benchmark_data_smithi047_474001_object102:head# 2024-09-18T16:03:31.135 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:31.185 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e1d3a554:::benchmark_data_smithi047_474001_object369:head# 2024-09-18T16:03:31.185 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:31.235 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e437b5d2:::benchmark_data_smithi047_474001_object377:head# 2024-09-18T16:03:31.235 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:31.285 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e445afb5:::benchmark_data_smithi047_474001_object355:head# 2024-09-18T16:03:31.285 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:31.335 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e61a1975:::benchmark_data_smithi047_474001_object296:head# 2024-09-18T16:03:31.336 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:31.394 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e65db1db:::benchmark_data_smithi047_474001_object329:head# 2024-09-18T16:03:31.394 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:31.436 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e65e637a:::benchmark_data_smithi047_474001_object424:head# 2024-09-18T16:03:31.436 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:31.478 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e6b693e8:::benchmark_data_smithi047_474001_object308:head# 2024-09-18T16:03:31.478 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:31.528 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e6d85c1d:::benchmark_data_smithi047_474001_object220:head# 2024-09-18T16:03:31.528 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:31.570 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e6f3dc67:::benchmark_data_smithi047_474001_object293:head# 2024-09-18T16:03:31.570 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:31.612 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e6f460e2:::benchmark_data_smithi047_474001_object130:head# 2024-09-18T16:03:31.612 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:31.662 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e6f5814f:::benchmark_data_smithi047_474001_object64:head# 2024-09-18T16:03:31.662 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:31.712 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e7a2c4e1:::benchmark_data_smithi047_474001_object94:head# 2024-09-18T16:03:31.713 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:31.763 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e945e46a:::benchmark_data_smithi047_474001_object171:head# 2024-09-18T16:03:31.763 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:31.813 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e9b3c7f8:::benchmark_data_smithi047_474001_object181:head# 2024-09-18T16:03:31.813 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:31.863 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ea0dab4b:::benchmark_data_smithi047_474001_object406:head# 2024-09-18T16:03:31.863 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:31.913 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ea0dd8c2:::benchmark_data_smithi047_474001_object381:head# 2024-09-18T16:03:31.913 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:31.972 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ea20ab79:::benchmark_data_smithi047_474001_object178:head# 2024-09-18T16:03:31.973 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:32.022 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ea95d8c2:::benchmark_data_smithi047_474001_object238:head# 2024-09-18T16:03:32.023 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:32.073 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ecaeb389:::benchmark_data_smithi047_474001_object135:head# 2024-09-18T16:03:32.073 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:32.123 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ed907482:::benchmark_data_smithi047_474001_object34:head# 2024-09-18T16:03:32.123 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:32.207 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ee898c6a:::benchmark_data_smithi047_474001_object184:head# 2024-09-18T16:03:32.207 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:32.257 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:eeceac30:::benchmark_data_smithi047_474001_object97:head# 2024-09-18T16:03:32.257 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:32.307 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:eee17d95:::benchmark_data_smithi047_474001_object187:head# 2024-09-18T16:03:32.307 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:32.358 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:eefc7324:::benchmark_data_smithi047_474001_object91:head# 2024-09-18T16:03:32.358 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:32.428 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ef61b0b9:::benchmark_data_smithi047_474001_object428:head# 2024-09-18T16:03:32.428 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:32.475 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:efbee6f8:::benchmark_data_smithi047_474001_object211:head# 2024-09-18T16:03:32.475 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:32.550 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:eff64798:::benchmark_data_smithi047_474001_object93:head# 2024-09-18T16:03:32.550 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:32.642 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f09ab14b:::benchmark_data_smithi047_474001_object339:head# 2024-09-18T16:03:32.642 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:32.701 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f09c2c79:::benchmark_data_smithi047_474001_object258:head# 2024-09-18T16:03:32.701 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:32.743 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f0a2b2e0:::benchmark_data_smithi047_474001_object234:head# 2024-09-18T16:03:32.743 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:32.793 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f0e21778:::benchmark_data_smithi047_474001_object210:head# 2024-09-18T16:03:32.793 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:32.843 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f12591b2:::benchmark_data_smithi047_474001_object142:head# 2024-09-18T16:03:32.844 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:32.894 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f15f0e45:::benchmark_data_smithi047_474001_object353:head# 2024-09-18T16:03:32.894 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:32.944 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f17effda:::benchmark_data_smithi047_474001_object121:head# 2024-09-18T16:03:32.944 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:32.994 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f188e48b:::benchmark_data_smithi047_474001_object82:head# 2024-09-18T16:03:32.994 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:33.036 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f286bf95:::benchmark_data_smithi047_474001_object115:head# 2024-09-18T16:03:33.036 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:33.078 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f400c3e0:::benchmark_data_smithi047_474001_object305:head# 2024-09-18T16:03:33.078 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:33.120 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f40d5282:::benchmark_data_smithi047_474001_object48:head# 2024-09-18T16:03:33.120 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:33.170 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f4d9af26:::benchmark_data_smithi047_474001_object311:head# 2024-09-18T16:03:33.170 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:33.212 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f50aa0e5:::benchmark_data_smithi047_474001_object65:head# 2024-09-18T16:03:33.212 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:33.254 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f58a2ed4:::benchmark_data_smithi047_474001_object393:head# 2024-09-18T16:03:33.254 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:33.304 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f61edbb2:::benchmark_data_smithi047_474001_object140:head# 2024-09-18T16:03:33.305 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:33.354 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f6596924:::benchmark_data_smithi047_474001_object256:head# 2024-09-18T16:03:33.355 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:33.405 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f68bfbc5:::benchmark_data_smithi047_474001_object295:head# 2024-09-18T16:03:33.405 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:33.455 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f7031fa9:::benchmark_data_smithi047_474001_object318:head# 2024-09-18T16:03:33.455 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:33.505 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f8576353:::benchmark_data_smithi047_474001_object336:head# 2024-09-18T16:03:33.505 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:33.557 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f8e14a8e:::benchmark_data_smithi047_474001_object63:head# 2024-09-18T16:03:33.558 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:33.608 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f9e01ed3:::benchmark_data_smithi047_474001_object20:head# 2024-09-18T16:03:33.608 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:33.658 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fa676313:::benchmark_data_smithi047_474001_object268:head# 2024-09-18T16:03:33.658 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:33.717 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fb2704e8:::benchmark_data_smithi047_474001_object294:head# 2024-09-18T16:03:33.717 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:33.759 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fb557b5c:::benchmark_data_smithi047_474001_object25:head# 2024-09-18T16:03:33.759 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:33.817 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fba861de:::benchmark_data_smithi047_474001_object233:head# 2024-09-18T16:03:33.817 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:33.867 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fc48c5f5:::benchmark_data_smithi047_474001_object128:head# 2024-09-18T16:03:33.867 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:33.909 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fcc00bd9:::benchmark_data_smithi047_474001_object267:head# 2024-09-18T16:03:33.909 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:33.951 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fcc267e2:::benchmark_data_smithi047_474001_object420:head# 2024-09-18T16:03:33.951 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:33.993 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fcd59592:::benchmark_data_smithi047_474001_object387:head# 2024-09-18T16:03:33.993 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:34.035 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fcff1581:::benchmark_data_smithi047_474001_object229:head# 2024-09-18T16:03:34.035 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:34.077 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fd5c7c49:::benchmark_data_smithi047_474001_object375:head# 2024-09-18T16:03:34.077 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:34.136 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fdd0ff05:::benchmark_data_smithi047_474001_object67:head# 2024-09-18T16:03:34.136 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:34.228 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fdf729e9:::benchmark_data_smithi047_474001_object230:head# 2024-09-18T16:03:34.228 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:34.278 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fe6c9d0e:::benchmark_data_smithi047_474001_object52:head# 2024-09-18T16:03:34.278 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:34.320 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fe938a07:::benchmark_data_smithi047_474001_object207:head# 2024-09-18T16:03:34.320 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:34.362 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fea2446c:::benchmark_data_smithi047_474001_object105:head# 2024-09-18T16:03:34.362 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:34.412 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fee86c8a:::benchmark_data_smithi047_474001_object412:head# 2024-09-18T16:03:34.412 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:34.454 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ff03a3e6:::benchmark_data_smithi047_474001_object37:head# 2024-09-18T16:03:34.454 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:34.503 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ff8f6396:::benchmark_data_smithi047_474001_object114:head# 2024-09-18T16:03:34.503 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:34.599 INFO:tasks.workunit.client.0.smithi047.stdout:write_pg epoch 9 info 1.1( v 9'229 (0'0,9'229] local-lis/les=7/8 n=229 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-09-18T16:03:34.680 INFO:tasks.workunit.client.0.smithi047.stdout:Import successful 2024-09-18T16:03:35.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:148: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0 --force 2024-09-18T16:03:36.851 INFO:tasks.workunit.client.0.smithi047.stdout:get_pg_num_history pg_num_history pg_num_history(e228 pg_nums {1={19=1}} deleted_pools ) 2024-09-18T16:03:36.851 INFO:tasks.workunit.client.0.smithi047.stdout:Importing pgid 1.0 2024-09-18T16:03:36.851 INFO:tasks.workunit.client.0.smithi047.stderr:PG export's map 9 is older than OSD's oldest_map 121 2024-09-18T16:03:36.852 INFO:tasks.workunit.client.0.smithi047.stdout:pool 1 pg_num 2 -> 1 2024-09-18T16:03:36.852 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0118b086:::benchmark_data_smithi047_474001_object299:head# 2024-09-18T16:03:36.852 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:36.960 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:03a83bbd:::benchmark_data_smithi047_474001_object340:head# 2024-09-18T16:03:36.960 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:37.051 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:03ef44ea:::benchmark_data_smithi047_474001_object136:head# 2024-09-18T16:03:37.052 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:37.110 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:04ce55e6:::benchmark_data_smithi047_474001_object104:head# 2024-09-18T16:03:37.110 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:37.169 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:04e7bb72:::benchmark_data_smithi047_474001_object259:head# 2024-09-18T16:03:37.169 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:37.219 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0522b134:::benchmark_data_smithi047_474001_object390:head# 2024-09-18T16:03:37.219 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:37.278 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:065742a4:::benchmark_data_smithi047_474001_object240:head# 2024-09-18T16:03:37.278 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:37.345 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0667ed83:::benchmark_data_smithi047_474001_object29:head# 2024-09-18T16:03:37.345 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:37.395 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:067cfdc0:::benchmark_data_smithi047_474001_object373:head# 2024-09-18T16:03:37.395 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:37.445 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:06873111:::benchmark_data_smithi047_474001_object270:head# 2024-09-18T16:03:37.445 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:37.495 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:06fbd7c2:::benchmark_data_smithi047_474001_object127:head# 2024-09-18T16:03:37.495 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:37.545 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:07ded5f1:::benchmark_data_smithi047_474001_object266:head# 2024-09-18T16:03:37.546 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:37.596 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0846b0e1:::benchmark_data_smithi047_474001_object255:head# 2024-09-18T16:03:37.596 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:37.654 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0a425810:::benchmark_data_smithi047_474001_object410:head# 2024-09-18T16:03:37.654 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:37.696 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0b99baba:::benchmark_data_smithi047_474001_object215:head# 2024-09-18T16:03:37.696 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:37.738 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0cfa8c38:::benchmark_data_smithi047_474001_object163:head# 2024-09-18T16:03:37.738 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:37.788 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0e30d732:::benchmark_data_smithi047_474001_object402:head# 2024-09-18T16:03:37.788 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:37.830 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0e508c23:::benchmark_data_smithi047_474001_object196:head# 2024-09-18T16:03:37.830 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:37.872 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0e642663:::benchmark_data_smithi047_474001_object80:head# 2024-09-18T16:03:37.872 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:37.914 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0e743ace:::benchmark_data_smithi047_474001_object21:head# 2024-09-18T16:03:37.914 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:37.964 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0eaf1576:::benchmark_data_smithi047_474001_object172:head# 2024-09-18T16:03:37.964 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:38.014 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0fcd5078:::benchmark_data_smithi047_474001_object364:head# 2024-09-18T16:03:38.015 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:38.065 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0fe45a41:::benchmark_data_smithi047_474001_object333:head# 2024-09-18T16:03:38.065 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:38.127 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:104ccd46:::benchmark_data_smithi047_474001_object365:head# 2024-09-18T16:03:38.127 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:38.224 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:10c24a87:::benchmark_data_smithi047_474001_object141:head# 2024-09-18T16:03:38.224 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:38.283 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:11014d3d:::benchmark_data_smithi047_474001_object281:head# 2024-09-18T16:03:38.283 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:38.341 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1161e93e:::benchmark_data_smithi047_474001_object278:head# 2024-09-18T16:03:38.341 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:38.391 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:129bbd1b:::benchmark_data_smithi047_474001_object389:head# 2024-09-18T16:03:38.391 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:38.425 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:131e86da:::benchmark_data_smithi047_474001_object203:head# 2024-09-18T16:03:38.425 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:38.458 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1444d7df:::benchmark_data_smithi047_474001_object271:head# 2024-09-18T16:03:38.458 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:38.502 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:14ce8c64:::benchmark_data_smithi047_474001_object246:head# 2024-09-18T16:03:38.502 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:38.559 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:15241e69:::benchmark_data_smithi047_474001_object148:head# 2024-09-18T16:03:38.559 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:38.617 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1593a397:::benchmark_data_smithi047_474001_object347:head# 2024-09-18T16:03:38.618 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:38.684 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:17f3626e:::benchmark_data_smithi047_474001_object42:head# 2024-09-18T16:03:38.685 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:38.718 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:180c940a:::benchmark_data_smithi047_474001_object372:head# 2024-09-18T16:03:38.718 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:38.751 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:183bb03e:::benchmark_data_smithi047_474001_object323:head# 2024-09-18T16:03:38.751 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:38.785 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:18747883:::benchmark_data_smithi047_474001_object192:head# 2024-09-18T16:03:38.785 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:38.818 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1a19ff87:::benchmark_data_smithi047_474001_object2:head# 2024-09-18T16:03:38.818 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:38.852 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1a2c0aea:::benchmark_data_smithi047_474001_object274:head# 2024-09-18T16:03:38.852 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:38.885 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1bd588e8:::benchmark_data_smithi047_474001_object320:head# 2024-09-18T16:03:38.885 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:38.918 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1cc6e811:::benchmark_data_smithi047_474001_object149:head# 2024-09-18T16:03:38.919 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:38.952 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1ce6bbbf:::benchmark_data_smithi047_474001_object164:head# 2024-09-18T16:03:38.952 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:38.985 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1d83d665:::benchmark_data_smithi047_474001_object153:head# 2024-09-18T16:03:38.986 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:39.019 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1d849551:::benchmark_data_smithi047_474001_object395:head# 2024-09-18T16:03:39.019 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:39.052 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1df00925:::benchmark_data_smithi047_474001_object86:head# 2024-09-18T16:03:39.053 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:39.086 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1ea58aa2:::benchmark_data_smithi047_474001_object265:head# 2024-09-18T16:03:39.086 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:39.119 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:20be7217:::benchmark_data_smithi047_474001_object145:head# 2024-09-18T16:03:39.119 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:39.153 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:21c5c01f:::benchmark_data_smithi047_474001_object69:head# 2024-09-18T16:03:39.153 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:39.186 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:221f844d:::benchmark_data_smithi047_474001_object350:head# 2024-09-18T16:03:39.186 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:39.220 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:22d3be20:::benchmark_data_smithi047_474001_object315:head# 2024-09-18T16:03:39.220 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:39.253 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2334c1fe:::benchmark_data_smithi047_474001_object84:head# 2024-09-18T16:03:39.253 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:39.286 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:24393629:::benchmark_data_smithi047_474001_object239:head# 2024-09-18T16:03:39.287 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:39.320 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:25482445:::benchmark_data_smithi047_474001_object53:head# 2024-09-18T16:03:39.320 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:39.353 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2555ae90:::benchmark_data_smithi047_474001_object370:head# 2024-09-18T16:03:39.354 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:39.387 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:26a1c85e:::benchmark_data_smithi047_474001_object32:head# 2024-09-18T16:03:39.387 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:39.420 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:26b7b87f:::benchmark_data_smithi047_474001_object107:head# 2024-09-18T16:03:39.421 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:39.454 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:27190702:::benchmark_data_smithi047_474001_object218:head# 2024-09-18T16:03:39.454 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:39.487 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:27f9a01e:::benchmark_data_smithi047_474001_object349:head# 2024-09-18T16:03:39.487 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:39.521 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:284c9f3a:::benchmark_data_smithi047_474001_object254:head# 2024-09-18T16:03:39.521 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:39.563 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2856b42f:::benchmark_data_smithi047_474001_object81:head# 2024-09-18T16:03:39.563 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:39.604 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:29494f84:::benchmark_data_smithi047_474001_object248:head# 2024-09-18T16:03:39.605 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:39.646 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2a2243e8:::benchmark_data_smithi047_474001_object1:head# 2024-09-18T16:03:39.646 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:39.688 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2abea481:::benchmark_data_smithi047_474001_object23:head# 2024-09-18T16:03:39.688 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:39.729 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2ced8b80:::benchmark_data_smithi047_474001_object401:head# 2024-09-18T16:03:39.730 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:39.772 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2e2d9e2b:::benchmark_data_smithi047_474001_object367:head# 2024-09-18T16:03:39.772 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:39.813 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2e9ea224:::benchmark_data_smithi047_474001_object106:head# 2024-09-18T16:03:39.813 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:39.847 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2fb8dbb3:::benchmark_data_smithi047_474001_object170:head# 2024-09-18T16:03:39.847 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:39.880 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:300fdc07:::benchmark_data_smithi047_474001_object227:head# 2024-09-18T16:03:39.880 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:39.914 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3078741e:::benchmark_data_smithi047_474001_object139:head# 2024-09-18T16:03:39.914 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:39.947 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:31669bfe:::benchmark_data_smithi047_474001_object319:head# 2024-09-18T16:03:39.947 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:39.980 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:321ae169:::benchmark_data_smithi047_474001_object244:head# 2024-09-18T16:03:39.980 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:40.014 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:32ff3ac8:::benchmark_data_smithi047_474001_object261:head# 2024-09-18T16:03:40.014 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:40.048 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:33134995:::benchmark_data_smithi047_474001_object310:head# 2024-09-18T16:03:40.048 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:40.081 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:33b1fdf8:::benchmark_data_smithi047_474001_object161:head# 2024-09-18T16:03:40.081 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:40.106 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:33b93a31:::benchmark_data_smithi047_474001_object403:head# 2024-09-18T16:03:40.106 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:40.156 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:33fa48ba:::benchmark_data_smithi047_474001_object289:head# 2024-09-18T16:03:40.157 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:40.198 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:34348ae4:::benchmark_data_smithi047_474001_object6:head# 2024-09-18T16:03:40.198 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:40.223 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:34e0f433:::benchmark_data_smithi047_474001_object342:head# 2024-09-18T16:03:40.224 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:40.249 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:355eddbc:::benchmark_data_smithi047_474001_object198:head# 2024-09-18T16:03:40.249 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:40.282 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:35841d04:::benchmark_data_smithi047_474001_object186:head# 2024-09-18T16:03:40.282 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:40.316 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:35c32413:::benchmark_data_smithi047_474001_object413:head# 2024-09-18T16:03:40.316 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:40.349 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:35e67788:::benchmark_data_smithi047_474001_object388:head# 2024-09-18T16:03:40.349 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:40.382 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3613c729:::benchmark_data_smithi047_474001_object380:head# 2024-09-18T16:03:40.383 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:40.416 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:37ccfdd9:::benchmark_data_smithi047_474001_object219:head# 2024-09-18T16:03:40.416 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:40.449 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:380af478:::benchmark_data_smithi047_474001_object359:head# 2024-09-18T16:03:40.449 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:40.483 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3812ffca:::benchmark_data_smithi047_474001_object286:head# 2024-09-18T16:03:40.483 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:40.550 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:381ae33c:::benchmark_data_smithi047_474001_object152:head# 2024-09-18T16:03:40.550 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:40.600 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3994ed2b:::benchmark_data_smithi047_474001_object162:head# 2024-09-18T16:03:40.600 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:40.651 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3ad1e94f:::benchmark_data_smithi047_474001_object26:head# 2024-09-18T16:03:40.651 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:40.684 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3b2460a7:::benchmark_data_smithi047_474001_object418:head# 2024-09-18T16:03:40.684 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:40.718 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3b70d910:::benchmark_data_smithi047_474001_object157:head# 2024-09-18T16:03:40.718 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:40.751 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3b976a95:::benchmark_data_smithi047_474001_object392:head# 2024-09-18T16:03:40.751 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:40.785 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3c01f944:::benchmark_data_smithi047_474001_object85:head# 2024-09-18T16:03:40.785 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:40.818 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3c08c862:::benchmark_data_smithi047_474001_object209:head# 2024-09-18T16:03:40.818 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:40.851 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3c3b4221:::benchmark_data_smithi047_474001_object302:head# 2024-09-18T16:03:40.852 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:40.885 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3e2bc2d1:::benchmark_data_smithi047_474001_object290:head# 2024-09-18T16:03:40.885 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:40.918 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3f0792f2:::benchmark_data_smithi047_474001_object205:head# 2024-09-18T16:03:40.918 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:40.952 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3f0bbc7e:::benchmark_data_smithi047_474001_object214:head# 2024-09-18T16:03:40.952 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:40.985 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:40e84920:::benchmark_data_smithi047_474001_object201:head# 2024-09-18T16:03:40.985 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:41.019 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:410710e5:::benchmark_data_smithi047_474001_object360:head# 2024-09-18T16:03:41.019 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:41.052 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4286a502:::benchmark_data_smithi047_474001_object280:head# 2024-09-18T16:03:41.052 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:41.086 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:429b03e9:::benchmark_data_smithi047_474001_object249:head# 2024-09-18T16:03:41.086 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:41.119 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:43c8723a:::benchmark_data_smithi047_474001_object4:head# 2024-09-18T16:03:41.119 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:41.152 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:43c8e265:::benchmark_data_smithi047_474001_object208:head# 2024-09-18T16:03:41.152 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:41.186 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:43d378b4:::benchmark_data_smithi047_474001_object425:head# 2024-09-18T16:03:41.186 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:41.219 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:43fb4408:::benchmark_data_smithi047_474001_object399:head# 2024-09-18T16:03:41.220 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:41.253 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:44ab53e5:::benchmark_data_smithi047_474001_object27:head# 2024-09-18T16:03:41.253 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:41.286 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4515f62e:::benchmark_data_smithi047_474001_object143:head# 2024-09-18T16:03:41.286 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:41.320 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:457b93bc:::benchmark_data_smithi047_474001_object300:head# 2024-09-18T16:03:41.320 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:41.353 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:457e7435:::benchmark_data_smithi047_474001_object231:head# 2024-09-18T16:03:41.353 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:41.387 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:468b56fb:::benchmark_data_smithi047_474001_object242:head# 2024-09-18T16:03:41.387 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:41.437 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:47454808:::benchmark_data_smithi047_474001_object394:head# 2024-09-18T16:03:41.437 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:41.472 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4757d830:::benchmark_data_smithi047_474001_object147:head# 2024-09-18T16:03:41.472 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:41.504 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4790fa42:::benchmark_data_smithi047_474001_object16:head# 2024-09-18T16:03:41.504 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:41.537 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:481edef6:::benchmark_data_smithi047_474001_object129:head# 2024-09-18T16:03:41.537 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:41.571 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:498127df:::benchmark_data_smithi047_474001_object409:head# 2024-09-18T16:03:41.571 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:41.604 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4a94b2bf:::benchmark_data_smithi047_474001_object331:head# 2024-09-18T16:03:41.604 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:41.671 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4b0f17cd:::benchmark_data_smithi047_474001_object194:head# 2024-09-18T16:03:41.671 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:41.704 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4ccceada:::benchmark_data_smithi047_474001_object213:head# 2024-09-18T16:03:41.704 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:41.738 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4d145d82:::benchmark_data_smithi047_474001_object206:head# 2024-09-18T16:03:41.738 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:41.771 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4e67dcba:::benchmark_data_smithi047_474001_object144:head# 2024-09-18T16:03:41.771 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:41.805 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4e6bc7a4:::benchmark_data_smithi047_474001_object165:head# 2024-09-18T16:03:41.805 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:41.838 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4e7fe7b2:::benchmark_data_smithi047_474001_object431:head# 2024-09-18T16:03:41.838 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:41.872 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4f5e76df:::benchmark_data_smithi047_474001_object415:head# 2024-09-18T16:03:41.872 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:41.905 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5030ebeb:::benchmark_data_smithi047_474001_object335:head# 2024-09-18T16:03:41.905 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:41.938 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:505987dd:::benchmark_data_smithi047_474001_object313:head# 2024-09-18T16:03:41.938 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:41.972 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:50a1999f:::benchmark_data_smithi047_474001_object190:head# 2024-09-18T16:03:41.972 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:42.005 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:51905d78:::benchmark_data_smithi047_474001_object371:head# 2024-09-18T16:03:42.006 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:42.047 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:53516fa7:::benchmark_data_smithi047_474001_object47:head# 2024-09-18T16:03:42.047 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:42.097 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:53a75224:::benchmark_data_smithi047_474001_object277:head# 2024-09-18T16:03:42.097 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:42.143 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:53bc5374:::benchmark_data_smithi047_474001_object304:head# 2024-09-18T16:03:42.144 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:42.189 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:54037397:::benchmark_data_smithi047_474001_object168:head# 2024-09-18T16:03:42.190 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:42.231 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:54e93f24:::benchmark_data_smithi047_474001_object351:head# 2024-09-18T16:03:42.231 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:42.265 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5549c508:::benchmark_data_smithi047_474001_object154:head# 2024-09-18T16:03:42.265 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:42.306 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:55a8f195:::benchmark_data_smithi047_474001_object183:head# 2024-09-18T16:03:42.307 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:42.340 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:58bb18eb:::benchmark_data_smithi047_474001_object92:head# 2024-09-18T16:03:42.340 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:42.373 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:58e3824a:::benchmark_data_smithi047_474001_object22:head# 2024-09-18T16:03:42.374 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:42.407 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:58eacbbf:::benchmark_data_smithi047_474001_object100:head# 2024-09-18T16:03:42.407 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:42.440 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:590d7cd0:::benchmark_data_smithi047_474001_object328:head# 2024-09-18T16:03:42.440 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:42.474 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:59701efb:::benchmark_data_smithi047_474001_object243:head# 2024-09-18T16:03:42.474 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:42.533 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:59852438:::benchmark_data_smithi047_474001_object83:head# 2024-09-18T16:03:42.533 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:42.579 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5a123740:::benchmark_data_smithi047_474001_object36:head# 2024-09-18T16:03:42.579 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:42.675 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5b73d6c3:::benchmark_data_smithi047_474001_object284:head# 2024-09-18T16:03:42.675 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:42.709 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5c8e3731:::benchmark_data_smithi047_474001_object132:head# 2024-09-18T16:03:42.709 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:42.742 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5caa7c9f:::benchmark_data_smithi047_474001_object416:head# 2024-09-18T16:03:42.742 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:42.775 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5cf5ea91:::benchmark_data_smithi047_474001_object397:head# 2024-09-18T16:03:42.775 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:42.826 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5d1b58ad:::benchmark_data_smithi047_474001_object269:head# 2024-09-18T16:03:42.826 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:42.859 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5e669bb0:::benchmark_data_smithi047_474001_object405:head# 2024-09-18T16:03:42.859 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:42.892 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5eac6d5a:::benchmark_data_smithi047_474001_object411:head# 2024-09-18T16:03:42.893 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:42.926 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5ef5c1d9:::benchmark_data_smithi047_474001_object385:head# 2024-09-18T16:03:42.926 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:42.959 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5f7398fd:::benchmark_data_smithi047_474001_object68:head# 2024-09-18T16:03:42.959 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:42.993 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5f8ba50b:::benchmark_data_smithi047_474001_object174:head# 2024-09-18T16:03:42.993 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:43.026 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5fb38f21:::benchmark_data_smithi047_474001_object75:head# 2024-09-18T16:03:43.026 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:43.060 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:60a5ff1a:::benchmark_data_smithi047_474001_object298:head# 2024-09-18T16:03:43.060 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:43.093 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6159ad07:::benchmark_data_smithi047_474001_object221:head# 2024-09-18T16:03:43.093 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:43.127 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:62273e63:::benchmark_data_smithi047_474001_object62:head# 2024-09-18T16:03:43.127 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:43.160 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:63cf18a2:::benchmark_data_smithi047_474001_object272:head# 2024-09-18T16:03:43.160 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:43.193 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:63f415e6:::benchmark_data_smithi047_474001_object58:head# 2024-09-18T16:03:43.194 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:43.227 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:64ff073d:::benchmark_data_smithi047_474001_object362:head# 2024-09-18T16:03:43.227 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:43.260 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:657bccc4:::benchmark_data_smithi047_474001_object169:head# 2024-09-18T16:03:43.261 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:43.294 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:660836d4:::benchmark_data_smithi047_474001_object39:head# 2024-09-18T16:03:43.294 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:43.327 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:66682ebd:::benchmark_data_smithi047_474001_object101:head# 2024-09-18T16:03:43.327 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:43.361 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:66e730e2:::benchmark_data_smithi047_474001_object61:head# 2024-09-18T16:03:43.361 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:43.394 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:66fbc3ec:::benchmark_data_smithi047_474001_object71:head# 2024-09-18T16:03:43.395 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:43.428 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6717adea:::benchmark_data_smithi047_474001_object317:head# 2024-09-18T16:03:43.428 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:43.461 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:68107ec1:::benchmark_data_smithi047_474001_object195:head# 2024-09-18T16:03:43.462 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:43.495 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6812f8cb:::benchmark_data_smithi047_474001_object264:head# 2024-09-18T16:03:43.495 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:43.528 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:68f519f4:::benchmark_data_smithi047_474001_object343:head# 2024-09-18T16:03:43.528 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:43.562 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6a39a8d4:::benchmark_data_smithi047_474001_object66:head# 2024-09-18T16:03:43.562 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:43.595 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6ab23d26:::benchmark_data_smithi047_474001_object9:head# 2024-09-18T16:03:43.595 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:43.628 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6b25d55a:::benchmark_data_smithi047_474001_object72:head# 2024-09-18T16:03:43.629 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:43.662 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6b5fc221:::benchmark_data_smithi047_474001_object334:head# 2024-09-18T16:03:43.662 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:43.695 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6c2d825c:::benchmark_data_smithi047_474001_object5:head# 2024-09-18T16:03:43.696 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:43.729 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6caa7eaf:::benchmark_data_smithi047_474001_object263:head# 2024-09-18T16:03:43.729 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:43.762 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6dfd6f4e:::benchmark_data_smithi047_474001_object251:head# 2024-09-18T16:03:43.762 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:43.796 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6e9248a9:::benchmark_data_smithi047_474001_object245:head# 2024-09-18T16:03:43.796 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:43.829 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6f58cdd1:::benchmark_data_smithi047_474001_object430:head# 2024-09-18T16:03:43.829 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:43.862 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:71084a82:::benchmark_data_smithi047_474001_object398:head# 2024-09-18T16:03:43.863 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:43.896 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:722633c0:::benchmark_data_smithi047_474001_object7:head# 2024-09-18T16:03:43.896 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:43.930 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:73078ec4:::benchmark_data_smithi047_474001_object200:head# 2024-09-18T16:03:43.930 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:43.963 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7319939a:::benchmark_data_smithi047_474001_object378:head# 2024-09-18T16:03:43.963 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:43.997 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:732bbbc2:::benchmark_data_smithi047_474001_object113:head# 2024-09-18T16:03:43.997 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:44.030 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:73694cfd:::benchmark_data_smithi047_474001_object257:head# 2024-09-18T16:03:44.030 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:44.063 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:739a0e00:::benchmark_data_smithi047_474001_object11:head# 2024-09-18T16:03:44.063 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:44.097 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:748a50de:::benchmark_data_smithi047_474001_object223:head# 2024-09-18T16:03:44.097 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:44.130 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:74cbfe52:::benchmark_data_smithi047_474001_object309:head# 2024-09-18T16:03:44.130 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:44.181 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:751b68bd:::benchmark_data_smithi047_474001_object45:head# 2024-09-18T16:03:44.181 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:44.231 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:76583154:::benchmark_data_smithi047_474001_object13:head# 2024-09-18T16:03:44.231 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:44.264 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7669c90e:::benchmark_data_smithi047_474001_object216:head# 2024-09-18T16:03:44.264 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:44.298 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:76dc48ec:::benchmark_data_smithi047_474001_object79:head# 2024-09-18T16:03:44.298 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:44.331 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:77226950:::benchmark_data_smithi047_474001_object386:head# 2024-09-18T16:03:44.331 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:44.365 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:77a33037:::benchmark_data_smithi047_474001_object197:head# 2024-09-18T16:03:44.365 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:44.398 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:77f7974a:::benchmark_data_smithi047_474001_object123:head# 2024-09-18T16:03:44.398 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:44.449 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:78d90d4e:::benchmark_data_smithi047_474001_object301:head# 2024-09-18T16:03:44.449 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:44.501 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7b36e97b:::benchmark_data_smithi047_474001_object292:head# 2024-09-18T16:03:44.502 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:44.557 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7c8233bd:::benchmark_data_smithi047_474001_object76:head# 2024-09-18T16:03:44.558 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:44.608 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7cce9f64:::benchmark_data_smithi047_474001_object0:head# 2024-09-18T16:03:44.608 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:44.658 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7d8e31eb:::benchmark_data_smithi047_474001_object193:head# 2024-09-18T16:03:44.658 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:44.683 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7dc806f2:::benchmark_data_smithi047_474001_object126:head# 2024-09-18T16:03:44.683 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:44.708 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7dfaf25d:::benchmark_data_smithi047_474001_object260:head# 2024-09-18T16:03:44.708 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:44.733 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7e00f5f7:::benchmark_data_smithi047_474001_object98:head# 2024-09-18T16:03:44.733 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:44.759 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7f679bd9:::benchmark_data_smithi047_474001_object379:head# 2024-09-18T16:03:44.759 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:44.784 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7fa61220:::benchmark_data_smithi047_474001_object354:head# 2024-09-18T16:03:44.784 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:44.809 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7fb32424:::benchmark_data_smithi047_474001_object241:head# 2024-09-18T16:03:44.809 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:44.834 INFO:tasks.workunit.client.0.smithi047.stdout:write_pg epoch 9 info 1.0( v 9'204 (0'0,9'204] local-lis/les=7/8 n=204 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-09-18T16:03:44.859 INFO:tasks.workunit.client.0.smithi047.stdout:Import successful 2024-09-18T16:03:45.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:151: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op remove --pgid 1.0 --force 2024-09-18T16:03:47.067 INFO:tasks.workunit.client.0.smithi047.stdout: marking collection for removal 2024-09-18T16:03:47.067 INFO:tasks.workunit.client.0.smithi047.stdout:setting '_remove' omap key 2024-09-18T16:03:47.105 INFO:tasks.workunit.client.0.smithi047.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-09-18T16:03:47.465 INFO:tasks.workunit.client.0.smithi047.stdout:Remove successful 2024-09-18T16:03:47.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:152: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op remove --pgid 1.1 --force 2024-09-18T16:03:49.343 INFO:tasks.workunit.client.0.smithi047.stdout: marking collection for removal 2024-09-18T16:03:49.344 INFO:tasks.workunit.client.0.smithi047.stdout:setting '_remove' omap key 2024-09-18T16:03:49.360 INFO:tasks.workunit.client.0.smithi047.stdout:finish_remove_pgs 1.1_head removing 1.1 2024-09-18T16:03:49.946 INFO:tasks.workunit.client.0.smithi047.stdout:Remove successful 2024-09-18T16:03:50.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:154: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0.later --force 2024-09-18T16:03:51.943 INFO:tasks.workunit.client.0.smithi047.stdout:get_pg_num_history pg_num_history pg_num_history(e228 pg_nums {1={19=1}} deleted_pools ) 2024-09-18T16:03:51.943 INFO:tasks.workunit.client.0.smithi047.stdout:Importing pgid 1.0 2024-09-18T16:03:51.943 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:008f3a82:::benchmark_data_smithi047_479866_object170:head# 2024-09-18T16:03:51.944 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:52.060 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0118b086:::benchmark_data_smithi047_474001_object299:head# 2024-09-18T16:03:52.060 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:52.110 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:02a07fce:::benchmark_data_smithi047_479866_object166:head# 2024-09-18T16:03:52.110 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:52.169 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:03a83bbd:::benchmark_data_smithi047_474001_object340:head# 2024-09-18T16:03:52.169 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:52.211 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:03ef44ea:::benchmark_data_smithi047_474001_object136:head# 2024-09-18T16:03:52.211 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:52.244 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:04ce55e6:::benchmark_data_smithi047_474001_object104:head# 2024-09-18T16:03:52.245 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:52.278 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:04e7bb72:::benchmark_data_smithi047_474001_object259:head# 2024-09-18T16:03:52.278 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:52.311 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0522b134:::benchmark_data_smithi047_474001_object390:head# 2024-09-18T16:03:52.312 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:52.345 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0538e625:::benchmark_data_smithi047_479866_object35:head# 2024-09-18T16:03:52.345 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:52.378 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0621c7be:::benchmark_data_smithi047_479866_object70:head# 2024-09-18T16:03:52.378 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:52.411 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:065742a4:::benchmark_data_smithi047_474001_object240:head# 2024-09-18T16:03:52.412 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:52.445 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0667ed83:::benchmark_data_smithi047_474001_object29:head# 2024-09-18T16:03:52.445 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:52.478 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:067cfdc0:::benchmark_data_smithi047_474001_object373:head# 2024-09-18T16:03:52.479 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:52.545 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:06873111:::benchmark_data_smithi047_474001_object270:head# 2024-09-18T16:03:52.546 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:52.604 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:06fbd7c2:::benchmark_data_smithi047_474001_object127:head# 2024-09-18T16:03:52.605 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:52.671 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:07ded5f1:::benchmark_data_smithi047_474001_object266:head# 2024-09-18T16:03:52.671 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:52.705 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0846b0e1:::benchmark_data_smithi047_474001_object255:head# 2024-09-18T16:03:52.705 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:52.738 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:084dae8d:::benchmark_data_smithi047_479866_object46:head# 2024-09-18T16:03:52.738 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:52.771 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:088043d9:::benchmark_data_smithi047_479866_object133:head# 2024-09-18T16:03:52.771 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:52.805 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0a425810:::benchmark_data_smithi047_474001_object410:head# 2024-09-18T16:03:52.805 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:52.838 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0b99baba:::benchmark_data_smithi047_474001_object215:head# 2024-09-18T16:03:52.839 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:52.872 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0cbc1619:::benchmark_data_smithi047_479866_object92:head# 2024-09-18T16:03:52.872 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:52.905 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0cfa8c38:::benchmark_data_smithi047_474001_object163:head# 2024-09-18T16:03:52.905 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:52.939 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0cffcd84:::benchmark_data_smithi047_479866_object21:head# 2024-09-18T16:03:52.939 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:52.972 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0e30d732:::benchmark_data_smithi047_474001_object402:head# 2024-09-18T16:03:52.972 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:53.006 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0e508c23:::benchmark_data_smithi047_474001_object196:head# 2024-09-18T16:03:53.006 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:53.039 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0e642663:::benchmark_data_smithi047_474001_object80:head# 2024-09-18T16:03:53.039 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:53.073 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0e743ace:::benchmark_data_smithi047_474001_object21:head# 2024-09-18T16:03:53.073 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:53.106 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0eaf1576:::benchmark_data_smithi047_474001_object172:head# 2024-09-18T16:03:53.106 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:53.140 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0f5d34c4:::benchmark_data_smithi047_479866_object5:head# 2024-09-18T16:03:53.140 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:53.173 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0fcd5078:::benchmark_data_smithi047_474001_object364:head# 2024-09-18T16:03:53.173 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:53.206 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0fe45a41:::benchmark_data_smithi047_474001_object333:head# 2024-09-18T16:03:53.207 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:53.249 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:104ccd46:::benchmark_data_smithi047_474001_object365:head# 2024-09-18T16:03:53.249 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:53.282 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:10c24a87:::benchmark_data_smithi047_474001_object141:head# 2024-09-18T16:03:53.282 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:53.315 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:11014d3d:::benchmark_data_smithi047_474001_object281:head# 2024-09-18T16:03:53.315 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:53.349 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1161e93e:::benchmark_data_smithi047_474001_object278:head# 2024-09-18T16:03:53.349 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:53.382 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:129bbd1b:::benchmark_data_smithi047_474001_object389:head# 2024-09-18T16:03:53.382 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:53.416 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:12b8132b:::benchmark_data_smithi047_479866_object40:head# 2024-09-18T16:03:53.416 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:53.449 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:131e86da:::benchmark_data_smithi047_474001_object203:head# 2024-09-18T16:03:53.449 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:53.483 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1444d7df:::benchmark_data_smithi047_474001_object271:head# 2024-09-18T16:03:53.483 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:53.516 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:14ce8c64:::benchmark_data_smithi047_474001_object246:head# 2024-09-18T16:03:53.516 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:53.549 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:15241e69:::benchmark_data_smithi047_474001_object148:head# 2024-09-18T16:03:53.550 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:53.583 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:153afee9:::benchmark_data_smithi047_479866_object104:head# 2024-09-18T16:03:53.583 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:53.616 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:158af36b:::benchmark_data_smithi047_479866_object30:head# 2024-09-18T16:03:53.617 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:53.650 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1593a397:::benchmark_data_smithi047_474001_object347:head# 2024-09-18T16:03:53.650 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:53.684 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:173ef99d:::benchmark_data_smithi047_479866_object137:head# 2024-09-18T16:03:53.684 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:53.717 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:175d5f0f:::benchmark_data_smithi047_479866_object48:head# 2024-09-18T16:03:53.717 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:53.750 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:17f3626e:::benchmark_data_smithi047_474001_object42:head# 2024-09-18T16:03:53.751 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:53.784 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:180c940a:::benchmark_data_smithi047_474001_object372:head# 2024-09-18T16:03:53.784 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:53.817 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:183bb03e:::benchmark_data_smithi047_474001_object323:head# 2024-09-18T16:03:53.817 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:53.851 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:18417460:::benchmark_data_smithi047_479866_object60:head# 2024-09-18T16:03:53.851 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:53.884 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:18747883:::benchmark_data_smithi047_474001_object192:head# 2024-09-18T16:03:53.884 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:53.926 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:189ef644:::benchmark_data_smithi047_479866_object39:head# 2024-09-18T16:03:53.926 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:53.959 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1a19ff87:::benchmark_data_smithi047_474001_object2:head# 2024-09-18T16:03:53.959 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:53.993 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1a2c0aea:::benchmark_data_smithi047_474001_object274:head# 2024-09-18T16:03:53.993 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:54.026 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1b28af72:::benchmark_data_smithi047_479866_object139:head# 2024-09-18T16:03:54.026 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:54.060 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1bd588e8:::benchmark_data_smithi047_474001_object320:head# 2024-09-18T16:03:54.060 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:54.085 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1c1035dd:::benchmark_data_smithi047_479866_object161:head# 2024-09-18T16:03:54.085 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:54.110 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1c648bcf:::benchmark_data_smithi047_479866_object138:head# 2024-09-18T16:03:54.110 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:54.160 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1cc6e811:::benchmark_data_smithi047_474001_object149:head# 2024-09-18T16:03:54.160 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:54.194 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1ce6bbbf:::benchmark_data_smithi047_474001_object164:head# 2024-09-18T16:03:54.194 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:54.227 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1d83d665:::benchmark_data_smithi047_474001_object153:head# 2024-09-18T16:03:54.227 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:54.261 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1d849551:::benchmark_data_smithi047_474001_object395:head# 2024-09-18T16:03:54.261 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:54.294 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1df00925:::benchmark_data_smithi047_474001_object86:head# 2024-09-18T16:03:54.294 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:54.328 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1e14a4fe:::benchmark_data_smithi047_479866_object115:head# 2024-09-18T16:03:54.328 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:54.370 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1ea58aa2:::benchmark_data_smithi047_474001_object265:head# 2024-09-18T16:03:54.370 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:54.403 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1fa7c67c:::benchmark_data_smithi047_479866_object134:head# 2024-09-18T16:03:54.403 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:54.437 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:20724c42:::benchmark_data_smithi047_479866_object64:head# 2024-09-18T16:03:54.437 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:54.462 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:20be7217:::benchmark_data_smithi047_474001_object145:head# 2024-09-18T16:03:54.462 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:54.487 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:21551d46:::benchmark_data_smithi047_479866_object1:head# 2024-09-18T16:03:54.487 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:54.545 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:21c5c01f:::benchmark_data_smithi047_474001_object69:head# 2024-09-18T16:03:54.545 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:54.604 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:221f844d:::benchmark_data_smithi047_474001_object350:head# 2024-09-18T16:03:54.604 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:54.654 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:223dcfbd:::benchmark_data_smithi047_479866_object94:head# 2024-09-18T16:03:54.655 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:54.688 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:22d3be20:::benchmark_data_smithi047_474001_object315:head# 2024-09-18T16:03:54.688 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:54.721 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2334c1fe:::benchmark_data_smithi047_474001_object84:head# 2024-09-18T16:03:54.721 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:54.755 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:23ea4327:::benchmark_data_smithi047_479866_object148:head# 2024-09-18T16:03:54.755 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:54.788 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:24393629:::benchmark_data_smithi047_474001_object239:head# 2024-09-18T16:03:54.788 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:54.822 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:25482445:::benchmark_data_smithi047_474001_object53:head# 2024-09-18T16:03:54.822 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:54.855 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2555ae90:::benchmark_data_smithi047_474001_object370:head# 2024-09-18T16:03:54.855 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:54.889 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:26a1c85e:::benchmark_data_smithi047_474001_object32:head# 2024-09-18T16:03:54.889 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:54.922 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:26a53c7b:::benchmark_data_smithi047_479866_object98:head# 2024-09-18T16:03:54.922 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:54.955 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:26b7b87f:::benchmark_data_smithi047_474001_object107:head# 2024-09-18T16:03:54.955 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:54.989 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:26d7a228:::benchmark_data_smithi047_479866_object12:head# 2024-09-18T16:03:54.989 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:55.022 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:27190702:::benchmark_data_smithi047_474001_object218:head# 2024-09-18T16:03:55.022 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:55.056 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:27f9a01e:::benchmark_data_smithi047_474001_object349:head# 2024-09-18T16:03:55.056 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:55.089 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:284c9f3a:::benchmark_data_smithi047_474001_object254:head# 2024-09-18T16:03:55.089 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:55.123 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2856b42f:::benchmark_data_smithi047_474001_object81:head# 2024-09-18T16:03:55.123 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:55.156 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:28ca5326:::benchmark_data_smithi047_479866_object27:head# 2024-09-18T16:03:55.157 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:55.190 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:29494f84:::benchmark_data_smithi047_474001_object248:head# 2024-09-18T16:03:55.190 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:55.223 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2a1b326f:::benchmark_data_smithi047_479866_object151:head# 2024-09-18T16:03:55.223 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:55.257 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2a2243e8:::benchmark_data_smithi047_474001_object1:head# 2024-09-18T16:03:55.257 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:55.290 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2abea481:::benchmark_data_smithi047_474001_object23:head# 2024-09-18T16:03:55.290 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:55.324 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2b1d853f:::benchmark_data_smithi047_479866_object163:head# 2024-09-18T16:03:55.324 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:55.357 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2bcedee4:::benchmark_data_smithi047_479866_object86:head# 2024-09-18T16:03:55.357 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:55.391 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2c9f9183:::benchmark_data_smithi047_479866_object130:head# 2024-09-18T16:03:55.391 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:55.424 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2ced8b80:::benchmark_data_smithi047_474001_object401:head# 2024-09-18T16:03:55.424 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:55.483 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2e2d9e2b:::benchmark_data_smithi047_474001_object367:head# 2024-09-18T16:03:55.483 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:55.516 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2e9ea224:::benchmark_data_smithi047_474001_object106:head# 2024-09-18T16:03:55.516 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:55.549 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2fb8dbb3:::benchmark_data_smithi047_474001_object170:head# 2024-09-18T16:03:55.549 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:55.583 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:300fdc07:::benchmark_data_smithi047_474001_object227:head# 2024-09-18T16:03:55.583 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:55.616 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3078741e:::benchmark_data_smithi047_474001_object139:head# 2024-09-18T16:03:55.617 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:55.650 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:30d72d7d:::benchmark_data_smithi047_479866_object82:head# 2024-09-18T16:03:55.650 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:55.683 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:31669bfe:::benchmark_data_smithi047_474001_object319:head# 2024-09-18T16:03:55.684 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:55.717 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:321ae169:::benchmark_data_smithi047_474001_object244:head# 2024-09-18T16:03:55.717 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:55.750 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:32ff3ac8:::benchmark_data_smithi047_474001_object261:head# 2024-09-18T16:03:55.751 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:55.784 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:33134995:::benchmark_data_smithi047_474001_object310:head# 2024-09-18T16:03:55.784 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:55.817 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:338c48e8:::benchmark_data_smithi047_479866_object168:head# 2024-09-18T16:03:55.817 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:55.850 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:33933c4f:::benchmark_data_smithi047_479866_object22:head# 2024-09-18T16:03:55.851 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:55.884 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:33b1fdf8:::benchmark_data_smithi047_474001_object161:head# 2024-09-18T16:03:55.884 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:55.917 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:33b93a31:::benchmark_data_smithi047_474001_object403:head# 2024-09-18T16:03:55.918 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:55.951 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:33d9125e:::benchmark_data_smithi047_479866_object85:head# 2024-09-18T16:03:55.951 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:55.984 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:33fa48ba:::benchmark_data_smithi047_474001_object289:head# 2024-09-18T16:03:55.984 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:55.999 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:34348ae4:::benchmark_data_smithi047_474001_object6:head# 2024-09-18T16:03:55.999 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:56.035 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:348ab036:::benchmark_data_smithi047_479866_object129:head# 2024-09-18T16:03:56.035 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:56.068 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:34e0f433:::benchmark_data_smithi047_474001_object342:head# 2024-09-18T16:03:56.068 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:56.102 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:355eddbc:::benchmark_data_smithi047_474001_object198:head# 2024-09-18T16:03:56.102 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:56.135 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:35841d04:::benchmark_data_smithi047_474001_object186:head# 2024-09-18T16:03:56.135 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:56.185 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:35c32413:::benchmark_data_smithi047_474001_object413:head# 2024-09-18T16:03:56.185 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:56.235 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:35d09d62:::benchmark_data_smithi047_479866_object3:head# 2024-09-18T16:03:56.235 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:56.269 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:35db8d7b:::benchmark_data_smithi047_479866_object118:head# 2024-09-18T16:03:56.269 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:56.302 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:35e67788:::benchmark_data_smithi047_474001_object388:head# 2024-09-18T16:03:56.302 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:56.336 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:35fd02f5:::benchmark_data_smithi047_479866_object36:head# 2024-09-18T16:03:56.336 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:56.361 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3613c729:::benchmark_data_smithi047_474001_object380:head# 2024-09-18T16:03:56.361 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:56.386 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:365d6d16:::benchmark_data_smithi047_479866_object171:head# 2024-09-18T16:03:56.386 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:56.411 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:36f643e7:::benchmark_data_smithi047_479866_object113:head# 2024-09-18T16:03:56.411 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:56.436 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:37ccfdd9:::benchmark_data_smithi047_474001_object219:head# 2024-09-18T16:03:56.436 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:56.461 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:380af478:::benchmark_data_smithi047_474001_object359:head# 2024-09-18T16:03:56.461 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:56.487 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3812ffca:::benchmark_data_smithi047_474001_object286:head# 2024-09-18T16:03:56.487 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:56.531 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:381ae33c:::benchmark_data_smithi047_474001_object152:head# 2024-09-18T16:03:56.531 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:56.566 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3994ed2b:::benchmark_data_smithi047_474001_object162:head# 2024-09-18T16:03:56.566 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:56.629 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3ad1e94f:::benchmark_data_smithi047_474001_object26:head# 2024-09-18T16:03:56.629 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:56.662 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3b2460a7:::benchmark_data_smithi047_474001_object418:head# 2024-09-18T16:03:56.662 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:56.696 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3b70d910:::benchmark_data_smithi047_474001_object157:head# 2024-09-18T16:03:56.696 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:56.721 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3b94819a:::benchmark_data_smithi047_479866_object159:head# 2024-09-18T16:03:56.721 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:56.746 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3b976a95:::benchmark_data_smithi047_474001_object392:head# 2024-09-18T16:03:56.746 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:56.771 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3c01f944:::benchmark_data_smithi047_474001_object85:head# 2024-09-18T16:03:56.771 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:56.797 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3c08c862:::benchmark_data_smithi047_474001_object209:head# 2024-09-18T16:03:56.797 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:56.830 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3c3b4221:::benchmark_data_smithi047_474001_object302:head# 2024-09-18T16:03:56.830 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:56.863 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3c65ea78:::benchmark_data_smithi047_479866_object88:head# 2024-09-18T16:03:56.863 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:56.897 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3cc8b88e:::benchmark_data_smithi047_479866_object100:head# 2024-09-18T16:03:56.897 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:56.930 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3d6d3489:::benchmark_data_smithi047_479866_object23:head# 2024-09-18T16:03:56.930 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:56.963 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3dfa0593:::benchmark_data_smithi047_479866_object120:head# 2024-09-18T16:03:56.964 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:56.997 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3e01dac2:::benchmark_data_smithi047_479866_object25:head# 2024-09-18T16:03:56.997 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.030 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3e2bc2d1:::benchmark_data_smithi047_474001_object290:head# 2024-09-18T16:03:57.031 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.050 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3e83fb4d:::benchmark_data_smithi047_479866_object140:head# 2024-09-18T16:03:57.050 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.066 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3f0792f2:::benchmark_data_smithi047_474001_object205:head# 2024-09-18T16:03:57.066 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.083 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3f0bbc7e:::benchmark_data_smithi047_474001_object214:head# 2024-09-18T16:03:57.083 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.100 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:40e84920:::benchmark_data_smithi047_474001_object201:head# 2024-09-18T16:03:57.100 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.116 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:410710e5:::benchmark_data_smithi047_474001_object360:head# 2024-09-18T16:03:57.116 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.133 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4286a502:::benchmark_data_smithi047_474001_object280:head# 2024-09-18T16:03:57.133 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.149 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:429b03e9:::benchmark_data_smithi047_474001_object249:head# 2024-09-18T16:03:57.150 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.166 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:43c8723a:::benchmark_data_smithi047_474001_object4:head# 2024-09-18T16:03:57.166 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.183 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:43c8e265:::benchmark_data_smithi047_474001_object208:head# 2024-09-18T16:03:57.183 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.200 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:43d378b4:::benchmark_data_smithi047_474001_object425:head# 2024-09-18T16:03:57.200 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.216 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:43f9c3e1:::benchmark_data_smithi047_479866_object45:head# 2024-09-18T16:03:57.216 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.247 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:43fb4408:::benchmark_data_smithi047_474001_object399:head# 2024-09-18T16:03:57.247 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.281 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:44ab53e5:::benchmark_data_smithi047_474001_object27:head# 2024-09-18T16:03:57.281 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.314 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4515f62e:::benchmark_data_smithi047_474001_object143:head# 2024-09-18T16:03:57.314 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.348 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:457b93bc:::benchmark_data_smithi047_474001_object300:head# 2024-09-18T16:03:57.348 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.381 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:457e7435:::benchmark_data_smithi047_474001_object231:head# 2024-09-18T16:03:57.381 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.400 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4594ea0a:::benchmark_data_smithi047_479866_object87:head# 2024-09-18T16:03:57.400 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.419 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:45a3456f:::benchmark_data_smithi047_479866_object74:head# 2024-09-18T16:03:57.419 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.436 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:45f35e08:::benchmark_data_smithi047_479866_object99:head# 2024-09-18T16:03:57.436 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.453 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4640e213:::benchmark_data_smithi047_479866_object66:head# 2024-09-18T16:03:57.453 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.469 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:468b56fb:::benchmark_data_smithi047_474001_object242:head# 2024-09-18T16:03:57.469 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.486 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:47454808:::benchmark_data_smithi047_474001_object394:head# 2024-09-18T16:03:57.486 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.502 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4757d830:::benchmark_data_smithi047_474001_object147:head# 2024-09-18T16:03:57.503 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.519 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4790fa42:::benchmark_data_smithi047_474001_object16:head# 2024-09-18T16:03:57.519 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.536 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:481edef6:::benchmark_data_smithi047_474001_object129:head# 2024-09-18T16:03:57.536 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.553 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:498127df:::benchmark_data_smithi047_474001_object409:head# 2024-09-18T16:03:57.553 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.569 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4a94b2bf:::benchmark_data_smithi047_474001_object331:head# 2024-09-18T16:03:57.569 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.586 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4b0f17cd:::benchmark_data_smithi047_474001_object194:head# 2024-09-18T16:03:57.586 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.603 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4be7a76e:::benchmark_data_smithi047_479866_object108:head# 2024-09-18T16:03:57.603 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.620 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4ccceada:::benchmark_data_smithi047_474001_object213:head# 2024-09-18T16:03:57.620 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.636 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4ce89ee7:::benchmark_data_smithi047_479866_object26:head# 2024-09-18T16:03:57.636 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.653 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4d145d82:::benchmark_data_smithi047_474001_object206:head# 2024-09-18T16:03:57.653 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.669 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4dead740:::benchmark_data_smithi047_479866_object41:head# 2024-09-18T16:03:57.669 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.686 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4e67dcba:::benchmark_data_smithi047_474001_object144:head# 2024-09-18T16:03:57.686 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.703 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4e6bc7a4:::benchmark_data_smithi047_474001_object165:head# 2024-09-18T16:03:57.703 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.719 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4e75155f:::benchmark_data_smithi047_479866_object29:head# 2024-09-18T16:03:57.720 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.736 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4e7fe7b2:::benchmark_data_smithi047_474001_object431:head# 2024-09-18T16:03:57.736 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.753 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4f5e76df:::benchmark_data_smithi047_474001_object415:head# 2024-09-18T16:03:57.753 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.770 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5030ebeb:::benchmark_data_smithi047_474001_object335:head# 2024-09-18T16:03:57.770 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.786 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:505987dd:::benchmark_data_smithi047_474001_object313:head# 2024-09-18T16:03:57.786 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.803 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:50a1999f:::benchmark_data_smithi047_474001_object190:head# 2024-09-18T16:03:57.803 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.820 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:510b9f73:::benchmark_data_smithi047_479866_object125:head# 2024-09-18T16:03:57.820 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.836 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:51905d78:::benchmark_data_smithi047_474001_object371:head# 2024-09-18T16:03:57.836 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.853 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5275f435:::benchmark_data_smithi047_479866_object145:head# 2024-09-18T16:03:57.853 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.870 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5295c252:::benchmark_data_smithi047_479866_object91:head# 2024-09-18T16:03:57.870 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.886 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:53516fa7:::benchmark_data_smithi047_474001_object47:head# 2024-09-18T16:03:57.886 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.903 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:53a75224:::benchmark_data_smithi047_474001_object277:head# 2024-09-18T16:03:57.903 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.920 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:53bc5374:::benchmark_data_smithi047_474001_object304:head# 2024-09-18T16:03:57.920 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.936 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:54037397:::benchmark_data_smithi047_474001_object168:head# 2024-09-18T16:03:57.936 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.953 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:54e93f24:::benchmark_data_smithi047_474001_object351:head# 2024-09-18T16:03:57.953 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.969 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5549c508:::benchmark_data_smithi047_474001_object154:head# 2024-09-18T16:03:57.970 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:57.986 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:55a8f195:::benchmark_data_smithi047_474001_object183:head# 2024-09-18T16:03:57.987 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.003 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:58bb18eb:::benchmark_data_smithi047_474001_object92:head# 2024-09-18T16:03:58.003 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.020 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:58e3824a:::benchmark_data_smithi047_474001_object22:head# 2024-09-18T16:03:58.020 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.036 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:58eacbbf:::benchmark_data_smithi047_474001_object100:head# 2024-09-18T16:03:58.037 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.053 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:590d7cd0:::benchmark_data_smithi047_474001_object328:head# 2024-09-18T16:03:58.053 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.070 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:595ccf9f:::benchmark_data_smithi047_479866_object20:head# 2024-09-18T16:03:58.070 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.086 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:59701efb:::benchmark_data_smithi047_474001_object243:head# 2024-09-18T16:03:58.087 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.103 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:59852438:::benchmark_data_smithi047_474001_object83:head# 2024-09-18T16:03:58.103 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.120 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5a123740:::benchmark_data_smithi047_474001_object36:head# 2024-09-18T16:03:58.120 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.172 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5b73d6c3:::benchmark_data_smithi047_474001_object284:head# 2024-09-18T16:03:58.172 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.203 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5be37116:::benchmark_data_smithi047_479866_object33:head# 2024-09-18T16:03:58.203 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.220 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5c8e3731:::benchmark_data_smithi047_474001_object132:head# 2024-09-18T16:03:58.220 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.237 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5caa7c9f:::benchmark_data_smithi047_474001_object416:head# 2024-09-18T16:03:58.237 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.256 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5cf5ea91:::benchmark_data_smithi047_474001_object397:head# 2024-09-18T16:03:58.256 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.270 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5d1b58ad:::benchmark_data_smithi047_474001_object269:head# 2024-09-18T16:03:58.270 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.287 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5da527aa:::benchmark_data_smithi047_479866_object8:head# 2024-09-18T16:03:58.287 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.303 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5e669bb0:::benchmark_data_smithi047_474001_object405:head# 2024-09-18T16:03:58.303 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.328 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5eac6d5a:::benchmark_data_smithi047_474001_object411:head# 2024-09-18T16:03:58.329 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.345 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5ef5c1d9:::benchmark_data_smithi047_474001_object385:head# 2024-09-18T16:03:58.345 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.362 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5f7398fd:::benchmark_data_smithi047_474001_object68:head# 2024-09-18T16:03:58.362 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.378 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5f8ba50b:::benchmark_data_smithi047_474001_object174:head# 2024-09-18T16:03:58.378 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.395 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5fb38f21:::benchmark_data_smithi047_474001_object75:head# 2024-09-18T16:03:58.395 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.412 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5fc0bd7a:::benchmark_data_smithi047_479866_object4:head# 2024-09-18T16:03:58.412 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.429 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:60a5ff1a:::benchmark_data_smithi047_474001_object298:head# 2024-09-18T16:03:58.429 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.445 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:60e704db:::benchmark_data_smithi047_479866_object31:head# 2024-09-18T16:03:58.445 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.462 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6159ad07:::benchmark_data_smithi047_474001_object221:head# 2024-09-18T16:03:58.462 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.478 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:62273e63:::benchmark_data_smithi047_474001_object62:head# 2024-09-18T16:03:58.479 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.515 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:63cf18a2:::benchmark_data_smithi047_474001_object272:head# 2024-09-18T16:03:58.515 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.565 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:63f415e6:::benchmark_data_smithi047_474001_object58:head# 2024-09-18T16:03:58.566 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.596 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:64b93fc8:::benchmark_data_smithi047_479866_object150:head# 2024-09-18T16:03:58.596 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.629 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:64ff073d:::benchmark_data_smithi047_474001_object362:head# 2024-09-18T16:03:58.629 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.645 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:657bccc4:::benchmark_data_smithi047_474001_object169:head# 2024-09-18T16:03:58.646 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.664 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:660836d4:::benchmark_data_smithi047_474001_object39:head# 2024-09-18T16:03:58.664 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.679 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:66682ebd:::benchmark_data_smithi047_474001_object101:head# 2024-09-18T16:03:58.679 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.695 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:66e730e2:::benchmark_data_smithi047_474001_object61:head# 2024-09-18T16:03:58.695 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.712 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:66fbc3ec:::benchmark_data_smithi047_474001_object71:head# 2024-09-18T16:03:58.712 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.729 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6717adea:::benchmark_data_smithi047_474001_object317:head# 2024-09-18T16:03:58.729 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.745 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:67bf8aa9:::benchmark_data_smithi047_479866_object160:head# 2024-09-18T16:03:58.745 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.762 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:67e02299:::benchmark_data_smithi047_479866_object67:head# 2024-09-18T16:03:58.762 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.779 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:68016fab:::benchmark_data_smithi047_479866_object127:head# 2024-09-18T16:03:58.779 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.795 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:68107ec1:::benchmark_data_smithi047_474001_object195:head# 2024-09-18T16:03:58.796 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.812 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6812f8cb:::benchmark_data_smithi047_474001_object264:head# 2024-09-18T16:03:58.812 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.829 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:68f519f4:::benchmark_data_smithi047_474001_object343:head# 2024-09-18T16:03:58.829 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.846 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:694b8617:::benchmark_data_smithi047_479866_object121:head# 2024-09-18T16:03:58.846 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.862 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6a39a8d4:::benchmark_data_smithi047_474001_object66:head# 2024-09-18T16:03:58.862 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.879 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6ab23d26:::benchmark_data_smithi047_474001_object9:head# 2024-09-18T16:03:58.879 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.895 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6aeb612f:::benchmark_data_smithi047_479866_object43:head# 2024-09-18T16:03:58.896 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.912 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6b25d55a:::benchmark_data_smithi047_474001_object72:head# 2024-09-18T16:03:58.913 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.937 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6b5fc221:::benchmark_data_smithi047_474001_object334:head# 2024-09-18T16:03:58.937 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.962 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6c2d825c:::benchmark_data_smithi047_474001_object5:head# 2024-09-18T16:03:58.962 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:58.987 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6caa7eaf:::benchmark_data_smithi047_474001_object263:head# 2024-09-18T16:03:58.987 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:59.012 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6df9d633:::benchmark_data_smithi047_479866_object37:head# 2024-09-18T16:03:59.012 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:59.037 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6dfd6f4e:::benchmark_data_smithi047_474001_object251:head# 2024-09-18T16:03:59.037 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:59.062 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6e9248a9:::benchmark_data_smithi047_474001_object245:head# 2024-09-18T16:03:59.062 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:59.087 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6f38583d:::benchmark_data_smithi047_479866_object47:head# 2024-09-18T16:03:59.088 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:59.112 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6f58cdd1:::benchmark_data_smithi047_474001_object430:head# 2024-09-18T16:03:59.113 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:59.137 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6f9d3f5a:::benchmark_data_smithi047_479866_object73:head# 2024-09-18T16:03:59.138 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:59.171 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6fb45753:::benchmark_data_smithi047_479866_object103:head# 2024-09-18T16:03:59.171 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:59.204 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:71084a82:::benchmark_data_smithi047_474001_object398:head# 2024-09-18T16:03:59.204 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:59.237 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:712abb08:::benchmark_data_smithi047_479866_object13:head# 2024-09-18T16:03:59.238 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:59.271 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:712fda89:::benchmark_data_smithi047_479866_object141:head# 2024-09-18T16:03:59.271 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:59.304 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:715472aa:::benchmark_data_smithi047_479866_object32:head# 2024-09-18T16:03:59.304 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:59.363 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:722633c0:::benchmark_data_smithi047_474001_object7:head# 2024-09-18T16:03:59.363 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:59.396 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:73078ec4:::benchmark_data_smithi047_474001_object200:head# 2024-09-18T16:03:59.396 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:59.429 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7319939a:::benchmark_data_smithi047_474001_object378:head# 2024-09-18T16:03:59.429 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:59.463 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:732bbbc2:::benchmark_data_smithi047_474001_object113:head# 2024-09-18T16:03:59.463 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:59.496 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:73694cfd:::benchmark_data_smithi047_474001_object257:head# 2024-09-18T16:03:59.496 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:59.529 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:739a0e00:::benchmark_data_smithi047_474001_object11:head# 2024-09-18T16:03:59.530 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:59.563 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:74060ef1:::benchmark_data_smithi047_479866_object19:head# 2024-09-18T16:03:59.563 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:59.596 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:748a50de:::benchmark_data_smithi047_474001_object223:head# 2024-09-18T16:03:59.596 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:59.629 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:74cbfe52:::benchmark_data_smithi047_474001_object309:head# 2024-09-18T16:03:59.630 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:59.663 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:751b68bd:::benchmark_data_smithi047_474001_object45:head# 2024-09-18T16:03:59.663 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:59.696 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7634e380:::benchmark_data_smithi047_479866_object112:head# 2024-09-18T16:03:59.696 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:59.729 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:76583154:::benchmark_data_smithi047_474001_object13:head# 2024-09-18T16:03:59.730 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:59.763 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7669c90e:::benchmark_data_smithi047_474001_object216:head# 2024-09-18T16:03:59.763 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:59.796 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:76dc48ec:::benchmark_data_smithi047_474001_object79:head# 2024-09-18T16:03:59.796 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:59.829 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:77226950:::benchmark_data_smithi047_474001_object386:head# 2024-09-18T16:03:59.830 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:59.863 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:77a33037:::benchmark_data_smithi047_474001_object197:head# 2024-09-18T16:03:59.863 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:59.896 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:77f7974a:::benchmark_data_smithi047_474001_object123:head# 2024-09-18T16:03:59.896 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:59.930 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:78d90d4e:::benchmark_data_smithi047_474001_object301:head# 2024-09-18T16:03:59.930 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:59.963 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:78f80541:::benchmark_data_smithi047_479866_object173:head# 2024-09-18T16:03:59.963 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:03:59.996 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7ab906db:::benchmark_data_smithi047_479866_object131:head# 2024-09-18T16:03:59.996 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:00.030 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7b36e97b:::benchmark_data_smithi047_474001_object292:head# 2024-09-18T16:04:00.030 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:00.088 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7bde396b:::benchmark_data_smithi047_479866_object38:head# 2024-09-18T16:04:00.088 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:00.121 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7c6e25b9:::benchmark_data_smithi047_479866_object0:head# 2024-09-18T16:04:00.121 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:00.204 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7c70570e:::benchmark_data_smithi047_479866_object105:head# 2024-09-18T16:04:00.204 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:00.238 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7c8233bd:::benchmark_data_smithi047_474001_object76:head# 2024-09-18T16:04:00.238 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:00.271 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7cce9f64:::benchmark_data_smithi047_474001_object0:head# 2024-09-18T16:04:00.271 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:00.305 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7d8e31eb:::benchmark_data_smithi047_474001_object193:head# 2024-09-18T16:04:00.305 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:00.338 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7dc806f2:::benchmark_data_smithi047_474001_object126:head# 2024-09-18T16:04:00.338 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:00.371 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7dfaf25d:::benchmark_data_smithi047_474001_object260:head# 2024-09-18T16:04:00.371 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:00.405 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7e00f5f7:::benchmark_data_smithi047_474001_object98:head# 2024-09-18T16:04:00.405 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:00.438 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7f679bd9:::benchmark_data_smithi047_474001_object379:head# 2024-09-18T16:04:00.438 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:00.471 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7fa61220:::benchmark_data_smithi047_474001_object354:head# 2024-09-18T16:04:00.472 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:00.530 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7fb32424:::benchmark_data_smithi047_474001_object241:head# 2024-09-18T16:04:00.530 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:00.614 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:806a8ff0:::benchmark_data_smithi047_474001_object14:head# 2024-09-18T16:04:00.614 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:00.663 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8114a0ce:::benchmark_data_smithi047_479866_object84:head# 2024-09-18T16:04:00.663 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:00.696 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:811d44c1:::benchmark_data_smithi047_479866_object110:head# 2024-09-18T16:04:00.697 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:00.730 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:81d272b1:::benchmark_data_smithi047_479866_object78:head# 2024-09-18T16:04:00.730 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:00.763 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:81f3f33d:::benchmark_data_smithi047_479866_object119:head# 2024-09-18T16:04:00.763 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:00.796 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:827184ed:::benchmark_data_smithi047_474001_object202:head# 2024-09-18T16:04:00.796 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:00.838 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:829e9bf2:::benchmark_data_smithi047_479866_object135:head# 2024-09-18T16:04:00.838 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:00.863 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:83cb4be8:::benchmark_data_smithi047_474001_object189:head# 2024-09-18T16:04:00.863 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:00.888 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:83d7b3d6:::benchmark_data_smithi047_474001_object427:head# 2024-09-18T16:04:00.888 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:00.913 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:83e6a7ba:::benchmark_data_smithi047_474001_object326:head# 2024-09-18T16:04:00.913 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:00.938 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:83e714c1:::benchmark_data_smithi047_474001_object384:head# 2024-09-18T16:04:00.938 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:00.963 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:84fc4a07:::benchmark_data_smithi047_474001_object235:head# 2024-09-18T16:04:00.963 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:00.988 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:859c4c12:::benchmark_data_smithi047_479866_object54:head# 2024-09-18T16:04:00.988 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:01.013 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:85afe96e:::benchmark_data_smithi047_479866_object71:head# 2024-09-18T16:04:01.013 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:01.038 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8635a002:::benchmark_data_smithi047_474001_object8:head# 2024-09-18T16:04:01.038 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:01.063 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8659e9ad:::benchmark_data_smithi047_474001_object111:head# 2024-09-18T16:04:01.063 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:01.088 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:865f4a69:::benchmark_data_smithi047_474001_object312:head# 2024-09-18T16:04:01.089 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:01.113 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:87201b49:::benchmark_data_smithi047_474001_object185:head# 2024-09-18T16:04:01.113 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:01.138 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8727bd0b:::benchmark_data_smithi047_479866_object107:head# 2024-09-18T16:04:01.138 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:01.163 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:87f935e7:::benchmark_data_smithi047_479866_object62:head# 2024-09-18T16:04:01.163 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:01.188 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:889f60a0:::benchmark_data_smithi047_474001_object109:head# 2024-09-18T16:04:01.188 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:01.213 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8904e269:::benchmark_data_smithi047_474001_object228:head# 2024-09-18T16:04:01.213 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:01.238 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:89ec4c33:::benchmark_data_smithi047_474001_object133:head# 2024-09-18T16:04:01.238 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:01.263 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8a2258fa:::benchmark_data_smithi047_474001_object119:head# 2024-09-18T16:04:01.263 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:01.289 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8a67fb29:::benchmark_data_smithi047_479866_object136:head# 2024-09-18T16:04:01.289 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:01.313 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8aba1a3a:::benchmark_data_smithi047_474001_object291:head# 2024-09-18T16:04:01.314 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:01.338 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8cb0b5a1:::benchmark_data_smithi047_474001_object3:head# 2024-09-18T16:04:01.338 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:01.363 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8d65845a:::benchmark_data_smithi047_474001_object324:head# 2024-09-18T16:04:01.364 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:01.389 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8d83eb55:::benchmark_data_smithi047_479866_object142:head# 2024-09-18T16:04:01.389 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:01.422 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8e21472c:::benchmark_data_smithi047_474001_object110:head# 2024-09-18T16:04:01.422 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:01.455 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8e8844e6:::benchmark_data_smithi047_479866_object172:head# 2024-09-18T16:04:01.455 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:01.497 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8efa467b:::benchmark_data_smithi047_474001_object247:head# 2024-09-18T16:04:01.497 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:01.530 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8f162f27:::benchmark_data_smithi047_474001_object222:head# 2024-09-18T16:04:01.530 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:01.564 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8f2ab4de:::benchmark_data_smithi047_474001_object112:head# 2024-09-18T16:04:01.564 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:01.597 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8f3016de:::benchmark_data_smithi047_474001_object337:head# 2024-09-18T16:04:01.597 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:01.630 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8faff132:::benchmark_data_smithi047_479866_object167:head# 2024-09-18T16:04:01.630 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:01.663 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:906470c5:::benchmark_data_smithi047_474001_object108:head# 2024-09-18T16:04:01.664 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:01.697 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:90a887d3:::benchmark_data_smithi047_479866_object175:head# 2024-09-18T16:04:01.697 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:01.730 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:90a988ef:::benchmark_data_smithi047_474001_object391:head# 2024-09-18T16:04:01.730 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:01.764 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:90fba3e5:::benchmark_data_smithi047_479866_object111:head# 2024-09-18T16:04:01.764 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:01.797 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:91461d4c:::benchmark_data_smithi047_474001_object252:head# 2024-09-18T16:04:01.797 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:01.831 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:92438533:::benchmark_data_smithi047_474001_object17:head# 2024-09-18T16:04:01.831 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:01.864 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9256e853:::benchmark_data_smithi047_474001_object314:head# 2024-09-18T16:04:01.864 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:01.897 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:928bb93a:::benchmark_data_smithi047_474001_object51:head# 2024-09-18T16:04:01.897 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:01.930 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:92b0a2c6:::benchmark_data_smithi047_474001_object250:head# 2024-09-18T16:04:01.931 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:01.964 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9324d622:::benchmark_data_smithi047_474001_object95:head# 2024-09-18T16:04:01.964 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:01.997 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:939c1aa8:::benchmark_data_smithi047_474001_object382:head# 2024-09-18T16:04:01.997 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:02.030 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:94353a0c:::benchmark_data_smithi047_479866_object169:head# 2024-09-18T16:04:02.031 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:02.064 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:945bc203:::benchmark_data_smithi047_474001_object177:head# 2024-09-18T16:04:02.064 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:02.097 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:94707a46:::benchmark_data_smithi047_474001_object138:head# 2024-09-18T16:04:02.097 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:02.147 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:94f79b31:::benchmark_data_smithi047_474001_object253:head# 2024-09-18T16:04:02.147 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:02.197 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9502e979:::benchmark_data_smithi047_474001_object50:head# 2024-09-18T16:04:02.197 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:02.231 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:953c3a4c:::benchmark_data_smithi047_474001_object74:head# 2024-09-18T16:04:02.231 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:02.264 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:95921a67:::benchmark_data_smithi047_479866_object174:head# 2024-09-18T16:04:02.264 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:02.297 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:95e04f4b:::benchmark_data_smithi047_479866_object72:head# 2024-09-18T16:04:02.297 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:02.331 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:95edf5ad:::benchmark_data_smithi047_474001_object345:head# 2024-09-18T16:04:02.331 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:02.364 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9612cb9d:::benchmark_data_smithi047_474001_object59:head# 2024-09-18T16:04:02.364 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:02.381 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9670fdab:::benchmark_data_smithi047_474001_object116:head# 2024-09-18T16:04:02.381 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:02.397 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:978a53fd:::benchmark_data_smithi047_474001_object358:head# 2024-09-18T16:04:02.398 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:02.431 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:986d3d20:::benchmark_data_smithi047_474001_object38:head# 2024-09-18T16:04:02.431 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:02.464 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:989e240c:::benchmark_data_smithi047_479866_object77:head# 2024-09-18T16:04:02.464 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:02.514 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:99a611fc:::benchmark_data_smithi047_479866_object158:head# 2024-09-18T16:04:02.514 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:02.564 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9a0edcb5:::benchmark_data_smithi047_474001_object89:head# 2024-09-18T16:04:02.564 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:02.631 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9b79df93:::benchmark_data_smithi047_479866_object132:head# 2024-09-18T16:04:02.631 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:02.664 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9bef3576:::benchmark_data_smithi047_474001_object180:head# 2024-09-18T16:04:02.664 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:02.697 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9dafb813:::benchmark_data_smithi047_474001_object226:head# 2024-09-18T16:04:02.697 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:02.731 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9dc7b4a6:::benchmark_data_smithi047_474001_object346:head# 2024-09-18T16:04:02.731 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:02.764 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9de9d04a:::benchmark_data_smithi047_474001_object338:head# 2024-09-18T16:04:02.764 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:02.806 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9e17b853:::benchmark_data_smithi047_474001_object124:head# 2024-09-18T16:04:02.806 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:02.839 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9e534a18:::benchmark_data_smithi047_474001_object70:head# 2024-09-18T16:04:02.839 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:02.873 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9e5ea7f3:::benchmark_data_smithi047_474001_object348:head# 2024-09-18T16:04:02.873 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:02.906 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a0254336:::benchmark_data_smithi047_474001_object188:head# 2024-09-18T16:04:02.906 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:02.939 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a0c154fe:::benchmark_data_smithi047_474001_object383:head# 2024-09-18T16:04:02.939 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:02.973 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a16d06ab:::benchmark_data_smithi047_479866_object76:head# 2024-09-18T16:04:02.973 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:03.006 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a1981567:::benchmark_data_smithi047_474001_object414:head# 2024-09-18T16:04:03.006 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:03.039 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a23c3345:::benchmark_data_smithi047_474001_object408:head# 2024-09-18T16:04:03.039 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:03.073 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a244fe5e:::benchmark_data_smithi047_474001_object307:head# 2024-09-18T16:04:03.073 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:03.106 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a256b83b:::benchmark_data_smithi047_474001_object417:head# 2024-09-18T16:04:03.106 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:03.139 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a331eaa2:::benchmark_data_smithi047_474001_object103:head# 2024-09-18T16:04:03.140 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:03.173 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a3674a4c:::benchmark_data_smithi047_479866_object96:head# 2024-09-18T16:04:03.173 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:03.206 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a36fdbde:::benchmark_data_smithi047_474001_object125:head# 2024-09-18T16:04:03.206 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:03.231 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a388d7f6:::benchmark_data_smithi047_474001_object46:head# 2024-09-18T16:04:03.232 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:03.264 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a473749a:::benchmark_data_smithi047_474001_object396:head# 2024-09-18T16:04:03.265 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:03.290 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a49be1f1:::benchmark_data_smithi047_479866_object44:head# 2024-09-18T16:04:03.290 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:03.315 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a53e9cf3:::benchmark_data_smithi047_479866_object53:head# 2024-09-18T16:04:03.315 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:03.340 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a546fc38:::benchmark_data_smithi047_474001_object344:head# 2024-09-18T16:04:03.340 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:03.365 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a56684c3:::benchmark_data_smithi047_474001_object73:head# 2024-09-18T16:04:03.365 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:03.390 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a6a637ca:::benchmark_data_smithi047_474001_object352:head# 2024-09-18T16:04:03.390 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:03.415 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a6ac7233:::benchmark_data_smithi047_474001_object273:head# 2024-09-18T16:04:03.415 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:03.439 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a6be61c0:::benchmark_data_smithi047_474001_object212:head# 2024-09-18T16:04:03.440 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:03.465 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a6d03d77:::benchmark_data_smithi047_474001_object429:head# 2024-09-18T16:04:03.465 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:03.489 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a7029afc:::benchmark_data_smithi047_474001_object57:head# 2024-09-18T16:04:03.490 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:03.514 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a75b0434:::benchmark_data_smithi047_479866_object95:head# 2024-09-18T16:04:03.515 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:03.540 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a87abb60:::benchmark_data_smithi047_474001_object28:head# 2024-09-18T16:04:03.540 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:03.565 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a99c9566:::benchmark_data_smithi047_474001_object285:head# 2024-09-18T16:04:03.565 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:03.610 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a9ee7f09:::benchmark_data_smithi047_474001_object60:head# 2024-09-18T16:04:03.610 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:03.631 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:aaf069cf:::benchmark_data_smithi047_474001_object426:head# 2024-09-18T16:04:03.631 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:03.656 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ab22f0d1:::benchmark_data_smithi047_479866_object63:head# 2024-09-18T16:04:03.657 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:03.690 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ac9a95a5:::benchmark_data_smithi047_474001_object327:head# 2024-09-18T16:04:03.690 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:03.723 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ad75c20a:::benchmark_data_smithi047_474001_object322:head# 2024-09-18T16:04:03.723 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:03.757 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:add97b11:::benchmark_data_smithi047_474001_object120:head# 2024-09-18T16:04:03.757 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:03.790 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ae2bc6da:::benchmark_data_smithi047_474001_object374:head# 2024-09-18T16:04:03.790 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:03.823 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ae32eaa9:::benchmark_data_smithi047_474001_object217:head# 2024-09-18T16:04:03.823 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:03.856 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ae5f08a5:::benchmark_data_smithi047_474001_object156:head# 2024-09-18T16:04:03.857 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:03.890 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ae90731a:::benchmark_data_smithi047_474001_object275:head# 2024-09-18T16:04:03.890 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:03.923 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:aeb96a61:::benchmark_data_smithi047_474001_object35:head# 2024-09-18T16:04:03.923 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:03.957 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:af3252fa:::benchmark_data_smithi047_479866_object56:head# 2024-09-18T16:04:03.957 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:03.998 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:af8861c2:::benchmark_data_smithi047_474001_object19:head# 2024-09-18T16:04:03.999 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:04.032 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:afa325d5:::benchmark_data_smithi047_479866_object155:head# 2024-09-18T16:04:04.032 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:04.065 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:afa79d73:::benchmark_data_smithi047_474001_object297:head# 2024-09-18T16:04:04.065 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:04.098 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b06f5ff6:::benchmark_data_smithi047_474001_object96:head# 2024-09-18T16:04:04.098 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:04.146 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b09fcb0b:::benchmark_data_smithi047_474001_object330:head# 2024-09-18T16:04:04.147 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:04.213 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b0d19e8f:::benchmark_data_smithi047_479866_object116:head# 2024-09-18T16:04:04.213 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:04.248 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b0f4cb26:::benchmark_data_smithi047_474001_object262:head# 2024-09-18T16:04:04.248 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:04.282 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b0faad12:::benchmark_data_smithi047_479866_object89:head# 2024-09-18T16:04:04.282 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:04.315 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b1589877:::benchmark_data_smithi047_474001_object332:head# 2024-09-18T16:04:04.315 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:04.348 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b1c4832b:::benchmark_data_smithi047_479866_object157:head# 2024-09-18T16:04:04.348 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:04.382 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b1dd1fef:::benchmark_data_smithi047_474001_object134:head# 2024-09-18T16:04:04.382 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:04.415 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b2ada191:::benchmark_data_smithi047_479866_object59:head# 2024-09-18T16:04:04.415 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:04.448 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b3f3163f:::benchmark_data_smithi047_474001_object341:head# 2024-09-18T16:04:04.449 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:04.482 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b5265e20:::benchmark_data_smithi047_474001_object159:head# 2024-09-18T16:04:04.482 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:04.539 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b533e7e6:::benchmark_data_smithi047_474001_object78:head# 2024-09-18T16:04:04.539 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:04.623 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b539a2e0:::benchmark_data_smithi047_479866_object149:head# 2024-09-18T16:04:04.623 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:04.673 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b585c96f:::benchmark_data_smithi047_474001_object422:head# 2024-09-18T16:04:04.674 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:04.707 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b5d2794c:::benchmark_data_smithi047_479866_object10:head# 2024-09-18T16:04:04.707 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:04.749 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b5e588a2:::benchmark_data_smithi047_474001_object30:head# 2024-09-18T16:04:04.749 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:04.784 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b606ef85:::benchmark_data_smithi047_474001_object12:head# 2024-09-18T16:04:04.784 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:04.815 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b61ac151:::benchmark_data_smithi047_474001_object361:head# 2024-09-18T16:04:04.815 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:04.849 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b69470e5:::benchmark_data_smithi047_479866_object126:head# 2024-09-18T16:04:04.849 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:04.882 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b6f6d98b:::benchmark_data_smithi047_479866_object143:head# 2024-09-18T16:04:04.882 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:04.915 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b707ad78:::benchmark_data_smithi047_474001_object33:head# 2024-09-18T16:04:04.916 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:04.949 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b72090f3:::benchmark_data_smithi047_474001_object225:head# 2024-09-18T16:04:04.949 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:04.982 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b74f9085:::benchmark_data_smithi047_474001_object224:head# 2024-09-18T16:04:04.982 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:05.015 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b7d7a44f:::benchmark_data_smithi047_479866_object6:head# 2024-09-18T16:04:05.015 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:05.049 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b809a3ef:::benchmark_data_smithi047_479866_object11:head# 2024-09-18T16:04:05.049 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:05.082 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b81c2890:::benchmark_data_smithi047_479866_object83:head# 2024-09-18T16:04:05.082 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:05.133 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b854e709:::benchmark_data_smithi047_474001_object279:head# 2024-09-18T16:04:05.133 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:05.207 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b8615e71:::benchmark_last_metadata:head# 2024-09-18T16:04:05.207 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:05.240 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b88427b9:::benchmark_data_smithi047_474001_object204:head# 2024-09-18T16:04:05.241 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:05.274 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b8a01441:::benchmark_data_smithi047_474001_object99:head# 2024-09-18T16:04:05.274 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:05.307 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b8d632ef:::benchmark_data_smithi047_474001_object49:head# 2024-09-18T16:04:05.307 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:05.340 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b9a03ab3:::benchmark_data_smithi047_474001_object41:head# 2024-09-18T16:04:05.341 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:05.374 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ba233a46:::benchmark_data_smithi047_474001_object368:head# 2024-09-18T16:04:05.374 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:05.407 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ba80b0e7:::benchmark_data_smithi047_479866_object65:head# 2024-09-18T16:04:05.407 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:05.440 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:bb9dcf18:::benchmark_data_smithi047_474001_object306:head# 2024-09-18T16:04:05.440 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:05.474 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:bc2e5e68:::benchmark_data_smithi047_474001_object24:head# 2024-09-18T16:04:05.474 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:05.507 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:bc5f16d1:::benchmark_data_smithi047_479866_object16:head# 2024-09-18T16:04:05.508 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:05.532 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:bcb5235a:::benchmark_data_smithi047_479866_object146:head# 2024-09-18T16:04:05.532 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:05.557 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:bd291308:::benchmark_data_smithi047_474001_object316:head# 2024-09-18T16:04:05.557 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:05.582 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:bd68e671:::benchmark_data_smithi047_474001_object44:head# 2024-09-18T16:04:05.582 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:05.607 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:bdd1c304:::benchmark_data_smithi047_474001_object237:head# 2024-09-18T16:04:05.607 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:05.632 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:bdff383c:::benchmark_data_smithi047_479866_object7:head# 2024-09-18T16:04:05.632 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:05.657 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:be3f8098:::benchmark_data_smithi047_474001_object423:head# 2024-09-18T16:04:05.657 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:05.682 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:be77543b:::benchmark_data_smithi047_474001_object77:head# 2024-09-18T16:04:05.682 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:05.707 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:be86e7ec:::benchmark_data_smithi047_474001_object122:head# 2024-09-18T16:04:05.708 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:05.732 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:bed16cbf:::benchmark_data_smithi047_474001_object137:head# 2024-09-18T16:04:05.732 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:05.757 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:bf89fe68:::benchmark_data_smithi047_474001_object54:head# 2024-09-18T16:04:05.757 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:05.782 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:bfe9f633:::benchmark_data_smithi047_479866_object2:head# 2024-09-18T16:04:05.782 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:05.807 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c08db5a2:::benchmark_data_smithi047_474001_object236:head# 2024-09-18T16:04:05.808 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:05.853 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c18ab8fc:::benchmark_data_smithi047_474001_object131:head# 2024-09-18T16:04:05.853 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:05.874 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c1fcb644:::benchmark_data_smithi047_474001_object158:head# 2024-09-18T16:04:05.874 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:05.899 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c24a7c46:::benchmark_data_smithi047_474001_object282:head# 2024-09-18T16:04:05.899 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:05.924 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c313dcbc:::benchmark_data_smithi047_474001_object321:head# 2024-09-18T16:04:05.924 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:05.958 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c31fafef:::benchmark_data_smithi047_479866_object69:head# 2024-09-18T16:04:05.958 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:05.991 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c35a5205:::benchmark_data_smithi047_474001_object303:head# 2024-09-18T16:04:05.991 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:06.024 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c3d61783:::benchmark_data_smithi047_474001_object288:head# 2024-09-18T16:04:06.024 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:06.057 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c3fe223d:::benchmark_data_smithi047_474001_object356:head# 2024-09-18T16:04:06.057 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:06.091 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c4b79889:::benchmark_data_smithi047_474001_object325:head# 2024-09-18T16:04:06.091 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:06.124 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c4c4398f:::benchmark_data_smithi047_479866_object147:head# 2024-09-18T16:04:06.124 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:06.196 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c598be3a:::benchmark_data_smithi047_479866_object51:head# 2024-09-18T16:04:06.196 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:06.233 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c60c9d19:::benchmark_data_smithi047_474001_object146:head# 2024-09-18T16:04:06.233 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:06.266 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c6123b6f:::benchmark_data_smithi047_474001_object55:head# 2024-09-18T16:04:06.266 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:06.299 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c6869eae:::benchmark_data_smithi047_474001_object407:head# 2024-09-18T16:04:06.300 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:06.333 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c79249b8:::benchmark_data_smithi047_479866_object164:head# 2024-09-18T16:04:06.333 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:06.366 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c849831c:::benchmark_data_smithi047_474001_object118:head# 2024-09-18T16:04:06.366 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:06.400 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c960d8c2:::benchmark_data_smithi047_479866_object93:head# 2024-09-18T16:04:06.400 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:06.433 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c9eb3ea0:::benchmark_data_smithi047_474001_object87:head# 2024-09-18T16:04:06.433 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:06.466 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ca4e33a3:::benchmark_data_smithi047_474001_object283:head# 2024-09-18T16:04:06.467 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:06.523 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ca6f5c73:::benchmark_data_smithi047_479866_object52:head# 2024-09-18T16:04:06.523 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:06.606 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cb18044d:::benchmark_data_smithi047_479866_object165:head# 2024-09-18T16:04:06.606 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:06.658 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cb2f2b17:::benchmark_data_smithi047_479866_object114:head# 2024-09-18T16:04:06.658 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:06.700 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cb6747e8:::benchmark_data_smithi047_474001_object357:head# 2024-09-18T16:04:06.700 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:06.733 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cb84b5d4:::benchmark_data_smithi047_474001_object173:head# 2024-09-18T16:04:06.733 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:06.766 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cbdaedd2:::benchmark_data_smithi047_474001_object179:head# 2024-09-18T16:04:06.766 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:06.799 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cbef6232:::benchmark_data_smithi047_479866_object42:head# 2024-09-18T16:04:06.799 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:06.833 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cc244efe:::benchmark_data_smithi047_479866_object97:head# 2024-09-18T16:04:06.833 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:06.866 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cc26e7a6:::benchmark_data_smithi047_479866_object49:head# 2024-09-18T16:04:06.866 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:06.899 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cc8b0a99:::benchmark_data_smithi047_479866_object144:head# 2024-09-18T16:04:06.900 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:06.933 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cd271f33:::benchmark_data_smithi047_474001_object166:head# 2024-09-18T16:04:06.933 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:06.967 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cd28a341:::benchmark_data_smithi047_474001_object150:head# 2024-09-18T16:04:06.967 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:07.000 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cd4d6810:::benchmark_data_smithi047_474001_object363:head# 2024-09-18T16:04:07.000 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:07.033 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cd5cc2dc:::benchmark_data_smithi047_474001_object88:head# 2024-09-18T16:04:07.033 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:07.066 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cd77e455:::benchmark_data_smithi047_474001_object404:head# 2024-09-18T16:04:07.066 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:07.100 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cdedbf36:::benchmark_data_smithi047_474001_object155:head# 2024-09-18T16:04:07.100 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:07.133 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ceb22576:::benchmark_data_smithi047_479866_object79:head# 2024-09-18T16:04:07.133 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:07.166 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d007de6f:::benchmark_data_smithi047_474001_object151:head# 2024-09-18T16:04:07.166 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:07.200 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d00fd5c1:::benchmark_data_smithi047_479866_object55:head# 2024-09-18T16:04:07.200 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:07.233 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d02aa9a9:::benchmark_data_smithi047_479866_object123:head# 2024-09-18T16:04:07.236 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:07.266 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d275e6f0:::benchmark_data_smithi047_474001_object419:head# 2024-09-18T16:04:07.266 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:07.300 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d28bdb82:::benchmark_data_smithi047_479866_object154:head# 2024-09-18T16:04:07.300 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:07.333 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d295be0f:::benchmark_data_smithi047_474001_object167:head# 2024-09-18T16:04:07.333 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:07.366 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d2992f10:::benchmark_data_smithi047_474001_object175:head# 2024-09-18T16:04:07.367 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:07.400 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d2cdad48:::benchmark_data_smithi047_474001_object90:head# 2024-09-18T16:04:07.400 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:07.433 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d43afad7:::benchmark_data_smithi047_474001_object376:head# 2024-09-18T16:04:07.433 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:07.467 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d564afb6:::benchmark_data_smithi047_479866_object117:head# 2024-09-18T16:04:07.467 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:07.500 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d63d4966:::benchmark_data_smithi047_479866_object162:head# 2024-09-18T16:04:07.500 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:07.533 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d6762583:::benchmark_data_smithi047_474001_object15:head# 2024-09-18T16:04:07.533 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:07.567 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d6d0e63b:::benchmark_data_smithi047_474001_object117:head# 2024-09-18T16:04:07.567 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:07.600 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d74454f2:::benchmark_data_smithi047_474001_object191:head# 2024-09-18T16:04:07.600 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:07.633 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d793612f:::benchmark_data_smithi047_479866_object80:head# 2024-09-18T16:04:07.633 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:07.667 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d7ba857c:::benchmark_data_smithi047_474001_object160:head# 2024-09-18T16:04:07.667 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:07.700 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d7ca34da:::benchmark_data_smithi047_474001_object40:head# 2024-09-18T16:04:07.700 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:07.733 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d7f93f31:::benchmark_data_smithi047_479866_object153:head# 2024-09-18T16:04:07.734 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:07.767 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d8141741:::benchmark_data_smithi047_474001_object287:head# 2024-09-18T16:04:07.767 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:07.800 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:da6a8dc2:::benchmark_data_smithi047_474001_object18:head# 2024-09-18T16:04:07.800 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:07.825 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:da6c2c4d:::benchmark_data_smithi047_474001_object31:head# 2024-09-18T16:04:07.825 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:07.850 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:db2849da:::benchmark_data_smithi047_479866_object24:head# 2024-09-18T16:04:07.850 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:07.875 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:db2f6939:::benchmark_data_smithi047_474001_object400:head# 2024-09-18T16:04:07.875 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:07.900 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:dbe821c7:::benchmark_data_smithi047_479866_object57:head# 2024-09-18T16:04:07.900 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:07.925 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:dc4893ce:::benchmark_data_smithi047_474001_object232:head# 2024-09-18T16:04:07.925 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:07.950 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:dc6daefe:::benchmark_data_smithi047_474001_object276:head# 2024-09-18T16:04:07.950 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:07.975 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:dcdad940:::benchmark_data_smithi047_474001_object182:head# 2024-09-18T16:04:07.975 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:08.000 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:dcef5ff5:::benchmark_data_smithi047_479866_object102:head# 2024-09-18T16:04:08.000 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:08.033 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:dd4ec68f:::benchmark_data_smithi047_474001_object56:head# 2024-09-18T16:04:08.034 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:08.058 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ddc4bb3d:::benchmark_data_smithi047_474001_object199:head# 2024-09-18T16:04:08.059 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:08.083 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ddece86f:::benchmark_data_smithi047_474001_object43:head# 2024-09-18T16:04:08.083 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:08.108 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:de8db7e0:::benchmark_data_smithi047_479866_object34:head# 2024-09-18T16:04:08.109 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:08.163 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ded47609:::benchmark_data_smithi047_479866_object109:head# 2024-09-18T16:04:08.163 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:08.217 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:df65649c:::benchmark_data_smithi047_474001_object176:head# 2024-09-18T16:04:08.270 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:08.271 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:dfa0b551:::benchmark_data_smithi047_474001_object421:head# 2024-09-18T16:04:08.271 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:08.283 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e07f15b6:::benchmark_data_smithi047_474001_object10:head# 2024-09-18T16:04:08.283 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:08.317 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e1757a69:::benchmark_data_smithi047_474001_object366:head# 2024-09-18T16:04:08.317 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:08.350 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e1859c12:::benchmark_data_smithi047_474001_object102:head# 2024-09-18T16:04:08.350 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:08.383 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e1d3a554:::benchmark_data_smithi047_474001_object369:head# 2024-09-18T16:04:08.383 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:08.417 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e437b5d2:::benchmark_data_smithi047_474001_object377:head# 2024-09-18T16:04:08.417 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:08.450 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e445afb5:::benchmark_data_smithi047_474001_object355:head# 2024-09-18T16:04:08.450 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:08.483 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e45ac62c:::benchmark_data_smithi047_479866_object128:head# 2024-09-18T16:04:08.484 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:08.539 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e61a1975:::benchmark_data_smithi047_474001_object296:head# 2024-09-18T16:04:08.539 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:08.614 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e61d1049:::benchmark_data_smithi047_479866_object28:head# 2024-09-18T16:04:08.615 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:08.675 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e65db1db:::benchmark_data_smithi047_474001_object329:head# 2024-09-18T16:04:08.676 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:08.709 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e65e637a:::benchmark_data_smithi047_474001_object424:head# 2024-09-18T16:04:08.709 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:08.751 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e69ffe03:::benchmark_data_smithi047_479866_object156:head# 2024-09-18T16:04:08.751 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:08.784 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e6b693e8:::benchmark_data_smithi047_474001_object308:head# 2024-09-18T16:04:08.784 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:08.817 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e6d85c1d:::benchmark_data_smithi047_474001_object220:head# 2024-09-18T16:04:08.817 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:08.850 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e6f3dc67:::benchmark_data_smithi047_474001_object293:head# 2024-09-18T16:04:08.851 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:08.884 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e6f460e2:::benchmark_data_smithi047_474001_object130:head# 2024-09-18T16:04:08.884 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:08.917 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e6f5814f:::benchmark_data_smithi047_474001_object64:head# 2024-09-18T16:04:08.917 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:08.951 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e7a2c4e1:::benchmark_data_smithi047_474001_object94:head# 2024-09-18T16:04:08.951 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:08.984 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e8b890a6:::benchmark_data_smithi047_479866_object176:head# 2024-09-18T16:04:08.984 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:09.017 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e8de693f:::benchmark_data_smithi047_479866_object122:head# 2024-09-18T16:04:09.018 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:09.051 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e9129935:::benchmark_data_smithi047_479866_object17:head# 2024-09-18T16:04:09.051 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:09.084 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e921a909:::benchmark_data_smithi047_479866_object101:head# 2024-09-18T16:04:09.084 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:09.117 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e945e46a:::benchmark_data_smithi047_474001_object171:head# 2024-09-18T16:04:09.118 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:09.151 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e9b2d046:::benchmark_data_smithi047_479866_object18:head# 2024-09-18T16:04:09.151 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:09.184 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e9b3c7f8:::benchmark_data_smithi047_474001_object181:head# 2024-09-18T16:04:09.184 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:09.217 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e9bb35fb:::benchmark_data_smithi047_479866_object58:head# 2024-09-18T16:04:09.218 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:09.251 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e9bd9230:::benchmark_data_smithi047_479866_object15:head# 2024-09-18T16:04:09.251 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:09.284 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ea0dab4b:::benchmark_data_smithi047_474001_object406:head# 2024-09-18T16:04:09.284 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:09.317 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ea0dd8c2:::benchmark_data_smithi047_474001_object381:head# 2024-09-18T16:04:09.318 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:09.368 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ea20ab79:::benchmark_data_smithi047_474001_object178:head# 2024-09-18T16:04:09.368 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:09.401 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ea95d8c2:::benchmark_data_smithi047_474001_object238:head# 2024-09-18T16:04:09.401 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:09.434 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ecaeb389:::benchmark_data_smithi047_474001_object135:head# 2024-09-18T16:04:09.434 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:09.468 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ed1fe358:::benchmark_data_smithi047_479866_object106:head# 2024-09-18T16:04:09.468 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:09.501 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ed907482:::benchmark_data_smithi047_474001_object34:head# 2024-09-18T16:04:09.501 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:09.543 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ee898c6a:::benchmark_data_smithi047_474001_object184:head# 2024-09-18T16:04:09.543 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:09.584 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:eeceac30:::benchmark_data_smithi047_474001_object97:head# 2024-09-18T16:04:09.585 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:09.618 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:eee17d95:::benchmark_data_smithi047_474001_object187:head# 2024-09-18T16:04:09.618 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:09.651 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:eefc7324:::benchmark_data_smithi047_474001_object91:head# 2024-09-18T16:04:09.651 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:09.684 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ef61b0b9:::benchmark_data_smithi047_474001_object428:head# 2024-09-18T16:04:09.684 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:09.718 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:efbee6f8:::benchmark_data_smithi047_474001_object211:head# 2024-09-18T16:04:09.718 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:09.751 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:eff64798:::benchmark_data_smithi047_474001_object93:head# 2024-09-18T16:04:09.751 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:09.818 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f09ab14b:::benchmark_data_smithi047_474001_object339:head# 2024-09-18T16:04:09.818 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:09.851 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f09c2c79:::benchmark_data_smithi047_474001_object258:head# 2024-09-18T16:04:09.852 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:09.885 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f0a2b2e0:::benchmark_data_smithi047_474001_object234:head# 2024-09-18T16:04:09.885 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:09.918 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f0e21778:::benchmark_data_smithi047_474001_object210:head# 2024-09-18T16:04:09.918 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:09.951 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f12591b2:::benchmark_data_smithi047_474001_object142:head# 2024-09-18T16:04:09.951 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:09.984 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f15f0e45:::benchmark_data_smithi047_474001_object353:head# 2024-09-18T16:04:09.985 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:10.018 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f17effda:::benchmark_data_smithi047_474001_object121:head# 2024-09-18T16:04:10.018 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:10.051 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f188e48b:::benchmark_data_smithi047_474001_object82:head# 2024-09-18T16:04:10.051 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:10.085 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f286bf95:::benchmark_data_smithi047_474001_object115:head# 2024-09-18T16:04:10.085 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:10.118 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f400c3e0:::benchmark_data_smithi047_474001_object305:head# 2024-09-18T16:04:10.118 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:10.195 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f40d5282:::benchmark_data_smithi047_474001_object48:head# 2024-09-18T16:04:10.196 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:10.255 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f4d9af26:::benchmark_data_smithi047_474001_object311:head# 2024-09-18T16:04:10.255 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:10.330 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f50aa0e5:::benchmark_data_smithi047_474001_object65:head# 2024-09-18T16:04:10.330 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:10.380 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f58a2ed4:::benchmark_data_smithi047_474001_object393:head# 2024-09-18T16:04:10.380 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:10.448 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f5cd98c8:::benchmark_data_smithi047_479866_object14:head# 2024-09-18T16:04:10.448 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:10.522 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f61edbb2:::benchmark_data_smithi047_474001_object140:head# 2024-09-18T16:04:10.522 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:10.585 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f62ec792:::benchmark_data_smithi047_479866_object81:head# 2024-09-18T16:04:10.585 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:10.639 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f6596924:::benchmark_data_smithi047_474001_object256:head# 2024-09-18T16:04:10.640 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:10.702 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f68bfbc5:::benchmark_data_smithi047_474001_object295:head# 2024-09-18T16:04:10.702 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:10.735 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f7031fa9:::benchmark_data_smithi047_474001_object318:head# 2024-09-18T16:04:10.735 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:10.768 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f8576353:::benchmark_data_smithi047_474001_object336:head# 2024-09-18T16:04:10.769 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:10.801 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f87e20c7:::benchmark_data_smithi047_479866_object68:head# 2024-09-18T16:04:10.802 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:10.835 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f8e14a8e:::benchmark_data_smithi047_474001_object63:head# 2024-09-18T16:04:10.835 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:10.868 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f9e01ed3:::benchmark_data_smithi047_474001_object20:head# 2024-09-18T16:04:10.869 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:10.902 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fa676313:::benchmark_data_smithi047_474001_object268:head# 2024-09-18T16:04:10.902 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:10.935 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fb2704e8:::benchmark_data_smithi047_474001_object294:head# 2024-09-18T16:04:10.935 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:10.968 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fb557b5c:::benchmark_data_smithi047_474001_object25:head# 2024-09-18T16:04:10.968 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:11.001 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fba861de:::benchmark_data_smithi047_474001_object233:head# 2024-09-18T16:04:11.001 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:11.035 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fbc5f8c9:::benchmark_data_smithi047_479866_object75:head# 2024-09-18T16:04:11.035 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:11.068 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fc267536:::benchmark_data_smithi047_479866_object124:head# 2024-09-18T16:04:11.068 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:11.102 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fc48c5f5:::benchmark_data_smithi047_474001_object128:head# 2024-09-18T16:04:11.102 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:11.135 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fcc00bd9:::benchmark_data_smithi047_474001_object267:head# 2024-09-18T16:04:11.135 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:11.168 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fcc267e2:::benchmark_data_smithi047_474001_object420:head# 2024-09-18T16:04:11.169 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:11.202 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fcd59592:::benchmark_data_smithi047_474001_object387:head# 2024-09-18T16:04:11.202 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:11.235 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fcff1581:::benchmark_data_smithi047_474001_object229:head# 2024-09-18T16:04:11.235 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:11.268 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fd5c7c49:::benchmark_data_smithi047_474001_object375:head# 2024-09-18T16:04:11.269 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:11.302 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fdc8029d:::benchmark_data_smithi047_479866_object50:head# 2024-09-18T16:04:11.302 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:11.335 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fdd0ff05:::benchmark_data_smithi047_474001_object67:head# 2024-09-18T16:04:11.336 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:11.369 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fdf729e9:::benchmark_data_smithi047_474001_object230:head# 2024-09-18T16:04:11.369 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:11.402 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fe6bbe76:::benchmark_data_smithi047_479866_object61:head# 2024-09-18T16:04:11.402 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:11.435 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fe6c9d0e:::benchmark_data_smithi047_474001_object52:head# 2024-09-18T16:04:11.435 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:11.468 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fe938a07:::benchmark_data_smithi047_474001_object207:head# 2024-09-18T16:04:11.469 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:11.502 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fe96169b:::benchmark_data_smithi047_479866_object9:head# 2024-09-18T16:04:11.502 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:11.536 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fea2446c:::benchmark_data_smithi047_474001_object105:head# 2024-09-18T16:04:11.536 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:11.568 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:feb4dfe4:::benchmark_data_smithi047_479866_object152:head# 2024-09-18T16:04:11.569 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:11.602 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fee86c8a:::benchmark_data_smithi047_474001_object412:head# 2024-09-18T16:04:11.602 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:11.635 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ff03a3e6:::benchmark_data_smithi047_474001_object37:head# 2024-09-18T16:04:11.635 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:11.669 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ff401af6:::benchmark_data_smithi047_479866_object90:head# 2024-09-18T16:04:11.669 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:11.702 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ff8f6396:::benchmark_data_smithi047_474001_object114:head# 2024-09-18T16:04:11.702 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:04:11.735 INFO:tasks.workunit.client.0.smithi047.stdout:write_pg epoch 224 info 1.0( v 224'407 (9'229,224'407] local-lis/les=23/24 n=610 ec=7/7 lis/c=23/23 les/c/f=24/24/0 sis=23) 2024-09-18T16:04:11.769 INFO:tasks.workunit.client.0.smithi047.stdout:Import successful 2024-09-18T16:04:12.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:157: TEST_import_after_merge_and_gap: activate_osd td/pg-split-merge 0 2024-09-18T16:04:12.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-09-18T16:04:12.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T16:04:12.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T16:04:12.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T16:04:12.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/0 2024-09-18T16:04:12.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1a671e43-8866-44a5-abe3-c1e1f4ff765b --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-18T16:04:12.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:04:12.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:04:12.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:04:12.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-09-18T16:04:12.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-09-18T16:04:12.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T16:04:12.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T16:04:12.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-09-18T16:04:12.293 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T16:04:12.293 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:04:12.293 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:04:12.293 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:04:12.293 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:04:12.293 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.463129 2024-09-18T16:04:12.293 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.463129/$cluster-$name.asok' 2024-09-18T16:04:12.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.463129/$cluster-$name.asok' 2024-09-18T16:04:12.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:04:12.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-09-18T16:04:12.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-09-18T16:04:12.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:04:12.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:04:12.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:04:12.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:04:12.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T16:04:12.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T16:04:12.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/0 2024-09-18T16:04:12.294 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T16:04:12.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T16:04:12.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1a671e43-8866-44a5-abe3-c1e1f4ff765b --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.463129/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:04:12.295 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/0/whoami 2024-09-18T16:04:12.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T16:04:12.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T16:04:12.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T16:04:12.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T16:04:12.319 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:04:12.318+0000 7f6898b88440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:12.321 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:04:12.320+0000 7f6898b88440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:12.323 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:04:12.322+0000 7f6898b88440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:12.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T16:04:12.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:04:12.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T16:04:12.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:04:12.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:04:12.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:04:12.735 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:04:12.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:04:12.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:04:12.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:04:12.845 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:04:12.844+0000 7f6898b88440 -1 Falling back to public interface 2024-09-18T16:04:13.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:04:14.119 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:04:14.119 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T16:04:14.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:04:14.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:04:14.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:04:14.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:04:14.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:04:15.514 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:04:15.514 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:04:15.514 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T16:04:15.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:04:15.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:04:15.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:04:15.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:04:16.381 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:04:16.380+0000 7f6898b88440 -1 osd.0 228 log_to_monitors true 2024-09-18T16:04:16.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:04:16.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:04:16.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:04:16.882 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:04:16.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:04:16.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:04:17.269 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 231 up_thru 231 down_at 229 last_clean_interval [23,228) [v2:127.0.0.1:6802/483691,v1:127.0.0.1:6803/483691] [v2:127.0.0.1:6804/483691,v1:127.0.0.1:6805/483691] exists,up 1b6f27f6-2f00-44c7-8d93-1f3fa5ab4594 2024-09-18T16:04:17.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:04:17.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:04:17.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:04:17.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:159: TEST_import_after_merge_and_gap: wait_for_clean 2024-09-18T16:04:17.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:04:17.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:04:17.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:04:17.271 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:04:17.271 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:04:17.272 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:04:17.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:04:17.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:04:17.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:04:17.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T16:04:17.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:04:17.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:04:17.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:04:17.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:04:17.361 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:04:17.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T16:04:17.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:04:17.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:04:17.795 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:04:17.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=992137445379 2024-09-18T16:04:17.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 992137445379 2024-09-18T16:04:17.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-992137445379' 2024-09-18T16:04:17.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:04:17.905 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-992137445379 2024-09-18T16:04:17.905 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:04:17.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:04:17.906 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-992137445379 2024-09-18T16:04:17.906 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:04:17.907 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=992137445379 2024-09-18T16:04:17.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 992137445379' 2024-09-18T16:04:17.908 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 992137445379 2024-09-18T16:04:17.908 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:04:18.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 992137445379 -lt 992137445379 2024-09-18T16:04:18.305 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:04:18.305 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:04:18.305 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:04:18.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:04:18.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:04:18.803 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:04:18.803 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:04:18.803 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:04:18.803 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:04:18.803 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:04:18.803 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:04:19.185 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:04:19.185 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:04:19.185 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:04:19.186 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:04:19.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:04:19.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:04:19.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:04:19.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:17: run: teardown td/pg-split-merge 2024-09-18T16:04:19.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-09-18T16:04:19.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:04:19.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-09-18T16:04:19.718 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:04:19.718 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:04:19.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:04:19.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:04:19.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:04:19.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:04:19.823 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:04:19.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:04:19.824 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:04:19.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:04:19.825 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:04:19.826 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:04:19.826 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:04:19.826 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:04:19.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:04:19.827 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:04:19.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:04:19.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:04:19.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-09-18T16:04:19.844 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:04:19.845 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:04:19.845 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.463129 2024-09-18T16:04:19.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.463129 2024-09-18T16:04:19.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:04:19.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:04:19.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:14: run: for func in $funcs 2024-09-18T16:04:19.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:15: run: setup td/pg-split-merge 2024-09-18T16:04:19.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/pg-split-merge 2024-09-18T16:04:19.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/pg-split-merge 2024-09-18T16:04:19.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-09-18T16:04:19.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:04:19.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-09-18T16:04:19.847 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:04:19.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:04:19.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:04:19.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:04:19.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:04:19.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:04:19.848 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:04:19.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:04:19.849 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:04:19.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:04:19.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:04:19.850 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:04:19.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:04:19.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:04:19.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:04:19.852 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:04:19.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:04:19.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:04:19.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-09-18T16:04:19.854 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:04:19.855 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:04:19.855 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.463129 2024-09-18T16:04:19.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.463129 2024-09-18T16:04:19.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:04:19.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:04:19.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/pg-split-merge 2024-09-18T16:04:19.857 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T16:04:19.857 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:04:19.857 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.463129 2024-09-18T16:04:19.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.463129 2024-09-18T16:04:19.859 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T16:04:19.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T16:04:19.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T16:04:19.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/pg-split-merge 1' TERM HUP INT 2024-09-18T16:04:19.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:16: run: TEST_import_after_split td/pg-split-merge 2024-09-18T16:04:19.860 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:163: TEST_import_after_split: local dir=td/pg-split-merge 2024-09-18T16:04:19.860 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:165: TEST_import_after_split: run_mon td/pg-split-merge a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T16:04:19.860 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/pg-split-merge 2024-09-18T16:04:19.860 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T16:04:19.860 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T16:04:19.860 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T16:04:19.860 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/pg-split-merge/a 2024-09-18T16:04:19.860 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/pg-split-merge/a --run-dir=td/pg-split-merge --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T16:04:20.211 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T16:04:20.211 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:04:20.211 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:04:20.211 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:04:20.211 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:04:20.211 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.463129 2024-09-18T16:04:20.211 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.463129/$cluster-$name.asok' 2024-09-18T16:04:20.212 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/pg-split-merge/a '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.463129/$cluster-$name.asok' --mon-cluster-log-file=td/pg-split-merge/log --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-18T16:04:20.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T16:04:20.472 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T16:04:20.472 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:04:20.472 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:04:20.472 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T16:04:20.472 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T16:04:20.472 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:04:20.472 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:04:20.472 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:04:20.473 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:04:20.473 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:04:20.473 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.463129 2024-09-18T16:04:20.473 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.463129/ceph-mon.a.asok 2024-09-18T16:04:20.473 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:04:20.474 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.463129/ceph-mon.a.asok config get fsid 2024-09-18T16:04:20.554 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T16:04:20.555 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:04:20.555 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:04:20.555 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T16:04:20.555 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T16:04:20.555 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:04:20.555 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:04:20.555 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:04:20.555 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:04:20.555 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:04:20.555 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.463129 2024-09-18T16:04:20.555 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.463129/ceph-mon.a.asok 2024-09-18T16:04:20.556 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:04:20.556 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.463129/ceph-mon.a.asok config get mon_host 2024-09-18T16:04:20.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:166: TEST_import_after_split: run_mgr td/pg-split-merge x 2024-09-18T16:04:20.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/pg-split-merge 2024-09-18T16:04:20.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T16:04:20.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T16:04:20.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T16:04:20.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/pg-split-merge/x 2024-09-18T16:04:20.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T16:04:21.476 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T16:04:21.476 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:04:21.476 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:04:21.476 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:04:21.476 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:04:21.476 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.463129 2024-09-18T16:04:21.476 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.463129/$cluster-$name.asok' 2024-09-18T16:04:21.476 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:04:21.477 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/pg-split-merge/x '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.463129/$cluster-$name.asok' --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:04:21.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:167: TEST_import_after_split: run_osd td/pg-split-merge 0 2024-09-18T16:04:21.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/pg-split-merge 2024-09-18T16:04:21.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:04:21.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T16:04:21.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:04:21.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/pg-split-merge/0 2024-09-18T16:04:21.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a671e43-8866-44a5-abe3-c1e1f4ff765b --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-18T16:04:21.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:04:21.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:04:21.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:04:21.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-09-18T16:04:21.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-09-18T16:04:21.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:04:21.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:04:21.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-09-18T16:04:21.505 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:04:21.505 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:04:21.505 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:04:21.505 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:04:21.505 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:04:21.505 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.463129 2024-09-18T16:04:21.505 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.463129/$cluster-$name.asok' 2024-09-18T16:04:21.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.463129/$cluster-$name.asok' 2024-09-18T16:04:21.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:04:21.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:04:21.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:04:21.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-09-18T16:04:21.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-09-18T16:04:21.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:04:21.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:04:21.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:04:21.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:04:21.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:04:21.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:04:21.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/pg-split-merge/0 2024-09-18T16:04:21.507 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:04:21.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=be7e9a0f-d00f-4cf4-aaef-94da8b714cda 2024-09-18T16:04:21.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 be7e9a0f-d00f-4cf4-aaef-94da8b714cda' 2024-09-18T16:04:21.507 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 be7e9a0f-d00f-4cf4-aaef-94da8b714cda 2024-09-18T16:04:21.507 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:04:21.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAF+upmHoIBHxAACFnmjU94LsdKnS/I4kWjAw== 2024-09-18T16:04:21.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAF+upmHoIBHxAACFnmjU94LsdKnS/I4kWjAw=="}' 2024-09-18T16:04:21.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new be7e9a0f-d00f-4cf4-aaef-94da8b714cda -i td/pg-split-merge/0/new.json 2024-09-18T16:04:21.828 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:04:21.837 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/pg-split-merge/0/new.json 2024-09-18T16:04:21.838 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1a671e43-8866-44a5-abe3-c1e1f4ff765b --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.463129/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAF+upmHoIBHxAACFnmjU94LsdKnS/I4kWjAw== --osd-uuid be7e9a0f-d00f-4cf4-aaef-94da8b714cda 2024-09-18T16:04:21.865 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:04:21.864+0000 7f6bc3b23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:21.872 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:04:21.871+0000 7f6bc3b23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:21.873 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:04:21.872+0000 7f6bc3b23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:21.874 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:04:21.872+0000 7f6bc3b23440 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-09-18T16:04:21.874 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:04:21.872+0000 7f6bc3b23440 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-09-18T16:04:21.874 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:04:21.872+0000 7f6bc3b23440 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-09-18T16:04:21.912 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:04:21.911+0000 7f6bc3b23440 -1 bluestore(td/pg-split-merge/0) _read_fsid unparsable uuid 2024-09-18T16:04:27.242 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/pg-split-merge/0/keyring 2024-09-18T16:04:27.242 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:04:27.243 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T16:04:27.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T16:04:27.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/pg-split-merge/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:04:27.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T16:04:27.747 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T16:04:27.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1a671e43-8866-44a5-abe3-c1e1f4ff765b --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.463129/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:04:27.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:04:27.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:04:27.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:04:27.772 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:04:27.771+0000 7f2250809440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:27.794 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:04:27.793+0000 7f2250809440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:27.795 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:04:27.794+0000 7f2250809440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:04:28.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T16:04:28.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:04:28.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T16:04:28.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:04:28.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:04:28.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:04:28.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:04:28.200 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:04:28.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:04:28.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:04:28.328 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:04:28.327+0000 7f2250809440 -1 Falling back to public interface 2024-09-18T16:04:28.564 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:04:29.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:04:29.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:04:29.566 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T16:04:29.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:04:29.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:04:29.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:04:29.985 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:04:30.816 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:04:30.815+0000 7f2250809440 -1 osd.0 0 log_to_monitors true 2024-09-18T16:04:30.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:04:30.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:04:30.986 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T16:04:30.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:04:30.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:04:30.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:04:31.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:04:32.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:04:32.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:04:32.465 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:04:32.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:04:32.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:04:32.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:04:32.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:04:33.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:04:33.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:04:33.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:04:33.885 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T16:04:33.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:04:33.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:04:34.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:04:35.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:04:35.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:04:35.272 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T16:04:35.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:04:35.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:04:35.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:04:35.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:04:36.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:04:36.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:04:36.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:04:36.702 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T16:04:36.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:04:36.703 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:04:37.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:04:38.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:04:38.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:04:38.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:04:38.089 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T16:04:38.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:04:38.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:04:38.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:04:39.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:04:39.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:04:39.638 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T16:04:39.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:04:39.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:04:39.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:04:40.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:04:41.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:04:41.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:04:41.020 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T16:04:41.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:04:41.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:04:41.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:04:41.491 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:04:42.492 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:04:42.492 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:04:42.492 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:04:42.493 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T16:04:42.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:04:42.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:04:43.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:04:44.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:04:44.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:04:44.145 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T16:04:44.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:04:44.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:04:44.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:04:44.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:04:45.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:04:45.620 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:04:45.621 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T16:04:45.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:04:45.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:04:45.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:04:46.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:04:47.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:04:47.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:04:47.511 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T16:04:47.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:04:47.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:04:47.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:04:48.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:04:49.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:04:49.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:04:49.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:04:49.166 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T16:04:49.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:04:49.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:04:49.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:04:50.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:04:50.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:04:50.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:04:50.928 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T16:04:50.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:04:50.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:04:51.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:04:52.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:04:52.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:04:52.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:04:52.449 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T16:04:52.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:04:52.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:04:52.915 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/485565,v1:127.0.0.1:6803/485565] [v2:127.0.0.1:6804/485565,v1:127.0.0.1:6805/485565] exists,up be7e9a0f-d00f-4cf4-aaef-94da8b714cda 2024-09-18T16:04:52.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:04:52.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:04:52.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:04:52.916 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:169: TEST_import_after_split: ceph osd pool create foo 1 2024-09-18T16:04:53.605 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'foo' already exists 2024-09-18T16:04:53.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:170: TEST_import_after_split: wait_for_clean 2024-09-18T16:04:53.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:04:53.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:04:53.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:04:53.617 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:04:53.618 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:04:53.618 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:04:53.618 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:04:53.618 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:04:53.618 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:04:53.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T16:04:53.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:04:53.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:04:53.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:04:53.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:04:53.707 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:04:54.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T16:04:54.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:04:54.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:04:54.078 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:04:54.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836490 2024-09-18T16:04:54.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836490 2024-09-18T16:04:54.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836490' 2024-09-18T16:04:54.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:04:54.227 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836490 2024-09-18T16:04:54.227 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:04:54.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:04:54.228 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836490 2024-09-18T16:04:54.228 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:04:54.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836490 2024-09-18T16:04:54.229 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2024-09-18T16:04:54.229 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836490 2024-09-18T16:04:54.230 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:04:54.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 0 -lt 21474836490 2024-09-18T16:04:54.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:04:55.638 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:04:55.638 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:04:56.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836490 -lt 21474836490 2024-09-18T16:04:56.015 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:04:56.015 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:04:56.015 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:04:56.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:04:56.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:04:56.508 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:04:56.508 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:04:56.508 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:04:56.508 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:04:56.508 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:04:56.508 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:04:56.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:04:56.893 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:04:56.893 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:04:56.893 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:04:57.390 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:04:57.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:04:57.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:04:57.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:171: TEST_import_after_split: rados -p foo bench 3 write -b 1024 --no-cleanup 2024-09-18T16:04:57.425 INFO:tasks.workunit.client.0.smithi047.stdout:hints = 1 2024-09-18T16:04:57.426 INFO:tasks.workunit.client.0.smithi047.stdout:Maintaining 16 concurrent writes of 1024 bytes to objects of size 1024 for up to 3 seconds or 0 objects 2024-09-18T16:04:57.426 INFO:tasks.workunit.client.0.smithi047.stdout:Object prefix: benchmark_data_smithi047_486709 2024-09-18T16:04:57.426 INFO:tasks.workunit.client.0.smithi047.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T16:04:57.426 INFO:tasks.workunit.client.0.smithi047.stdout: 0 0 0 0 0 0 - 0 2024-09-18T16:04:58.425 INFO:tasks.workunit.client.0.smithi047.stdout: 1 16 160 144 0.140612 0.140625 0.100753 0.10828 2024-09-18T16:04:59.425 INFO:tasks.workunit.client.0.smithi047.stdout: 2 16 305 289 0.141098 0.141602 0.117199 0.107653 2024-09-18T16:05:00.425 INFO:tasks.workunit.client.0.smithi047.stdout: 3 16 448 432 0.140608 0.139648 0.100636 0.109639 2024-09-18T16:05:01.426 INFO:tasks.workunit.client.0.smithi047.stdout:Total time run: 3.0642 2024-09-18T16:05:01.426 INFO:tasks.workunit.client.0.smithi047.stdout:Total writes made: 448 2024-09-18T16:05:01.426 INFO:tasks.workunit.client.0.smithi047.stdout:Write size: 1024 2024-09-18T16:05:01.426 INFO:tasks.workunit.client.0.smithi047.stdout:Object size: 1024 2024-09-18T16:05:01.427 INFO:tasks.workunit.client.0.smithi047.stdout:Bandwidth (MB/sec): 0.142778 2024-09-18T16:05:01.427 INFO:tasks.workunit.client.0.smithi047.stdout:Stddev Bandwidth: 0.000976562 2024-09-18T16:05:01.427 INFO:tasks.workunit.client.0.smithi047.stdout:Max bandwidth (MB/sec): 0.141602 2024-09-18T16:05:01.427 INFO:tasks.workunit.client.0.smithi047.stdout:Min bandwidth (MB/sec): 0.139648 2024-09-18T16:05:01.427 INFO:tasks.workunit.client.0.smithi047.stdout:Average IOPS: 146 2024-09-18T16:05:01.427 INFO:tasks.workunit.client.0.smithi047.stdout:Stddev IOPS: 1 2024-09-18T16:05:01.427 INFO:tasks.workunit.client.0.smithi047.stdout:Max IOPS: 145 2024-09-18T16:05:01.427 INFO:tasks.workunit.client.0.smithi047.stdout:Min IOPS: 143 2024-09-18T16:05:01.427 INFO:tasks.workunit.client.0.smithi047.stdout:Average Latency(s): 0.10931 2024-09-18T16:05:01.427 INFO:tasks.workunit.client.0.smithi047.stdout:Stddev Latency(s): 0.0258529 2024-09-18T16:05:01.427 INFO:tasks.workunit.client.0.smithi047.stdout:Max latency(s): 0.192778 2024-09-18T16:05:01.427 INFO:tasks.workunit.client.0.smithi047.stdout:Min latency(s): 0.0838024 2024-09-18T16:05:01.483 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:173: TEST_import_after_split: kill_daemons td/pg-split-merge TERM osd.0 2024-09-18T16:05:01.484 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:05:01.484 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:05:01.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:05:01.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:05:01.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:05:01.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:05:01.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:174: TEST_import_after_split: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op export --pgid 1.0 --file td/pg-split-merge/1.0 --force 2024-09-18T16:05:02.946 INFO:tasks.workunit.client.0.smithi047.stderr:Exporting 1.0 info 1.0( v 8'449 (0'0,8'449] local-lis/les=6/7 n=449 ec=6/6 lis/c=6/6 les/c/f=7/7/0 sis=6) 2024-09-18T16:05:02.950 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:00455b6d:::benchmark_data_smithi047_486709_object58:head# 2024-09-18T16:05:02.950 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:00bb5925:::benchmark_data_smithi047_486709_object299:head# 2024-09-18T16:05:02.950 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0190b0e9:::benchmark_data_smithi047_486709_object10:head# 2024-09-18T16:05:02.951 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:01b14c7b:::benchmark_data_smithi047_486709_object118:head# 2024-09-18T16:05:02.951 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:01f804ce:::benchmark_data_smithi047_486709_object394:head# 2024-09-18T16:05:02.952 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:04421016:::benchmark_data_smithi047_486709_object20:head# 2024-09-18T16:05:02.952 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:04ecef8d:::benchmark_data_smithi047_486709_object150:head# 2024-09-18T16:05:02.952 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:04f9a10f:::benchmark_data_smithi047_486709_object44:head# 2024-09-18T16:05:02.953 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:06d460d4:::benchmark_data_smithi047_486709_object370:head# 2024-09-18T16:05:02.953 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0709c473:::benchmark_data_smithi047_486709_object249:head# 2024-09-18T16:05:02.954 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:08e3c220:::benchmark_data_smithi047_486709_object292:head# 2024-09-18T16:05:02.954 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0a74bda8:::benchmark_data_smithi047_486709_object239:head# 2024-09-18T16:05:02.955 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0d17e7da:::benchmark_data_smithi047_486709_object78:head# 2024-09-18T16:05:02.955 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0e62fc90:::benchmark_data_smithi047_486709_object350:head# 2024-09-18T16:05:02.956 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:0eb0a557:::benchmark_data_smithi047_486709_object116:head# 2024-09-18T16:05:02.956 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1112f9ed:::benchmark_data_smithi047_486709_object420:head# 2024-09-18T16:05:02.957 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:11368039:::benchmark_data_smithi047_486709_object311:head# 2024-09-18T16:05:02.962 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:11d92add:::benchmark_data_smithi047_486709_object269:head# 2024-09-18T16:05:02.962 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1210e9b6:::benchmark_data_smithi047_486709_object197:head# 2024-09-18T16:05:02.962 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:147965e0:::benchmark_data_smithi047_486709_object251:head# 2024-09-18T16:05:02.963 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:148835e5:::benchmark_data_smithi047_486709_object139:head# 2024-09-18T16:05:02.963 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:155430aa:::benchmark_data_smithi047_486709_object447:head# 2024-09-18T16:05:02.965 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:15fc55cf:::benchmark_data_smithi047_486709_object345:head# 2024-09-18T16:05:02.965 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:172bd864:::benchmark_data_smithi047_486709_object115:head# 2024-09-18T16:05:02.974 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:17700f86:::benchmark_data_smithi047_486709_object181:head# 2024-09-18T16:05:02.976 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:17bc617d:::benchmark_data_smithi047_486709_object352:head# 2024-09-18T16:05:02.976 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:18238136:::benchmark_data_smithi047_486709_object228:head# 2024-09-18T16:05:02.977 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:184a6b0e:::benchmark_data_smithi047_486709_object434:head# 2024-09-18T16:05:02.977 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:186e5996:::benchmark_data_smithi047_486709_object303:head# 2024-09-18T16:05:02.978 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1889d760:::benchmark_data_smithi047_486709_object106:head# 2024-09-18T16:05:02.978 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:190aee5c:::benchmark_data_smithi047_486709_object2:head# 2024-09-18T16:05:02.988 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:19e2fb60:::benchmark_data_smithi047_486709_object51:head# 2024-09-18T16:05:02.989 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1a1c1df2:::benchmark_data_smithi047_486709_object244:head# 2024-09-18T16:05:02.990 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1a84ad1c:::benchmark_data_smithi047_486709_object216:head# 2024-09-18T16:05:02.993 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1c29bfaa:::benchmark_data_smithi047_486709_object76:head# 2024-09-18T16:05:02.993 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1db7bb6b:::benchmark_data_smithi047_486709_object117:head# 2024-09-18T16:05:02.994 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1ddafb67:::benchmark_data_smithi047_486709_object357:head# 2024-09-18T16:05:02.996 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1e263654:::benchmark_data_smithi047_486709_object124:head# 2024-09-18T16:05:02.996 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1ee7ec61:::benchmark_data_smithi047_486709_object22:head# 2024-09-18T16:05:02.996 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1f89eb4d:::benchmark_data_smithi047_486709_object71:head# 2024-09-18T16:05:02.997 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1fca7266:::benchmark_data_smithi047_486709_object236:head# 2024-09-18T16:05:02.997 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:1fce0a50:::benchmark_data_smithi047_486709_object141:head# 2024-09-18T16:05:02.998 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2019af5e:::benchmark_data_smithi047_486709_object132:head# 2024-09-18T16:05:02.998 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:208fc203:::benchmark_data_smithi047_486709_object298:head# 2024-09-18T16:05:02.999 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:20dfb558:::benchmark_data_smithi047_486709_object397:head# 2024-09-18T16:05:03.000 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:21fdd2cb:::benchmark_data_smithi047_486709_object391:head# 2024-09-18T16:05:03.000 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:23900c46:::benchmark_data_smithi047_486709_object103:head# 2024-09-18T16:05:03.000 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:23f7f590:::benchmark_data_smithi047_486709_object390:head# 2024-09-18T16:05:03.001 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2428039f:::benchmark_data_smithi047_486709_object178:head# 2024-09-18T16:05:03.001 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:24bb594b:::benchmark_data_smithi047_486709_object422:head# 2024-09-18T16:05:03.002 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:24ca8f82:::benchmark_data_smithi047_486709_object206:head# 2024-09-18T16:05:03.002 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:25e9f491:::benchmark_data_smithi047_486709_object220:head# 2024-09-18T16:05:03.003 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:25f093aa:::benchmark_data_smithi047_486709_object316:head# 2024-09-18T16:05:03.003 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:261b1b9f:::benchmark_data_smithi047_486709_object192:head# 2024-09-18T16:05:03.004 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:263bb008:::benchmark_data_smithi047_486709_object366:head# 2024-09-18T16:05:03.004 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:26c2dcf5:::benchmark_data_smithi047_486709_object395:head# 2024-09-18T16:05:03.005 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:28588a22:::benchmark_data_smithi047_486709_object421:head# 2024-09-18T16:05:03.006 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:291e913c:::benchmark_data_smithi047_486709_object323:head# 2024-09-18T16:05:03.006 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:297d98c0:::benchmark_data_smithi047_486709_object225:head# 2024-09-18T16:05:03.006 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2ae0ba07:::benchmark_data_smithi047_486709_object113:head# 2024-09-18T16:05:03.007 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2b0ac3f3:::benchmark_data_smithi047_486709_object407:head# 2024-09-18T16:05:03.008 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2b922108:::benchmark_data_smithi047_486709_object425:head# 2024-09-18T16:05:03.008 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2bb192a9:::benchmark_data_smithi047_486709_object42:head# 2024-09-18T16:05:03.008 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2bec4599:::benchmark_data_smithi047_486709_object296:head# 2024-09-18T16:05:03.009 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2c95f113:::benchmark_data_smithi047_486709_object146:head# 2024-09-18T16:05:03.009 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2cc4228c:::benchmark_data_smithi047_486709_object43:head# 2024-09-18T16:05:03.010 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2e0b5e21:::benchmark_data_smithi047_486709_object315:head# 2024-09-18T16:05:03.010 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2ec2cc08:::benchmark_data_smithi047_486709_object53:head# 2024-09-18T16:05:03.011 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2f1c11a8:::benchmark_data_smithi047_486709_object30:head# 2024-09-18T16:05:03.011 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:2f5f8553:::benchmark_data_smithi047_486709_object111:head# 2024-09-18T16:05:03.011 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:303347ff:::benchmark_data_smithi047_486709_object405:head# 2024-09-18T16:05:03.011 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:30c94f4d:::benchmark_data_smithi047_486709_object3:head# 2024-09-18T16:05:03.013 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:31cd2170:::benchmark_data_smithi047_486709_object302:head# 2024-09-18T16:05:03.014 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:31d8df38:::benchmark_data_smithi047_486709_object312:head# 2024-09-18T16:05:03.014 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:325fc1d4:::benchmark_data_smithi047_486709_object144:head# 2024-09-18T16:05:03.016 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:326bd0be:::benchmark_data_smithi047_486709_object293:head# 2024-09-18T16:05:03.017 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:33e7e262:::benchmark_data_smithi047_486709_object33:head# 2024-09-18T16:05:03.017 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:345a13c5:::benchmark_data_smithi047_486709_object436:head# 2024-09-18T16:05:03.018 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:345ca073:::benchmark_data_smithi047_486709_object322:head# 2024-09-18T16:05:03.019 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:34f0cac8:::benchmark_data_smithi047_486709_object134:head# 2024-09-18T16:05:03.019 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3520db44:::benchmark_data_smithi047_486709_object272:head# 2024-09-18T16:05:03.020 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:358928ae:::benchmark_data_smithi047_486709_object91:head# 2024-09-18T16:05:03.020 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3594212f:::benchmark_data_smithi047_486709_object148:head# 2024-09-18T16:05:03.021 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:37e99f13:::benchmark_data_smithi047_486709_object156:head# 2024-09-18T16:05:03.021 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:380f7c7b:::benchmark_data_smithi047_486709_object0:head# 2024-09-18T16:05:03.038 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:38102a61:::benchmark_data_smithi047_486709_object413:head# 2024-09-18T16:05:03.039 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3844bb99:::benchmark_data_smithi047_486709_object138:head# 2024-09-18T16:05:03.041 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:38ba1613:::benchmark_data_smithi047_486709_object90:head# 2024-09-18T16:05:03.041 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:38be71f8:::benchmark_data_smithi047_486709_object329:head# 2024-09-18T16:05:03.042 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:39341ae8:::benchmark_data_smithi047_486709_object365:head# 2024-09-18T16:05:03.043 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:39839f11:::benchmark_data_smithi047_486709_object151:head# 2024-09-18T16:05:03.043 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:398591fd:::benchmark_data_smithi047_486709_object218:head# 2024-09-18T16:05:03.044 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:39f2ac8a:::benchmark_data_smithi047_486709_object82:head# 2024-09-18T16:05:03.044 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3a9ed21e:::benchmark_data_smithi047_486709_object284:head# 2024-09-18T16:05:03.045 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3acde6b6:::benchmark_data_smithi047_486709_object237:head# 2024-09-18T16:05:03.045 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3b3a14d6:::benchmark_data_smithi047_486709_object125:head# 2024-09-18T16:05:03.046 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3c0f53ee:::benchmark_data_smithi047_486709_object330:head# 2024-09-18T16:05:03.046 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3c4d3f0b:::benchmark_data_smithi047_486709_object268:head# 2024-09-18T16:05:03.046 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3c92443c:::benchmark_data_smithi047_486709_object364:head# 2024-09-18T16:05:03.047 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3e04c091:::benchmark_data_smithi047_486709_object238:head# 2024-09-18T16:05:03.047 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3fbd99a7:::benchmark_data_smithi047_486709_object376:head# 2024-09-18T16:05:03.047 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3fd976ca:::benchmark_data_smithi047_486709_object427:head# 2024-09-18T16:05:03.047 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:3ffbf4f7:::benchmark_data_smithi047_486709_object207:head# 2024-09-18T16:05:03.049 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:40498dc8:::benchmark_data_smithi047_486709_object297:head# 2024-09-18T16:05:03.049 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:407f5073:::benchmark_data_smithi047_486709_object320:head# 2024-09-18T16:05:03.049 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:42999bca:::benchmark_data_smithi047_486709_object286:head# 2024-09-18T16:05:03.049 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:42a09cc6:::benchmark_data_smithi047_486709_object325:head# 2024-09-18T16:05:03.049 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:430b1235:::benchmark_data_smithi047_486709_object372:head# 2024-09-18T16:05:03.049 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:445355c8:::benchmark_data_smithi047_486709_object69:head# 2024-09-18T16:05:03.049 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:446b192a:::benchmark_data_smithi047_486709_object347:head# 2024-09-18T16:05:03.050 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:448c4d23:::benchmark_data_smithi047_486709_object255:head# 2024-09-18T16:05:03.051 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:44af3a1c:::benchmark_data_smithi047_486709_object143:head# 2024-09-18T16:05:03.051 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4568b959:::benchmark_data_smithi047_486709_object300:head# 2024-09-18T16:05:03.052 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:456fd22d:::benchmark_data_smithi047_486709_object36:head# 2024-09-18T16:05:03.052 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:45940b80:::benchmark_data_smithi047_486709_object160:head# 2024-09-18T16:05:03.052 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:461042d3:::benchmark_data_smithi047_486709_object254:head# 2024-09-18T16:05:03.053 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4659966f:::benchmark_data_smithi047_486709_object47:head# 2024-09-18T16:05:03.053 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:46f59358:::benchmark_data_smithi047_486709_object201:head# 2024-09-18T16:05:03.054 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:475b1b2b:::benchmark_data_smithi047_486709_object356:head# 2024-09-18T16:05:03.054 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:478b477a:::benchmark_data_smithi047_486709_object136:head# 2024-09-18T16:05:03.055 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:478dfdf5:::benchmark_data_smithi047_486709_object429:head# 2024-09-18T16:05:03.055 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:47a5884a:::benchmark_data_smithi047_486709_object433:head# 2024-09-18T16:05:03.055 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:47a5b05d:::benchmark_data_smithi047_486709_object416:head# 2024-09-18T16:05:03.056 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:47a5d972:::benchmark_data_smithi047_486709_object280:head# 2024-09-18T16:05:03.056 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:47dc2109:::benchmark_data_smithi047_486709_object445:head# 2024-09-18T16:05:03.057 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4805108f:::benchmark_data_smithi047_486709_object259:head# 2024-09-18T16:05:03.058 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:481be0d2:::benchmark_data_smithi047_486709_object52:head# 2024-09-18T16:05:03.058 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4868c009:::benchmark_data_smithi047_486709_object417:head# 2024-09-18T16:05:03.058 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:48882074:::benchmark_data_smithi047_486709_object401:head# 2024-09-18T16:05:03.059 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:494851e1:::benchmark_data_smithi047_486709_object75:head# 2024-09-18T16:05:03.059 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:494d1c4c:::benchmark_data_smithi047_486709_object336:head# 2024-09-18T16:05:03.059 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:49886492:::benchmark_data_smithi047_486709_object70:head# 2024-09-18T16:05:03.060 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4a0bc2c3:::benchmark_data_smithi047_486709_object313:head# 2024-09-18T16:05:03.060 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4a34e00d:::benchmark_data_smithi047_486709_object290:head# 2024-09-18T16:05:03.061 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4a3daa8b:::benchmark_data_smithi047_486709_object343:head# 2024-09-18T16:05:03.061 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4acf9792:::benchmark_data_smithi047_486709_object414:head# 2024-09-18T16:05:03.062 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4b3808de:::benchmark_data_smithi047_486709_object404:head# 2024-09-18T16:05:03.062 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4b6c4826:::benchmark_data_smithi047_486709_object355:head# 2024-09-18T16:05:03.062 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4ccc2f76:::benchmark_data_smithi047_486709_object349:head# 2024-09-18T16:05:03.062 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4cd1475e:::benchmark_data_smithi047_486709_object318:head# 2024-09-18T16:05:03.063 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4cd3eb30:::benchmark_data_smithi047_486709_object408:head# 2024-09-18T16:05:03.063 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4d8c261c:::benchmark_data_smithi047_486709_object398:head# 2024-09-18T16:05:03.063 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4df885cf:::benchmark_data_smithi047_486709_object359:head# 2024-09-18T16:05:03.063 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4ea03f52:::benchmark_data_smithi047_486709_object165:head# 2024-09-18T16:05:03.064 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:4f47b96f:::benchmark_data_smithi047_486709_object162:head# 2024-09-18T16:05:03.065 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5028023b:::benchmark_data_smithi047_486709_object96:head# 2024-09-18T16:05:03.065 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:504aec17:::benchmark_data_smithi047_486709_object209:head# 2024-09-18T16:05:03.065 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:50ee368e:::benchmark_data_smithi047_486709_object393:head# 2024-09-18T16:05:03.066 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:50fe9e36:::benchmark_data_smithi047_486709_object63:head# 2024-09-18T16:05:03.066 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:51596afe:::benchmark_data_smithi047_486709_object15:head# 2024-09-18T16:05:03.066 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5252b7a2:::benchmark_data_smithi047_486709_object140:head# 2024-09-18T16:05:03.067 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5255f4d2:::benchmark_data_smithi047_486709_object31:head# 2024-09-18T16:05:03.067 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:53c861f9:::benchmark_data_smithi047_486709_object25:head# 2024-09-18T16:05:03.067 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:540388ee:::benchmark_data_smithi047_486709_object66:head# 2024-09-18T16:05:03.068 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:54d23911:::benchmark_data_smithi047_486709_object227:head# 2024-09-18T16:05:03.068 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:54da63c3:::benchmark_data_smithi047_486709_object109:head# 2024-09-18T16:05:03.069 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5634e19f:::benchmark_data_smithi047_486709_object171:head# 2024-09-18T16:05:03.069 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5764dce9:::benchmark_data_smithi047_486709_object4:head# 2024-09-18T16:05:03.070 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:57985501:::benchmark_data_smithi047_486709_object152:head# 2024-09-18T16:05:03.070 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:57a66a66:::benchmark_data_smithi047_486709_object88:head# 2024-09-18T16:05:03.070 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:59358cf8:::benchmark_data_smithi047_486709_object431:head# 2024-09-18T16:05:03.071 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5a56ed48:::benchmark_data_smithi047_486709_object252:head# 2024-09-18T16:05:03.071 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5d6a3e70:::benchmark_data_smithi047_486709_object198:head# 2024-09-18T16:05:03.072 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5dc17652:::benchmark_data_smithi047_486709_object155:head# 2024-09-18T16:05:03.072 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5dde6884:::benchmark_data_smithi047_486709_object435:head# 2024-09-18T16:05:03.072 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5e6ea9b6:::benchmark_data_smithi047_486709_object231:head# 2024-09-18T16:05:03.072 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5ecfb9c6:::benchmark_data_smithi047_486709_object385:head# 2024-09-18T16:05:03.073 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5f39bf22:::benchmark_data_smithi047_486709_object161:head# 2024-09-18T16:05:03.074 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:5f914756:::benchmark_data_smithi047_486709_object363:head# 2024-09-18T16:05:03.074 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:603081ee:::benchmark_data_smithi047_486709_object386:head# 2024-09-18T16:05:03.074 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:61695f17:::benchmark_data_smithi047_486709_object276:head# 2024-09-18T16:05:03.075 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:61896787:::benchmark_data_smithi047_486709_object210:head# 2024-09-18T16:05:03.075 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6200b7be:::benchmark_data_smithi047_486709_object232:head# 2024-09-18T16:05:03.076 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:63dd2c56:::benchmark_data_smithi047_486709_object59:head# 2024-09-18T16:05:03.076 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:63f8e762:::benchmark_data_smithi047_486709_object189:head# 2024-09-18T16:05:03.077 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:64b920e4:::benchmark_data_smithi047_486709_object212:head# 2024-09-18T16:05:03.077 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:662e1d13:::benchmark_data_smithi047_486709_object145:head# 2024-09-18T16:05:03.078 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:66b0b6b6:::benchmark_data_smithi047_486709_object443:head# 2024-09-18T16:05:03.078 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:674a1646:::benchmark_data_smithi047_486709_object46:head# 2024-09-18T16:05:03.079 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:67796b68:::benchmark_data_smithi047_486709_object444:head# 2024-09-18T16:05:03.079 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:68194b01:::benchmark_data_smithi047_486709_object321:head# 2024-09-18T16:05:03.080 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:695a6c0d:::benchmark_data_smithi047_486709_object108:head# 2024-09-18T16:05:03.080 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:69aded39:::benchmark_data_smithi047_486709_object50:head# 2024-09-18T16:05:03.080 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:69bcfb35:::benchmark_data_smithi047_486709_object105:head# 2024-09-18T16:05:03.081 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6a18e542:::benchmark_data_smithi047_486709_object233:head# 2024-09-18T16:05:03.081 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6a729ddd:::benchmark_data_smithi047_486709_object126:head# 2024-09-18T16:05:03.081 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6bb01b51:::benchmark_data_smithi047_486709_object309:head# 2024-09-18T16:05:03.082 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6bfd2096:::benchmark_data_smithi047_486709_object9:head# 2024-09-18T16:05:03.082 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6c8b5c8f:::benchmark_data_smithi047_486709_object304:head# 2024-09-18T16:05:03.082 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6d0a80a2:::benchmark_data_smithi047_486709_object294:head# 2024-09-18T16:05:03.083 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6d3fb942:::benchmark_data_smithi047_486709_object32:head# 2024-09-18T16:05:03.083 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6d462708:::benchmark_data_smithi047_486709_object40:head# 2024-09-18T16:05:03.084 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6e282fc5:::benchmark_data_smithi047_486709_object185:head# 2024-09-18T16:05:03.084 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6e872272:::benchmark_data_smithi047_486709_object21:head# 2024-09-18T16:05:03.084 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6f425c18:::benchmark_data_smithi047_486709_object335:head# 2024-09-18T16:05:03.085 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6ff2695a:::benchmark_data_smithi047_486709_object275:head# 2024-09-18T16:05:03.085 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:6ff6d44c:::benchmark_data_smithi047_486709_object360:head# 2024-09-18T16:05:03.086 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:718fd8da:::benchmark_data_smithi047_486709_object380:head# 2024-09-18T16:05:03.086 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:71a2afe1:::benchmark_data_smithi047_486709_object173:head# 2024-09-18T16:05:03.087 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:72f50112:::benchmark_data_smithi047_486709_object64:head# 2024-09-18T16:05:03.087 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:73b413e6:::benchmark_data_smithi047_486709_object326:head# 2024-09-18T16:05:03.087 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:74ad9f9f:::benchmark_data_smithi047_486709_object426:head# 2024-09-18T16:05:03.088 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:75a01843:::benchmark_data_smithi047_486709_object196:head# 2024-09-18T16:05:03.088 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:75b0fdc6:::benchmark_data_smithi047_486709_object245:head# 2024-09-18T16:05:03.089 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:767caffb:::benchmark_data_smithi047_486709_object217:head# 2024-09-18T16:05:03.089 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:76ac5f7c:::benchmark_data_smithi047_486709_object94:head# 2024-09-18T16:05:03.090 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:77d8a708:::benchmark_data_smithi047_486709_object194:head# 2024-09-18T16:05:03.090 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:77f84ff3:::benchmark_data_smithi047_486709_object166:head# 2024-09-18T16:05:03.091 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:786fd7b3:::benchmark_data_smithi047_486709_object81:head# 2024-09-18T16:05:03.091 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:78fe554f:::benchmark_data_smithi047_486709_object389:head# 2024-09-18T16:05:03.092 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:79079601:::benchmark_data_smithi047_486709_object339:head# 2024-09-18T16:05:03.092 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7a1185f6:::benchmark_data_smithi047_486709_object127:head# 2024-09-18T16:05:03.093 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7ae8e174:::benchmark_data_smithi047_486709_object199:head# 2024-09-18T16:05:03.093 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7b489099:::benchmark_data_smithi047_486709_object273:head# 2024-09-18T16:05:03.094 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7c04e6a5:::benchmark_data_smithi047_486709_object179:head# 2024-09-18T16:05:03.094 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7c97e866:::benchmark_data_smithi047_486709_object41:head# 2024-09-18T16:05:03.095 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7dca798b:::benchmark_data_smithi047_486709_object378:head# 2024-09-18T16:05:03.096 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7deba6af:::benchmark_data_smithi047_486709_object367:head# 2024-09-18T16:05:03.096 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7e562ab8:::benchmark_data_smithi047_486709_object333:head# 2024-09-18T16:05:03.096 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7e7d69b4:::benchmark_data_smithi047_486709_object242:head# 2024-09-18T16:05:03.097 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7ea71f5d:::benchmark_data_smithi047_486709_object240:head# 2024-09-18T16:05:03.100 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7eca269f:::benchmark_data_smithi047_486709_object112:head# 2024-09-18T16:05:03.102 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:7ef16b08:::benchmark_data_smithi047_486709_object274:head# 2024-09-18T16:05:03.102 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8002f347:::benchmark_data_smithi047_486709_object446:head# 2024-09-18T16:05:03.102 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:80da430d:::benchmark_data_smithi047_486709_object375:head# 2024-09-18T16:05:03.102 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:83a39f85:::benchmark_data_smithi047_486709_object223:head# 2024-09-18T16:05:03.102 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8425483e:::benchmark_data_smithi047_486709_object362:head# 2024-09-18T16:05:03.102 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8482eb7a:::benchmark_data_smithi047_486709_object263:head# 2024-09-18T16:05:03.103 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:862ff835:::benchmark_data_smithi047_486709_object89:head# 2024-09-18T16:05:03.103 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8671313c:::benchmark_data_smithi047_486709_object307:head# 2024-09-18T16:05:03.103 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:86b59ff3:::benchmark_data_smithi047_486709_object271:head# 2024-09-18T16:05:03.104 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:870d4dc1:::benchmark_data_smithi047_486709_object331:head# 2024-09-18T16:05:03.104 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:87ed6200:::benchmark_data_smithi047_486709_object129:head# 2024-09-18T16:05:03.104 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8843183f:::benchmark_data_smithi047_486709_object37:head# 2024-09-18T16:05:03.104 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8845a7f4:::benchmark_data_smithi047_486709_object308:head# 2024-09-18T16:05:03.104 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:89858298:::benchmark_data_smithi047_486709_object188:head# 2024-09-18T16:05:03.104 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:898c3769:::benchmark_data_smithi047_486709_object358:head# 2024-09-18T16:05:03.105 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8bd776e3:::benchmark_data_smithi047_486709_object270:head# 2024-09-18T16:05:03.105 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8c32331e:::benchmark_data_smithi047_486709_object87:head# 2024-09-18T16:05:03.106 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8dc40d6a:::benchmark_data_smithi047_486709_object73:head# 2024-09-18T16:05:03.107 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8dc4695f:::benchmark_data_smithi047_486709_object48:head# 2024-09-18T16:05:03.107 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8dc4c715:::benchmark_data_smithi047_486709_object439:head# 2024-09-18T16:05:03.107 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8dfd4ffe:::benchmark_data_smithi047_486709_object384:head# 2024-09-18T16:05:03.108 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8e07b728:::benchmark_data_smithi047_486709_object383:head# 2024-09-18T16:05:03.108 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8ee7f414:::benchmark_data_smithi047_486709_object432:head# 2024-09-18T16:05:03.109 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8f5acf4b:::benchmark_data_smithi047_486709_object410:head# 2024-09-18T16:05:03.109 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:8f67f5b1:::benchmark_data_smithi047_486709_object423:head# 2024-09-18T16:05:03.109 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:90fbb92e:::benchmark_data_smithi047_486709_object301:head# 2024-09-18T16:05:03.110 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:90fcc685:::benchmark_data_smithi047_486709_object305:head# 2024-09-18T16:05:03.110 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9159a601:::benchmark_data_smithi047_486709_object374:head# 2024-09-18T16:05:03.110 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:92c38da7:::benchmark_data_smithi047_486709_object19:head# 2024-09-18T16:05:03.111 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9328bdc8:::benchmark_data_smithi047_486709_object235:head# 2024-09-18T16:05:03.111 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:938c62e8:::benchmark_data_smithi047_486709_object159:head# 2024-09-18T16:05:03.112 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:93bdd3e7:::benchmark_data_smithi047_486709_object342:head# 2024-09-18T16:05:03.113 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:93d36327:::benchmark_data_smithi047_486709_object381:head# 2024-09-18T16:05:03.113 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:949174da:::benchmark_data_smithi047_486709_object222:head# 2024-09-18T16:05:03.113 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:95d91041:::benchmark_data_smithi047_486709_object110:head# 2024-09-18T16:05:03.114 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:963da936:::benchmark_data_smithi047_486709_object177:head# 2024-09-18T16:05:03.114 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:96b2e2e9:::benchmark_data_smithi047_486709_object154:head# 2024-09-18T16:05:03.114 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:973fddb5:::benchmark_data_smithi047_486709_object412:head# 2024-09-18T16:05:03.115 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:97431e54:::benchmark_data_smithi047_486709_object219:head# 2024-09-18T16:05:03.115 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:97a0d655:::benchmark_data_smithi047_486709_object34:head# 2024-09-18T16:05:03.116 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9876fff6:::benchmark_data_smithi047_486709_object26:head# 2024-09-18T16:05:03.116 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:99bc712b:::benchmark_data_smithi047_486709_object13:head# 2024-09-18T16:05:03.116 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9a758675:::benchmark_data_smithi047_486709_object11:head# 2024-09-18T16:05:03.117 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9a990753:::benchmark_data_smithi047_486709_object190:head# 2024-09-18T16:05:03.117 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9c153ac6:::benchmark_data_smithi047_486709_object163:head# 2024-09-18T16:05:03.117 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9c33fcbc:::benchmark_data_smithi047_486709_object340:head# 2024-09-18T16:05:03.118 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9c57e2a2:::benchmark_data_smithi047_486709_object332:head# 2024-09-18T16:05:03.118 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9e10d84b:::benchmark_data_smithi047_486709_object97:head# 2024-09-18T16:05:03.118 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9e13bce9:::benchmark_data_smithi047_486709_object128:head# 2024-09-18T16:05:03.119 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9eeea816:::benchmark_data_smithi047_486709_object368:head# 2024-09-18T16:05:03.119 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9f669583:::benchmark_data_smithi047_486709_object327:head# 2024-09-18T16:05:03.120 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:9f9ad351:::benchmark_data_smithi047_486709_object369:head# 2024-09-18T16:05:03.120 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a030bf98:::benchmark_data_smithi047_486709_object392:head# 2024-09-18T16:05:03.121 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a08761ad:::benchmark_data_smithi047_486709_object137:head# 2024-09-18T16:05:03.121 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a08e227f:::benchmark_data_smithi047_486709_object213:head# 2024-09-18T16:05:03.121 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a0e2b275:::benchmark_data_smithi047_486709_object164:head# 2024-09-18T16:05:03.122 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a20dfb87:::benchmark_data_smithi047_486709_object406:head# 2024-09-18T16:05:03.123 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a2465ab3:::benchmark_data_smithi047_486709_object283:head# 2024-09-18T16:05:03.123 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a26624da:::benchmark_data_smithi047_486709_object202:head# 2024-09-18T16:05:03.124 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a277cbc5:::benchmark_data_smithi047_486709_object104:head# 2024-09-18T16:05:03.125 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a37ccba1:::benchmark_data_smithi047_486709_object114:head# 2024-09-18T16:05:03.125 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a38900dd:::benchmark_data_smithi047_486709_object157:head# 2024-09-18T16:05:03.125 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a39d29ab:::benchmark_data_smithi047_486709_object354:head# 2024-09-18T16:05:03.125 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a4bf55fd:::benchmark_data_smithi047_486709_object428:head# 2024-09-18T16:05:03.126 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a656f25f:::benchmark_data_smithi047_486709_object361:head# 2024-09-18T16:05:03.126 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a67088c8:::benchmark_data_smithi047_486709_object266:head# 2024-09-18T16:05:03.126 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a6b65415:::benchmark_data_smithi047_486709_object56:head# 2024-09-18T16:05:03.127 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a6ba7861:::benchmark_data_smithi047_486709_object95:head# 2024-09-18T16:05:03.127 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a70bf38a:::benchmark_data_smithi047_486709_object224:head# 2024-09-18T16:05:03.128 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:a8558376:::benchmark_data_smithi047_486709_object346:head# 2024-09-18T16:05:03.128 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ab718bbf:::benchmark_data_smithi047_486709_object169:head# 2024-09-18T16:05:03.128 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ab9ff358:::benchmark_data_smithi047_486709_object261:head# 2024-09-18T16:05:03.129 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ad1254a4:::benchmark_data_smithi047_486709_object92:head# 2024-09-18T16:05:03.129 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ad283ca8:::benchmark_data_smithi047_486709_object142:head# 2024-09-18T16:05:03.130 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ad529a1a:::benchmark_data_smithi047_486709_object27:head# 2024-09-18T16:05:03.130 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ad696ccc:::benchmark_data_smithi047_486709_object65:head# 2024-09-18T16:05:03.131 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ada8b08b:::benchmark_data_smithi047_486709_object168:head# 2024-09-18T16:05:03.131 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:adb2d934:::benchmark_data_smithi047_486709_object1:head# 2024-09-18T16:05:03.138 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:adc96826:::benchmark_data_smithi047_486709_object234:head# 2024-09-18T16:05:03.139 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ae847374:::benchmark_data_smithi047_486709_object289:head# 2024-09-18T16:05:03.140 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:aef68153:::benchmark_data_smithi047_486709_object388:head# 2024-09-18T16:05:03.140 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:afb67530:::benchmark_data_smithi047_486709_object35:head# 2024-09-18T16:05:03.140 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b23cee60:::benchmark_data_smithi047_486709_object176:head# 2024-09-18T16:05:03.142 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b43dea0a:::benchmark_data_smithi047_486709_object193:head# 2024-09-18T16:05:03.142 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b4937444:::benchmark_data_smithi047_486709_object49:head# 2024-09-18T16:05:03.143 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b548fc6c:::benchmark_data_smithi047_486709_object16:head# 2024-09-18T16:05:03.143 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b58d71e5:::benchmark_data_smithi047_486709_object243:head# 2024-09-18T16:05:03.143 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b66fabbe:::benchmark_data_smithi047_486709_object205:head# 2024-09-18T16:05:03.144 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b68f9b67:::benchmark_data_smithi047_486709_object123:head# 2024-09-18T16:05:03.144 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b6bb4500:::benchmark_data_smithi047_486709_object430:head# 2024-09-18T16:05:03.145 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b711ba00:::benchmark_data_smithi047_486709_object54:head# 2024-09-18T16:05:03.145 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b7aa9efd:::benchmark_data_smithi047_486709_object170:head# 2024-09-18T16:05:03.146 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b8001294:::benchmark_data_smithi047_486709_object211:head# 2024-09-18T16:05:03.146 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b8615e71:::benchmark_last_metadata:head# 2024-09-18T16:05:03.146 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b861831e:::benchmark_data_smithi047_486709_object377:head# 2024-09-18T16:05:03.154 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b86d1c91:::benchmark_data_smithi047_486709_object257:head# 2024-09-18T16:05:03.155 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b876a2da:::benchmark_data_smithi047_486709_object68:head# 2024-09-18T16:05:03.156 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b8a243fe:::benchmark_data_smithi047_486709_object62:head# 2024-09-18T16:05:03.156 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b8a76b51:::benchmark_data_smithi047_486709_object324:head# 2024-09-18T16:05:03.156 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b8aa8906:::benchmark_data_smithi047_486709_object379:head# 2024-09-18T16:05:03.157 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b92ada8d:::benchmark_data_smithi047_486709_object174:head# 2024-09-18T16:05:03.158 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:b96cca9d:::benchmark_data_smithi047_486709_object119:head# 2024-09-18T16:05:03.158 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:bba01b0c:::benchmark_data_smithi047_486709_object437:head# 2024-09-18T16:05:03.159 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:bc575efa:::benchmark_data_smithi047_486709_object191:head# 2024-09-18T16:05:03.159 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:be35df85:::benchmark_data_smithi047_486709_object85:head# 2024-09-18T16:05:03.160 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:bef8539c:::benchmark_data_smithi047_486709_object182:head# 2024-09-18T16:05:03.160 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:bf08fd7d:::benchmark_data_smithi047_486709_object215:head# 2024-09-18T16:05:03.160 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:bfebfba3:::benchmark_data_smithi047_486709_object17:head# 2024-09-18T16:05:03.161 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:bffd4c30:::benchmark_data_smithi047_486709_object441:head# 2024-09-18T16:05:03.162 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c01a9a33:::benchmark_data_smithi047_486709_object133:head# 2024-09-18T16:05:03.162 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c04b3582:::benchmark_data_smithi047_486709_object396:head# 2024-09-18T16:05:03.164 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c07e1d98:::benchmark_data_smithi047_486709_object72:head# 2024-09-18T16:05:03.164 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c0aeed8e:::benchmark_data_smithi047_486709_object8:head# 2024-09-18T16:05:03.164 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c0cb7a7f:::benchmark_data_smithi047_486709_object195:head# 2024-09-18T16:05:03.164 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c1154d1b:::benchmark_data_smithi047_486709_object262:head# 2024-09-18T16:05:03.165 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c1311f8e:::benchmark_data_smithi047_486709_object6:head# 2024-09-18T16:05:03.165 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c27edcf0:::benchmark_data_smithi047_486709_object14:head# 2024-09-18T16:05:03.165 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c2bc9557:::benchmark_data_smithi047_486709_object167:head# 2024-09-18T16:05:03.166 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c2cfa1ae:::benchmark_data_smithi047_486709_object122:head# 2024-09-18T16:05:03.166 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c2e4c850:::benchmark_data_smithi047_486709_object310:head# 2024-09-18T16:05:03.166 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c305b228:::benchmark_data_smithi047_486709_object18:head# 2024-09-18T16:05:03.167 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c39807c5:::benchmark_data_smithi047_486709_object186:head# 2024-09-18T16:05:03.167 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c4b18f6d:::benchmark_data_smithi047_486709_object153:head# 2024-09-18T16:05:03.168 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c59bd567:::benchmark_data_smithi047_486709_object147:head# 2024-09-18T16:05:03.168 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c611ed63:::benchmark_data_smithi047_486709_object187:head# 2024-09-18T16:05:03.169 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c69aa8b7:::benchmark_data_smithi047_486709_object382:head# 2024-09-18T16:05:03.169 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c76cb633:::benchmark_data_smithi047_486709_object98:head# 2024-09-18T16:05:03.169 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c77b16ab:::benchmark_data_smithi047_486709_object260:head# 2024-09-18T16:05:03.170 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c7a4be7e:::benchmark_data_smithi047_486709_object83:head# 2024-09-18T16:05:03.170 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c7bb3b1e:::benchmark_data_smithi047_486709_object314:head# 2024-09-18T16:05:03.171 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:c9ce43fe:::benchmark_data_smithi047_486709_object373:head# 2024-09-18T16:05:03.171 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:cab07825:::benchmark_data_smithi047_486709_object419:head# 2024-09-18T16:05:03.172 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:cb3c780b:::benchmark_data_smithi047_486709_object353:head# 2024-09-18T16:05:03.172 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:cbcebd78:::benchmark_data_smithi047_486709_object67:head# 2024-09-18T16:05:03.173 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:cc5b2aeb:::benchmark_data_smithi047_486709_object61:head# 2024-09-18T16:05:03.174 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ccf5e326:::benchmark_data_smithi047_486709_object319:head# 2024-09-18T16:05:03.174 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:cdc7b216:::benchmark_data_smithi047_486709_object277:head# 2024-09-18T16:05:03.174 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ce245c9e:::benchmark_data_smithi047_486709_object341:head# 2024-09-18T16:05:03.175 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ce3a05b5:::benchmark_data_smithi047_486709_object415:head# 2024-09-18T16:05:03.175 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:cea6f1a6:::benchmark_data_smithi047_486709_object183:head# 2024-09-18T16:05:03.175 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ceac51df:::benchmark_data_smithi047_486709_object208:head# 2024-09-18T16:05:03.176 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:cf9d71f7:::benchmark_data_smithi047_486709_object328:head# 2024-09-18T16:05:03.176 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d0e0ddef:::benchmark_data_smithi047_486709_object440:head# 2024-09-18T16:05:03.176 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d13e0997:::benchmark_data_smithi047_486709_object120:head# 2024-09-18T16:05:03.177 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d14c0890:::benchmark_data_smithi047_486709_object29:head# 2024-09-18T16:05:03.178 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d178597d:::benchmark_data_smithi047_486709_object38:head# 2024-09-18T16:05:03.178 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d1a87433:::benchmark_data_smithi047_486709_object101:head# 2024-09-18T16:05:03.179 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d231ee82:::benchmark_data_smithi047_486709_object23:head# 2024-09-18T16:05:03.179 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d2cc77b5:::benchmark_data_smithi047_486709_object411:head# 2024-09-18T16:05:03.179 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d30af311:::benchmark_data_smithi047_486709_object12:head# 2024-09-18T16:05:03.180 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d3689ae3:::benchmark_data_smithi047_486709_object131:head# 2024-09-18T16:05:03.180 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d3af68ed:::benchmark_data_smithi047_486709_object267:head# 2024-09-18T16:05:03.181 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d4d051bb:::benchmark_data_smithi047_486709_object442:head# 2024-09-18T16:05:03.181 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d4eed802:::benchmark_data_smithi047_486709_object418:head# 2024-09-18T16:05:03.181 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d51f7893:::benchmark_data_smithi047_486709_object203:head# 2024-09-18T16:05:03.182 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d55a9f5e:::benchmark_data_smithi047_486709_object282:head# 2024-09-18T16:05:03.182 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d6023347:::benchmark_data_smithi047_486709_object258:head# 2024-09-18T16:05:03.183 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d6507515:::benchmark_data_smithi047_486709_object84:head# 2024-09-18T16:05:03.184 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d67b3f40:::benchmark_data_smithi047_486709_object57:head# 2024-09-18T16:05:03.184 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d985ffbb:::benchmark_data_smithi047_486709_object221:head# 2024-09-18T16:05:03.184 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:d9c64963:::benchmark_data_smithi047_486709_object403:head# 2024-09-18T16:05:03.185 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:db5c1772:::benchmark_data_smithi047_486709_object399:head# 2024-09-18T16:05:03.185 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:dbc75972:::benchmark_data_smithi047_486709_object99:head# 2024-09-18T16:05:03.186 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:dc504976:::benchmark_data_smithi047_486709_object74:head# 2024-09-18T16:05:03.186 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:de0372e0:::benchmark_data_smithi047_486709_object348:head# 2024-09-18T16:05:03.186 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:de4ed419:::benchmark_data_smithi047_486709_object180:head# 2024-09-18T16:05:03.187 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:de9d26f3:::benchmark_data_smithi047_486709_object200:head# 2024-09-18T16:05:03.187 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:dfe26c10:::benchmark_data_smithi047_486709_object250:head# 2024-09-18T16:05:03.188 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:dffb606a:::benchmark_data_smithi047_486709_object86:head# 2024-09-18T16:05:03.188 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e00a47c6:::benchmark_data_smithi047_486709_object93:head# 2024-09-18T16:05:03.188 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e041d288:::benchmark_data_smithi047_486709_object295:head# 2024-09-18T16:05:03.189 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e1aa6946:::benchmark_data_smithi047_486709_object172:head# 2024-09-18T16:05:03.189 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e25d4c6c:::benchmark_data_smithi047_486709_object371:head# 2024-09-18T16:05:03.190 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e26ea5d9:::benchmark_data_smithi047_486709_object214:head# 2024-09-18T16:05:03.190 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e2d269ba:::benchmark_data_smithi047_486709_object351:head# 2024-09-18T16:05:03.191 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e3c63889:::benchmark_data_smithi047_486709_object247:head# 2024-09-18T16:05:03.191 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e436ca17:::benchmark_data_smithi047_486709_object338:head# 2024-09-18T16:05:03.192 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e4deba91:::benchmark_data_smithi047_486709_object28:head# 2024-09-18T16:05:03.192 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e53d80b3:::benchmark_data_smithi047_486709_object387:head# 2024-09-18T16:05:03.192 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e569c738:::benchmark_data_smithi047_486709_object241:head# 2024-09-18T16:05:03.193 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e60de3cc:::benchmark_data_smithi047_486709_object291:head# 2024-09-18T16:05:03.193 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e6a4319b:::benchmark_data_smithi047_486709_object402:head# 2024-09-18T16:05:03.194 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e9784ca2:::benchmark_data_smithi047_486709_object337:head# 2024-09-18T16:05:03.195 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e98a11e8:::benchmark_data_smithi047_486709_object438:head# 2024-09-18T16:05:03.195 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:e9b09430:::benchmark_data_smithi047_486709_object306:head# 2024-09-18T16:05:03.195 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ea1f5cdd:::benchmark_data_smithi047_486709_object80:head# 2024-09-18T16:05:03.195 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:eae3fd7e:::benchmark_data_smithi047_486709_object79:head# 2024-09-18T16:05:03.196 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ebcff5c1:::benchmark_data_smithi047_486709_object226:head# 2024-09-18T16:05:03.196 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ec0cd898:::benchmark_data_smithi047_486709_object253:head# 2024-09-18T16:05:03.197 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ec1827ae:::benchmark_data_smithi047_486709_object7:head# 2024-09-18T16:05:03.197 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ec3b5e98:::benchmark_data_smithi047_486709_object248:head# 2024-09-18T16:05:03.198 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ecdf4765:::benchmark_data_smithi047_486709_object265:head# 2024-09-18T16:05:03.198 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:edc3a588:::benchmark_data_smithi047_486709_object285:head# 2024-09-18T16:05:03.198 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:edcb7a06:::benchmark_data_smithi047_486709_object400:head# 2024-09-18T16:05:03.199 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ee09c109:::benchmark_data_smithi047_486709_object344:head# 2024-09-18T16:05:03.199 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ef289abb:::benchmark_data_smithi047_486709_object158:head# 2024-09-18T16:05:03.200 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ef4a0f3d:::benchmark_data_smithi047_486709_object184:head# 2024-09-18T16:05:03.200 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ef521e3e:::benchmark_data_smithi047_486709_object334:head# 2024-09-18T16:05:03.201 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:efe89b5b:::benchmark_data_smithi047_486709_object135:head# 2024-09-18T16:05:03.201 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f0638365:::benchmark_data_smithi047_486709_object121:head# 2024-09-18T16:05:03.202 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f1a51331:::benchmark_data_smithi047_486709_object102:head# 2024-09-18T16:05:03.202 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f26a1847:::benchmark_data_smithi047_486709_object39:head# 2024-09-18T16:05:03.202 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f35d757b:::benchmark_data_smithi047_486709_object278:head# 2024-09-18T16:05:03.203 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f3fb4ddd:::benchmark_data_smithi047_486709_object279:head# 2024-09-18T16:05:03.203 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f4cea92f:::benchmark_data_smithi047_486709_object424:head# 2024-09-18T16:05:03.204 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f4dd87be:::benchmark_data_smithi047_486709_object60:head# 2024-09-18T16:05:03.204 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f6597b7e:::benchmark_data_smithi047_486709_object229:head# 2024-09-18T16:05:03.205 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f6889cb7:::benchmark_data_smithi047_486709_object24:head# 2024-09-18T16:05:03.205 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f6b9845c:::benchmark_data_smithi047_486709_object130:head# 2024-09-18T16:05:03.205 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f7746cf1:::benchmark_data_smithi047_486709_object288:head# 2024-09-18T16:05:03.206 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f7a12768:::benchmark_data_smithi047_486709_object149:head# 2024-09-18T16:05:03.206 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f8060b73:::benchmark_data_smithi047_486709_object107:head# 2024-09-18T16:05:03.206 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f826a72f:::benchmark_data_smithi047_486709_object317:head# 2024-09-18T16:05:03.207 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f834e6c9:::benchmark_data_smithi047_486709_object204:head# 2024-09-18T16:05:03.207 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:f9a3718e:::benchmark_data_smithi047_486709_object230:head# 2024-09-18T16:05:03.208 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fa43b90e:::benchmark_data_smithi047_486709_object5:head# 2024-09-18T16:05:03.208 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fa71b4b0:::benchmark_data_smithi047_486709_object77:head# 2024-09-18T16:05:03.208 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fa9f4c57:::benchmark_data_smithi047_486709_object281:head# 2024-09-18T16:05:03.208 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fbe77135:::benchmark_data_smithi047_486709_object409:head# 2024-09-18T16:05:03.209 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fc2aef5a:::benchmark_data_smithi047_486709_object256:head# 2024-09-18T16:05:03.210 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fcd8d91e:::benchmark_data_smithi047_486709_object264:head# 2024-09-18T16:05:03.210 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fe24c144:::benchmark_data_smithi047_486709_object55:head# 2024-09-18T16:05:03.210 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fe3019dd:::benchmark_data_smithi047_486709_object100:head# 2024-09-18T16:05:03.210 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fe38134c:::benchmark_data_smithi047_486709_object246:head# 2024-09-18T16:05:03.211 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:fe56a8df:::benchmark_data_smithi047_486709_object175:head# 2024-09-18T16:05:03.211 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ff8c3386:::benchmark_data_smithi047_486709_object45:head# 2024-09-18T16:05:03.212 INFO:tasks.workunit.client.0.smithi047.stderr:Read #1:ffe0aa00:::benchmark_data_smithi047_486709_object287:head# 2024-09-18T16:05:03.212 INFO:tasks.workunit.client.0.smithi047.stderr:Export successful 2024-09-18T16:05:03.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:175: TEST_import_after_split: activate_osd td/pg-split-merge 0 2024-09-18T16:05:03.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-09-18T16:05:03.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T16:05:03.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T16:05:03.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T16:05:03.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/0 2024-09-18T16:05:03.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1a671e43-8866-44a5-abe3-c1e1f4ff765b --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-18T16:05:03.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:05:03.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:05:03.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:05:03.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-09-18T16:05:03.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-09-18T16:05:03.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T16:05:03.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T16:05:03.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-09-18T16:05:03.486 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T16:05:03.486 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:05:03.486 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:05:03.487 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:05:03.487 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:05:03.487 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.463129 2024-09-18T16:05:03.487 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.463129/$cluster-$name.asok' 2024-09-18T16:05:03.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.463129/$cluster-$name.asok' 2024-09-18T16:05:03.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:05:03.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-09-18T16:05:03.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-09-18T16:05:03.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:05:03.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:05:03.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:05:03.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:05:03.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T16:05:03.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T16:05:03.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/0 2024-09-18T16:05:03.488 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T16:05:03.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T16:05:03.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1a671e43-8866-44a5-abe3-c1e1f4ff765b --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.463129/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:05:03.489 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/0/whoami 2024-09-18T16:05:03.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T16:05:03.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T16:05:03.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T16:05:03.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T16:05:03.514 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:05:03.513+0000 7f3bffe0c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:05:03.516 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:05:03.515+0000 7f3bffe0c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:05:03.517 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:05:03.516+0000 7f3bffe0c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:05:03.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T16:05:03.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:05:03.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T16:05:03.901 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:05:03.902 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:05:03.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:05:03.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:03.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:05:03.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:03.903 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:05:04.037 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:05:04.036+0000 7f3bffe0c440 -1 Falling back to public interface 2024-09-18T16:05:04.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:05:05.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:05:05.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:05.275 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T16:05:05.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:05:05.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:05.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:05:05.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:05:06.514 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:05:06.512+0000 7f3bffe0c440 -1 osd.0 8 log_to_monitors true 2024-09-18T16:05:06.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:05:06.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:06.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:05:06.696 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T16:05:06.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:06.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:05:07.160 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 11 up_thru 11 down_at 9 last_clean_interval [5,8) [v2:127.0.0.1:6802/487140,v1:127.0.0.1:6803/487140] [v2:127.0.0.1:6804/487140,v1:127.0.0.1:6805/487140] exists,up be7e9a0f-d00f-4cf4-aaef-94da8b714cda 2024-09-18T16:05:07.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:05:07.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:05:07.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:05:07.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:177: TEST_import_after_split: ceph osd pool set foo pg_num 2 2024-09-18T16:05:07.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:178: TEST_import_after_split: sleep 5 2024-09-18T16:05:12.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:179: TEST_import_after_split: ceph daemon osd.0 perf dump 2024-09-18T16:05:12.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:179: TEST_import_after_split: jq .osd.numpg 2024-09-18T16:05:12.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:179: TEST_import_after_split: grep 1 2024-09-18T16:05:13.032 INFO:tasks.workunit.client.0.smithi047.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-18T16:05:13.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:180: TEST_import_after_split: wait_for_clean 2024-09-18T16:05:13.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:05:13.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:05:13.034 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:05:13.034 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:05:13.035 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:05:13.035 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:05:13.035 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:05:13.035 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:05:13.035 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:05:13.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T16:05:13.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:05:13.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:05:13.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:05:13.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:05:13.130 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:05:13.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T16:05:13.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:05:13.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:05:13.562 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:05:13.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=47244640260 2024-09-18T16:05:13.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 47244640260 2024-09-18T16:05:13.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-47244640260' 2024-09-18T16:05:13.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:05:13.673 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-47244640260 2024-09-18T16:05:13.673 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:05:13.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:05:13.674 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-47244640260 2024-09-18T16:05:13.674 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:05:13.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=47244640260 2024-09-18T16:05:13.675 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 47244640260' 2024-09-18T16:05:13.676 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 47244640260 2024-09-18T16:05:13.676 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:05:14.079 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 47244640258 -lt 47244640260 2024-09-18T16:05:14.080 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:05:15.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:05:15.081 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:05:15.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 47244640260 -lt 47244640260 2024-09-18T16:05:15.502 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:05:15.502 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:05:15.502 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:05:16.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-09-18T16:05:16.044 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:05:16.044 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:05:16.044 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:05:16.045 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:05:16.045 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:05:16.045 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:05:16.045 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:05:16.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T16:05:16.442 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:05:16.442 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:05:16.442 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:05:16.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-09-18T16:05:16.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:05:16.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:05:16.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:182: TEST_import_after_split: kill_daemons td/pg-split-merge TERM osd.0 2024-09-18T16:05:16.941 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:05:16.941 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:05:16.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:05:16.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:05:16.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:05:17.247 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:05:17.247 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:184: TEST_import_after_split: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op remove --pgid 1.0 --force 2024-09-18T16:05:18.353 INFO:tasks.workunit.client.0.smithi047.stdout: marking collection for removal 2024-09-18T16:05:18.354 INFO:tasks.workunit.client.0.smithi047.stdout:setting '_remove' omap key 2024-09-18T16:05:18.354 INFO:tasks.workunit.client.0.smithi047.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-09-18T16:05:18.604 INFO:tasks.workunit.client.0.smithi047.stdout:Remove successful 2024-09-18T16:05:18.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:187: TEST_import_after_split: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0 2024-09-18T16:05:20.041 INFO:tasks.workunit.client.0.smithi047.stdout:get_pg_num_history pg_num_history pg_num_history(e17 pg_nums {1={6=1,15=2}} deleted_pools ) 2024-09-18T16:05:20.041 INFO:tasks.workunit.client.0.smithi047.stdout:Importing pgid 1.0 2024-09-18T16:05:20.042 INFO:tasks.workunit.client.0.smithi047.stdout:pool 1 pg_num 1 -> 2 2024-09-18T16:05:20.042 INFO:tasks.workunit.client.0.smithi047.stderr: children are 1.1 2024-09-18T16:05:20.042 INFO:tasks.workunit.client.0.smithi047.stderr:pgid 1.0 splits to 1.1 and 1.1 exists 2024-09-18T16:05:20.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:189: TEST_import_after_split: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op remove --pgid 1.1 --force 2024-09-18T16:05:21.253 INFO:tasks.workunit.client.0.smithi047.stdout: marking collection for removal 2024-09-18T16:05:21.254 INFO:tasks.workunit.client.0.smithi047.stdout:setting '_remove' omap key 2024-09-18T16:05:21.254 INFO:tasks.workunit.client.0.smithi047.stdout:finish_remove_pgs 1.1_head removing 1.1 2024-09-18T16:05:21.645 INFO:tasks.workunit.client.0.smithi047.stdout:Remove successful 2024-09-18T16:05:22.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:191: TEST_import_after_split: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0 2024-09-18T16:05:22.907 INFO:tasks.workunit.client.0.smithi047.stdout:get_pg_num_history pg_num_history pg_num_history(e17 pg_nums {1={6=1,15=2}} deleted_pools ) 2024-09-18T16:05:22.907 INFO:tasks.workunit.client.0.smithi047.stdout:Importing pgid 1.0 2024-09-18T16:05:22.907 INFO:tasks.workunit.client.0.smithi047.stdout:pool 1 pg_num 1 -> 2 2024-09-18T16:05:22.907 INFO:tasks.workunit.client.0.smithi047.stderr: children are 1.1 2024-09-18T16:05:22.908 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:00455b6d:::benchmark_data_smithi047_486709_object58:head# 2024-09-18T16:05:22.908 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:23.057 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:00bb5925:::benchmark_data_smithi047_486709_object299:head# 2024-09-18T16:05:23.057 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:23.107 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0190b0e9:::benchmark_data_smithi047_486709_object10:head# 2024-09-18T16:05:23.107 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:23.158 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:01b14c7b:::benchmark_data_smithi047_486709_object118:head# 2024-09-18T16:05:23.158 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:23.216 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:01f804ce:::benchmark_data_smithi047_486709_object394:head# 2024-09-18T16:05:23.216 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:23.300 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:04421016:::benchmark_data_smithi047_486709_object20:head# 2024-09-18T16:05:23.300 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:23.375 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:04ecef8d:::benchmark_data_smithi047_486709_object150:head# 2024-09-18T16:05:23.376 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:23.425 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:04f9a10f:::benchmark_data_smithi047_486709_object44:head# 2024-09-18T16:05:23.425 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:23.476 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:06d460d4:::benchmark_data_smithi047_486709_object370:head# 2024-09-18T16:05:23.476 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:23.551 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0709c473:::benchmark_data_smithi047_486709_object249:head# 2024-09-18T16:05:23.551 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:23.635 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:08e3c220:::benchmark_data_smithi047_486709_object292:head# 2024-09-18T16:05:23.635 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:23.727 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0a74bda8:::benchmark_data_smithi047_486709_object239:head# 2024-09-18T16:05:23.727 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:23.802 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0d17e7da:::benchmark_data_smithi047_486709_object78:head# 2024-09-18T16:05:23.803 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:23.852 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0e62fc90:::benchmark_data_smithi047_486709_object350:head# 2024-09-18T16:05:23.853 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:23.903 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:0eb0a557:::benchmark_data_smithi047_486709_object116:head# 2024-09-18T16:05:23.903 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:23.945 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1112f9ed:::benchmark_data_smithi047_486709_object420:head# 2024-09-18T16:05:23.945 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:23.970 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:11368039:::benchmark_data_smithi047_486709_object311:head# 2024-09-18T16:05:23.970 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:23.995 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:11d92add:::benchmark_data_smithi047_486709_object269:head# 2024-09-18T16:05:23.995 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:24.020 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1210e9b6:::benchmark_data_smithi047_486709_object197:head# 2024-09-18T16:05:24.020 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:24.045 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:147965e0:::benchmark_data_smithi047_486709_object251:head# 2024-09-18T16:05:24.045 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:24.070 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:148835e5:::benchmark_data_smithi047_486709_object139:head# 2024-09-18T16:05:24.070 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:24.112 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:155430aa:::benchmark_data_smithi047_486709_object447:head# 2024-09-18T16:05:24.112 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:24.137 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:15fc55cf:::benchmark_data_smithi047_486709_object345:head# 2024-09-18T16:05:24.137 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:24.162 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:172bd864:::benchmark_data_smithi047_486709_object115:head# 2024-09-18T16:05:24.162 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:24.188 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:17700f86:::benchmark_data_smithi047_486709_object181:head# 2024-09-18T16:05:24.188 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:24.212 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:17bc617d:::benchmark_data_smithi047_486709_object352:head# 2024-09-18T16:05:24.213 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:24.237 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:18238136:::benchmark_data_smithi047_486709_object228:head# 2024-09-18T16:05:24.238 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:24.263 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:184a6b0e:::benchmark_data_smithi047_486709_object434:head# 2024-09-18T16:05:24.263 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:24.288 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:186e5996:::benchmark_data_smithi047_486709_object303:head# 2024-09-18T16:05:24.288 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:24.313 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1889d760:::benchmark_data_smithi047_486709_object106:head# 2024-09-18T16:05:24.313 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:24.338 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:190aee5c:::benchmark_data_smithi047_486709_object2:head# 2024-09-18T16:05:24.338 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:24.363 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:19e2fb60:::benchmark_data_smithi047_486709_object51:head# 2024-09-18T16:05:24.363 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:24.388 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1a1c1df2:::benchmark_data_smithi047_486709_object244:head# 2024-09-18T16:05:24.388 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:24.413 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1a84ad1c:::benchmark_data_smithi047_486709_object216:head# 2024-09-18T16:05:24.413 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:24.438 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1c29bfaa:::benchmark_data_smithi047_486709_object76:head# 2024-09-18T16:05:24.439 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:24.464 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1db7bb6b:::benchmark_data_smithi047_486709_object117:head# 2024-09-18T16:05:24.464 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:24.489 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1ddafb67:::benchmark_data_smithi047_486709_object357:head# 2024-09-18T16:05:24.489 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:24.547 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1e263654:::benchmark_data_smithi047_486709_object124:head# 2024-09-18T16:05:24.547 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:24.581 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1ee7ec61:::benchmark_data_smithi047_486709_object22:head# 2024-09-18T16:05:24.581 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:24.606 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1f89eb4d:::benchmark_data_smithi047_486709_object71:head# 2024-09-18T16:05:24.606 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:24.631 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1fca7266:::benchmark_data_smithi047_486709_object236:head# 2024-09-18T16:05:24.631 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:24.656 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:1fce0a50:::benchmark_data_smithi047_486709_object141:head# 2024-09-18T16:05:24.657 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:24.690 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2019af5e:::benchmark_data_smithi047_486709_object132:head# 2024-09-18T16:05:24.690 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:24.723 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:208fc203:::benchmark_data_smithi047_486709_object298:head# 2024-09-18T16:05:24.723 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:24.757 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:20dfb558:::benchmark_data_smithi047_486709_object397:head# 2024-09-18T16:05:24.757 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:24.790 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:21fdd2cb:::benchmark_data_smithi047_486709_object391:head# 2024-09-18T16:05:24.790 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:24.823 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:23900c46:::benchmark_data_smithi047_486709_object103:head# 2024-09-18T16:05:24.824 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:24.857 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:23f7f590:::benchmark_data_smithi047_486709_object390:head# 2024-09-18T16:05:24.857 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:24.890 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2428039f:::benchmark_data_smithi047_486709_object178:head# 2024-09-18T16:05:24.891 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:24.924 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:24bb594b:::benchmark_data_smithi047_486709_object422:head# 2024-09-18T16:05:24.924 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:24.957 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:24ca8f82:::benchmark_data_smithi047_486709_object206:head# 2024-09-18T16:05:24.957 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:24.991 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:25e9f491:::benchmark_data_smithi047_486709_object220:head# 2024-09-18T16:05:24.991 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:25.024 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:25f093aa:::benchmark_data_smithi047_486709_object316:head# 2024-09-18T16:05:25.025 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:25.058 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:261b1b9f:::benchmark_data_smithi047_486709_object192:head# 2024-09-18T16:05:25.058 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:25.091 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:263bb008:::benchmark_data_smithi047_486709_object366:head# 2024-09-18T16:05:25.091 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:25.125 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:26c2dcf5:::benchmark_data_smithi047_486709_object395:head# 2024-09-18T16:05:25.125 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:25.158 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:28588a22:::benchmark_data_smithi047_486709_object421:head# 2024-09-18T16:05:25.158 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:25.192 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:291e913c:::benchmark_data_smithi047_486709_object323:head# 2024-09-18T16:05:25.192 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:25.225 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:297d98c0:::benchmark_data_smithi047_486709_object225:head# 2024-09-18T16:05:25.225 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:25.284 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2ae0ba07:::benchmark_data_smithi047_486709_object113:head# 2024-09-18T16:05:25.284 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:25.334 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2b0ac3f3:::benchmark_data_smithi047_486709_object407:head# 2024-09-18T16:05:25.334 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:25.367 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2b922108:::benchmark_data_smithi047_486709_object425:head# 2024-09-18T16:05:25.367 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:25.409 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2bb192a9:::benchmark_data_smithi047_486709_object42:head# 2024-09-18T16:05:25.409 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:25.443 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2bec4599:::benchmark_data_smithi047_486709_object296:head# 2024-09-18T16:05:25.443 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:25.476 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2c95f113:::benchmark_data_smithi047_486709_object146:head# 2024-09-18T16:05:25.476 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:25.518 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2cc4228c:::benchmark_data_smithi047_486709_object43:head# 2024-09-18T16:05:25.518 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:25.614 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2e0b5e21:::benchmark_data_smithi047_486709_object315:head# 2024-09-18T16:05:25.614 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:25.777 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2ec2cc08:::benchmark_data_smithi047_486709_object53:head# 2024-09-18T16:05:25.777 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:25.835 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2f1c11a8:::benchmark_data_smithi047_486709_object30:head# 2024-09-18T16:05:25.836 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:25.902 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:2f5f8553:::benchmark_data_smithi047_486709_object111:head# 2024-09-18T16:05:25.903 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:25.936 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:303347ff:::benchmark_data_smithi047_486709_object405:head# 2024-09-18T16:05:25.936 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:25.961 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:30c94f4d:::benchmark_data_smithi047_486709_object3:head# 2024-09-18T16:05:25.961 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:25.986 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:31cd2170:::benchmark_data_smithi047_486709_object302:head# 2024-09-18T16:05:25.986 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.011 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:31d8df38:::benchmark_data_smithi047_486709_object312:head# 2024-09-18T16:05:26.011 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.036 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:325fc1d4:::benchmark_data_smithi047_486709_object144:head# 2024-09-18T16:05:26.036 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.062 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:326bd0be:::benchmark_data_smithi047_486709_object293:head# 2024-09-18T16:05:26.062 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.087 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:33e7e262:::benchmark_data_smithi047_486709_object33:head# 2024-09-18T16:05:26.087 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.112 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:345a13c5:::benchmark_data_smithi047_486709_object436:head# 2024-09-18T16:05:26.112 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.137 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:345ca073:::benchmark_data_smithi047_486709_object322:head# 2024-09-18T16:05:26.137 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.162 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:34f0cac8:::benchmark_data_smithi047_486709_object134:head# 2024-09-18T16:05:26.162 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.187 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3520db44:::benchmark_data_smithi047_486709_object272:head# 2024-09-18T16:05:26.187 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.212 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:358928ae:::benchmark_data_smithi047_486709_object91:head# 2024-09-18T16:05:26.212 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.238 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3594212f:::benchmark_data_smithi047_486709_object148:head# 2024-09-18T16:05:26.238 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.262 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:37e99f13:::benchmark_data_smithi047_486709_object156:head# 2024-09-18T16:05:26.263 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.288 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:380f7c7b:::benchmark_data_smithi047_486709_object0:head# 2024-09-18T16:05:26.288 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.313 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:38102a61:::benchmark_data_smithi047_486709_object413:head# 2024-09-18T16:05:26.313 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.338 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3844bb99:::benchmark_data_smithi047_486709_object138:head# 2024-09-18T16:05:26.338 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.363 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:38ba1613:::benchmark_data_smithi047_486709_object90:head# 2024-09-18T16:05:26.363 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.388 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:38be71f8:::benchmark_data_smithi047_486709_object329:head# 2024-09-18T16:05:26.388 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.413 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:39341ae8:::benchmark_data_smithi047_486709_object365:head# 2024-09-18T16:05:26.413 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.438 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:39839f11:::benchmark_data_smithi047_486709_object151:head# 2024-09-18T16:05:26.439 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.463 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:398591fd:::benchmark_data_smithi047_486709_object218:head# 2024-09-18T16:05:26.463 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.488 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:39f2ac8a:::benchmark_data_smithi047_486709_object82:head# 2024-09-18T16:05:26.489 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.513 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3a9ed21e:::benchmark_data_smithi047_486709_object284:head# 2024-09-18T16:05:26.514 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.539 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3acde6b6:::benchmark_data_smithi047_486709_object237:head# 2024-09-18T16:05:26.539 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.564 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3b3a14d6:::benchmark_data_smithi047_486709_object125:head# 2024-09-18T16:05:26.564 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.589 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3c0f53ee:::benchmark_data_smithi047_486709_object330:head# 2024-09-18T16:05:26.589 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.614 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3c4d3f0b:::benchmark_data_smithi047_486709_object268:head# 2024-09-18T16:05:26.615 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.639 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3c92443c:::benchmark_data_smithi047_486709_object364:head# 2024-09-18T16:05:26.639 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.665 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3e04c091:::benchmark_data_smithi047_486709_object238:head# 2024-09-18T16:05:26.665 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.690 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3fbd99a7:::benchmark_data_smithi047_486709_object376:head# 2024-09-18T16:05:26.690 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.715 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3fd976ca:::benchmark_data_smithi047_486709_object427:head# 2024-09-18T16:05:26.715 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.740 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:3ffbf4f7:::benchmark_data_smithi047_486709_object207:head# 2024-09-18T16:05:26.740 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.765 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:40498dc8:::benchmark_data_smithi047_486709_object297:head# 2024-09-18T16:05:26.765 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.790 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:407f5073:::benchmark_data_smithi047_486709_object320:head# 2024-09-18T16:05:26.790 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.815 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:42999bca:::benchmark_data_smithi047_486709_object286:head# 2024-09-18T16:05:26.815 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.840 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:42a09cc6:::benchmark_data_smithi047_486709_object325:head# 2024-09-18T16:05:26.840 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.866 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:430b1235:::benchmark_data_smithi047_486709_object372:head# 2024-09-18T16:05:26.866 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.891 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:445355c8:::benchmark_data_smithi047_486709_object69:head# 2024-09-18T16:05:26.891 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.916 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:446b192a:::benchmark_data_smithi047_486709_object347:head# 2024-09-18T16:05:26.916 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.941 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:448c4d23:::benchmark_data_smithi047_486709_object255:head# 2024-09-18T16:05:26.941 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.966 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:44af3a1c:::benchmark_data_smithi047_486709_object143:head# 2024-09-18T16:05:26.966 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:26.999 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4568b959:::benchmark_data_smithi047_486709_object300:head# 2024-09-18T16:05:26.999 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:27.033 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:456fd22d:::benchmark_data_smithi047_486709_object36:head# 2024-09-18T16:05:27.033 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:27.066 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:45940b80:::benchmark_data_smithi047_486709_object160:head# 2024-09-18T16:05:27.066 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:27.100 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:461042d3:::benchmark_data_smithi047_486709_object254:head# 2024-09-18T16:05:27.100 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:27.133 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4659966f:::benchmark_data_smithi047_486709_object47:head# 2024-09-18T16:05:27.133 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:27.167 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:46f59358:::benchmark_data_smithi047_486709_object201:head# 2024-09-18T16:05:27.167 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:27.200 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:475b1b2b:::benchmark_data_smithi047_486709_object356:head# 2024-09-18T16:05:27.200 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:27.242 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:478b477a:::benchmark_data_smithi047_486709_object136:head# 2024-09-18T16:05:27.242 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:27.292 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:478dfdf5:::benchmark_data_smithi047_486709_object429:head# 2024-09-18T16:05:27.292 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:27.334 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:47a5884a:::benchmark_data_smithi047_486709_object433:head# 2024-09-18T16:05:27.334 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:27.368 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:47a5b05d:::benchmark_data_smithi047_486709_object416:head# 2024-09-18T16:05:27.368 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:27.401 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:47a5d972:::benchmark_data_smithi047_486709_object280:head# 2024-09-18T16:05:27.401 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:27.434 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:47dc2109:::benchmark_data_smithi047_486709_object445:head# 2024-09-18T16:05:27.435 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:27.468 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4805108f:::benchmark_data_smithi047_486709_object259:head# 2024-09-18T16:05:27.468 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:27.501 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:481be0d2:::benchmark_data_smithi047_486709_object52:head# 2024-09-18T16:05:27.502 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:27.535 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4868c009:::benchmark_data_smithi047_486709_object417:head# 2024-09-18T16:05:27.535 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:27.569 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:48882074:::benchmark_data_smithi047_486709_object401:head# 2024-09-18T16:05:27.569 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:27.627 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:494851e1:::benchmark_data_smithi047_486709_object75:head# 2024-09-18T16:05:27.627 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:27.677 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:494d1c4c:::benchmark_data_smithi047_486709_object336:head# 2024-09-18T16:05:27.677 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:27.728 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:49886492:::benchmark_data_smithi047_486709_object70:head# 2024-09-18T16:05:27.728 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:27.761 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4a0bc2c3:::benchmark_data_smithi047_486709_object313:head# 2024-09-18T16:05:27.761 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:27.795 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4a34e00d:::benchmark_data_smithi047_486709_object290:head# 2024-09-18T16:05:27.795 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:27.828 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4a3daa8b:::benchmark_data_smithi047_486709_object343:head# 2024-09-18T16:05:27.828 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:27.861 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4acf9792:::benchmark_data_smithi047_486709_object414:head# 2024-09-18T16:05:27.862 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:27.895 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4b3808de:::benchmark_data_smithi047_486709_object404:head# 2024-09-18T16:05:27.895 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:27.929 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4b6c4826:::benchmark_data_smithi047_486709_object355:head# 2024-09-18T16:05:27.929 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:27.962 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4ccc2f76:::benchmark_data_smithi047_486709_object349:head# 2024-09-18T16:05:27.962 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:27.995 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4cd1475e:::benchmark_data_smithi047_486709_object318:head# 2024-09-18T16:05:27.995 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.029 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4cd3eb30:::benchmark_data_smithi047_486709_object408:head# 2024-09-18T16:05:28.029 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.054 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4d8c261c:::benchmark_data_smithi047_486709_object398:head# 2024-09-18T16:05:28.054 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.079 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4df885cf:::benchmark_data_smithi047_486709_object359:head# 2024-09-18T16:05:28.079 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.104 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4ea03f52:::benchmark_data_smithi047_486709_object165:head# 2024-09-18T16:05:28.104 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.129 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:4f47b96f:::benchmark_data_smithi047_486709_object162:head# 2024-09-18T16:05:28.129 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.154 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5028023b:::benchmark_data_smithi047_486709_object96:head# 2024-09-18T16:05:28.154 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.180 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:504aec17:::benchmark_data_smithi047_486709_object209:head# 2024-09-18T16:05:28.180 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.205 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:50ee368e:::benchmark_data_smithi047_486709_object393:head# 2024-09-18T16:05:28.205 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.230 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:50fe9e36:::benchmark_data_smithi047_486709_object63:head# 2024-09-18T16:05:28.230 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.255 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:51596afe:::benchmark_data_smithi047_486709_object15:head# 2024-09-18T16:05:28.255 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.280 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5252b7a2:::benchmark_data_smithi047_486709_object140:head# 2024-09-18T16:05:28.280 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.305 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5255f4d2:::benchmark_data_smithi047_486709_object31:head# 2024-09-18T16:05:28.305 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.330 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:53c861f9:::benchmark_data_smithi047_486709_object25:head# 2024-09-18T16:05:28.330 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.355 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:540388ee:::benchmark_data_smithi047_486709_object66:head# 2024-09-18T16:05:28.355 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.389 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:54d23911:::benchmark_data_smithi047_486709_object227:head# 2024-09-18T16:05:28.389 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.414 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:54da63c3:::benchmark_data_smithi047_486709_object109:head# 2024-09-18T16:05:28.414 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.448 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5634e19f:::benchmark_data_smithi047_486709_object171:head# 2024-09-18T16:05:28.448 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.473 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5764dce9:::benchmark_data_smithi047_486709_object4:head# 2024-09-18T16:05:28.473 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.498 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:57985501:::benchmark_data_smithi047_486709_object152:head# 2024-09-18T16:05:28.498 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.523 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:57a66a66:::benchmark_data_smithi047_486709_object88:head# 2024-09-18T16:05:28.523 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.548 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:59358cf8:::benchmark_data_smithi047_486709_object431:head# 2024-09-18T16:05:28.548 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.573 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5a56ed48:::benchmark_data_smithi047_486709_object252:head# 2024-09-18T16:05:28.573 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.598 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5d6a3e70:::benchmark_data_smithi047_486709_object198:head# 2024-09-18T16:05:28.598 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.623 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5dc17652:::benchmark_data_smithi047_486709_object155:head# 2024-09-18T16:05:28.623 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.648 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5dde6884:::benchmark_data_smithi047_486709_object435:head# 2024-09-18T16:05:28.649 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.673 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5e6ea9b6:::benchmark_data_smithi047_486709_object231:head# 2024-09-18T16:05:28.674 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.700 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5ecfb9c6:::benchmark_data_smithi047_486709_object385:head# 2024-09-18T16:05:28.700 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.724 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5f39bf22:::benchmark_data_smithi047_486709_object161:head# 2024-09-18T16:05:28.724 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.749 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:5f914756:::benchmark_data_smithi047_486709_object363:head# 2024-09-18T16:05:28.749 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.774 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:603081ee:::benchmark_data_smithi047_486709_object386:head# 2024-09-18T16:05:28.774 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.799 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:61695f17:::benchmark_data_smithi047_486709_object276:head# 2024-09-18T16:05:28.799 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.824 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:61896787:::benchmark_data_smithi047_486709_object210:head# 2024-09-18T16:05:28.824 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.849 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6200b7be:::benchmark_data_smithi047_486709_object232:head# 2024-09-18T16:05:28.850 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.874 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:63dd2c56:::benchmark_data_smithi047_486709_object59:head# 2024-09-18T16:05:28.875 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.900 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:63f8e762:::benchmark_data_smithi047_486709_object189:head# 2024-09-18T16:05:28.900 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.925 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:64b920e4:::benchmark_data_smithi047_486709_object212:head# 2024-09-18T16:05:28.925 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.950 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:662e1d13:::benchmark_data_smithi047_486709_object145:head# 2024-09-18T16:05:28.950 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:28.992 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:66b0b6b6:::benchmark_data_smithi047_486709_object443:head# 2024-09-18T16:05:28.992 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:29.017 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:674a1646:::benchmark_data_smithi047_486709_object46:head# 2024-09-18T16:05:29.017 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:29.042 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:67796b68:::benchmark_data_smithi047_486709_object444:head# 2024-09-18T16:05:29.042 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:29.075 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:68194b01:::benchmark_data_smithi047_486709_object321:head# 2024-09-18T16:05:29.075 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:29.109 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:695a6c0d:::benchmark_data_smithi047_486709_object108:head# 2024-09-18T16:05:29.109 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:29.157 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:69aded39:::benchmark_data_smithi047_486709_object50:head# 2024-09-18T16:05:29.158 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:29.192 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:69bcfb35:::benchmark_data_smithi047_486709_object105:head# 2024-09-18T16:05:29.193 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:29.217 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6a18e542:::benchmark_data_smithi047_486709_object233:head# 2024-09-18T16:05:29.218 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:29.243 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6a729ddd:::benchmark_data_smithi047_486709_object126:head# 2024-09-18T16:05:29.243 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:29.285 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6bb01b51:::benchmark_data_smithi047_486709_object309:head# 2024-09-18T16:05:29.285 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:29.327 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6bfd2096:::benchmark_data_smithi047_486709_object9:head# 2024-09-18T16:05:29.327 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:29.360 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6c8b5c8f:::benchmark_data_smithi047_486709_object304:head# 2024-09-18T16:05:29.360 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:29.393 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6d0a80a2:::benchmark_data_smithi047_486709_object294:head# 2024-09-18T16:05:29.394 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:29.427 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6d3fb942:::benchmark_data_smithi047_486709_object32:head# 2024-09-18T16:05:29.427 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:29.460 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6d462708:::benchmark_data_smithi047_486709_object40:head# 2024-09-18T16:05:29.461 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:29.494 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6e282fc5:::benchmark_data_smithi047_486709_object185:head# 2024-09-18T16:05:29.494 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:29.544 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6e872272:::benchmark_data_smithi047_486709_object21:head# 2024-09-18T16:05:29.544 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:29.580 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6f425c18:::benchmark_data_smithi047_486709_object335:head# 2024-09-18T16:05:29.581 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:29.628 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6ff2695a:::benchmark_data_smithi047_486709_object275:head# 2024-09-18T16:05:29.628 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:29.678 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:6ff6d44c:::benchmark_data_smithi047_486709_object360:head# 2024-09-18T16:05:29.678 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:29.728 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:718fd8da:::benchmark_data_smithi047_486709_object380:head# 2024-09-18T16:05:29.728 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:29.762 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:71a2afe1:::benchmark_data_smithi047_486709_object173:head# 2024-09-18T16:05:29.762 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:29.795 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:72f50112:::benchmark_data_smithi047_486709_object64:head# 2024-09-18T16:05:29.795 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:29.829 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:73b413e6:::benchmark_data_smithi047_486709_object326:head# 2024-09-18T16:05:29.829 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:29.862 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:74ad9f9f:::benchmark_data_smithi047_486709_object426:head# 2024-09-18T16:05:29.862 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:29.895 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:75a01843:::benchmark_data_smithi047_486709_object196:head# 2024-09-18T16:05:29.896 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:29.929 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:75b0fdc6:::benchmark_data_smithi047_486709_object245:head# 2024-09-18T16:05:29.929 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:29.963 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:767caffb:::benchmark_data_smithi047_486709_object217:head# 2024-09-18T16:05:29.963 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:29.996 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:76ac5f7c:::benchmark_data_smithi047_486709_object94:head# 2024-09-18T16:05:29.996 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.029 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:77d8a708:::benchmark_data_smithi047_486709_object194:head# 2024-09-18T16:05:30.030 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.071 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:77f84ff3:::benchmark_data_smithi047_486709_object166:head# 2024-09-18T16:05:30.071 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.105 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:786fd7b3:::benchmark_data_smithi047_486709_object81:head# 2024-09-18T16:05:30.105 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.138 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:78fe554f:::benchmark_data_smithi047_486709_object389:head# 2024-09-18T16:05:30.138 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.171 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:79079601:::benchmark_data_smithi047_486709_object339:head# 2024-09-18T16:05:30.172 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.205 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7a1185f6:::benchmark_data_smithi047_486709_object127:head# 2024-09-18T16:05:30.205 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.238 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7ae8e174:::benchmark_data_smithi047_486709_object199:head# 2024-09-18T16:05:30.239 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.272 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7b489099:::benchmark_data_smithi047_486709_object273:head# 2024-09-18T16:05:30.272 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.297 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7c04e6a5:::benchmark_data_smithi047_486709_object179:head# 2024-09-18T16:05:30.297 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.331 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7c97e866:::benchmark_data_smithi047_486709_object41:head# 2024-09-18T16:05:30.331 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.356 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7dca798b:::benchmark_data_smithi047_486709_object378:head# 2024-09-18T16:05:30.356 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.381 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7deba6af:::benchmark_data_smithi047_486709_object367:head# 2024-09-18T16:05:30.381 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.406 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7e562ab8:::benchmark_data_smithi047_486709_object333:head# 2024-09-18T16:05:30.406 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.432 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7e7d69b4:::benchmark_data_smithi047_486709_object242:head# 2024-09-18T16:05:30.432 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.456 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7ea71f5d:::benchmark_data_smithi047_486709_object240:head# 2024-09-18T16:05:30.456 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.481 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7eca269f:::benchmark_data_smithi047_486709_object112:head# 2024-09-18T16:05:30.482 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.506 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:7ef16b08:::benchmark_data_smithi047_486709_object274:head# 2024-09-18T16:05:30.507 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.532 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8002f347:::benchmark_data_smithi047_486709_object446:head# 2024-09-18T16:05:30.532 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.557 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:80da430d:::benchmark_data_smithi047_486709_object375:head# 2024-09-18T16:05:30.557 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.582 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:83a39f85:::benchmark_data_smithi047_486709_object223:head# 2024-09-18T16:05:30.582 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.607 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8425483e:::benchmark_data_smithi047_486709_object362:head# 2024-09-18T16:05:30.607 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.632 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8482eb7a:::benchmark_data_smithi047_486709_object263:head# 2024-09-18T16:05:30.632 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.657 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:862ff835:::benchmark_data_smithi047_486709_object89:head# 2024-09-18T16:05:30.657 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.682 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8671313c:::benchmark_data_smithi047_486709_object307:head# 2024-09-18T16:05:30.682 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.708 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:86b59ff3:::benchmark_data_smithi047_486709_object271:head# 2024-09-18T16:05:30.708 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.732 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:870d4dc1:::benchmark_data_smithi047_486709_object331:head# 2024-09-18T16:05:30.733 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.758 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:87ed6200:::benchmark_data_smithi047_486709_object129:head# 2024-09-18T16:05:30.758 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.783 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8843183f:::benchmark_data_smithi047_486709_object37:head# 2024-09-18T16:05:30.783 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.808 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8845a7f4:::benchmark_data_smithi047_486709_object308:head# 2024-09-18T16:05:30.808 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.833 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:89858298:::benchmark_data_smithi047_486709_object188:head# 2024-09-18T16:05:30.833 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.858 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:898c3769:::benchmark_data_smithi047_486709_object358:head# 2024-09-18T16:05:30.858 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.883 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8bd776e3:::benchmark_data_smithi047_486709_object270:head# 2024-09-18T16:05:30.883 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.908 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8c32331e:::benchmark_data_smithi047_486709_object87:head# 2024-09-18T16:05:30.909 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.934 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8dc40d6a:::benchmark_data_smithi047_486709_object73:head# 2024-09-18T16:05:30.934 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.959 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8dc4695f:::benchmark_data_smithi047_486709_object48:head# 2024-09-18T16:05:30.959 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:30.984 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8dc4c715:::benchmark_data_smithi047_486709_object439:head# 2024-09-18T16:05:30.984 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:31.009 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8dfd4ffe:::benchmark_data_smithi047_486709_object384:head# 2024-09-18T16:05:31.009 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:31.034 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8e07b728:::benchmark_data_smithi047_486709_object383:head# 2024-09-18T16:05:31.034 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:31.059 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8ee7f414:::benchmark_data_smithi047_486709_object432:head# 2024-09-18T16:05:31.059 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:31.084 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8f5acf4b:::benchmark_data_smithi047_486709_object410:head# 2024-09-18T16:05:31.084 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:31.110 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:8f67f5b1:::benchmark_data_smithi047_486709_object423:head# 2024-09-18T16:05:31.110 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:31.135 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:90fbb92e:::benchmark_data_smithi047_486709_object301:head# 2024-09-18T16:05:31.135 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:31.160 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:90fcc685:::benchmark_data_smithi047_486709_object305:head# 2024-09-18T16:05:31.160 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:31.185 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9159a601:::benchmark_data_smithi047_486709_object374:head# 2024-09-18T16:05:31.185 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:31.210 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:92c38da7:::benchmark_data_smithi047_486709_object19:head# 2024-09-18T16:05:31.210 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:31.239 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9328bdc8:::benchmark_data_smithi047_486709_object235:head# 2024-09-18T16:05:31.239 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:31.277 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:938c62e8:::benchmark_data_smithi047_486709_object159:head# 2024-09-18T16:05:31.277 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:31.319 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:93bdd3e7:::benchmark_data_smithi047_486709_object342:head# 2024-09-18T16:05:31.319 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:31.344 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:93d36327:::benchmark_data_smithi047_486709_object381:head# 2024-09-18T16:05:31.344 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:31.378 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:949174da:::benchmark_data_smithi047_486709_object222:head# 2024-09-18T16:05:31.378 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:31.411 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:95d91041:::benchmark_data_smithi047_486709_object110:head# 2024-09-18T16:05:31.411 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:31.461 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:963da936:::benchmark_data_smithi047_486709_object177:head# 2024-09-18T16:05:31.461 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:31.494 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:96b2e2e9:::benchmark_data_smithi047_486709_object154:head# 2024-09-18T16:05:31.495 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:31.528 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:973fddb5:::benchmark_data_smithi047_486709_object412:head# 2024-09-18T16:05:31.528 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:31.562 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:97431e54:::benchmark_data_smithi047_486709_object219:head# 2024-09-18T16:05:31.562 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:31.620 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:97a0d655:::benchmark_data_smithi047_486709_object34:head# 2024-09-18T16:05:31.620 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:31.670 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9876fff6:::benchmark_data_smithi047_486709_object26:head# 2024-09-18T16:05:31.671 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:31.729 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:99bc712b:::benchmark_data_smithi047_486709_object13:head# 2024-09-18T16:05:31.729 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:31.763 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9a758675:::benchmark_data_smithi047_486709_object11:head# 2024-09-18T16:05:31.763 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:31.796 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9a990753:::benchmark_data_smithi047_486709_object190:head# 2024-09-18T16:05:31.796 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:31.829 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9c153ac6:::benchmark_data_smithi047_486709_object163:head# 2024-09-18T16:05:31.830 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:31.863 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9c33fcbc:::benchmark_data_smithi047_486709_object340:head# 2024-09-18T16:05:31.863 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:31.896 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9c57e2a2:::benchmark_data_smithi047_486709_object332:head# 2024-09-18T16:05:31.897 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:31.930 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9e10d84b:::benchmark_data_smithi047_486709_object97:head# 2024-09-18T16:05:31.930 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:31.963 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9e13bce9:::benchmark_data_smithi047_486709_object128:head# 2024-09-18T16:05:31.963 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.013 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9eeea816:::benchmark_data_smithi047_486709_object368:head# 2024-09-18T16:05:32.014 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.047 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9f669583:::benchmark_data_smithi047_486709_object327:head# 2024-09-18T16:05:32.047 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.080 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:9f9ad351:::benchmark_data_smithi047_486709_object369:head# 2024-09-18T16:05:32.080 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.114 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a030bf98:::benchmark_data_smithi047_486709_object392:head# 2024-09-18T16:05:32.114 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.156 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a08761ad:::benchmark_data_smithi047_486709_object137:head# 2024-09-18T16:05:32.156 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.189 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a08e227f:::benchmark_data_smithi047_486709_object213:head# 2024-09-18T16:05:32.189 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.223 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a0e2b275:::benchmark_data_smithi047_486709_object164:head# 2024-09-18T16:05:32.223 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.256 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a20dfb87:::benchmark_data_smithi047_486709_object406:head# 2024-09-18T16:05:32.256 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.289 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a2465ab3:::benchmark_data_smithi047_486709_object283:head# 2024-09-18T16:05:32.290 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.323 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a26624da:::benchmark_data_smithi047_486709_object202:head# 2024-09-18T16:05:32.323 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.348 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a277cbc5:::benchmark_data_smithi047_486709_object104:head# 2024-09-18T16:05:32.348 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.373 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a37ccba1:::benchmark_data_smithi047_486709_object114:head# 2024-09-18T16:05:32.373 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.398 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a38900dd:::benchmark_data_smithi047_486709_object157:head# 2024-09-18T16:05:32.399 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.423 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a39d29ab:::benchmark_data_smithi047_486709_object354:head# 2024-09-18T16:05:32.423 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.448 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a4bf55fd:::benchmark_data_smithi047_486709_object428:head# 2024-09-18T16:05:32.449 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.474 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a656f25f:::benchmark_data_smithi047_486709_object361:head# 2024-09-18T16:05:32.474 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.499 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a67088c8:::benchmark_data_smithi047_486709_object266:head# 2024-09-18T16:05:32.499 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.524 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a6b65415:::benchmark_data_smithi047_486709_object56:head# 2024-09-18T16:05:32.524 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.549 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a6ba7861:::benchmark_data_smithi047_486709_object95:head# 2024-09-18T16:05:32.549 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.574 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a70bf38a:::benchmark_data_smithi047_486709_object224:head# 2024-09-18T16:05:32.574 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.599 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:a8558376:::benchmark_data_smithi047_486709_object346:head# 2024-09-18T16:05:32.599 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.625 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ab718bbf:::benchmark_data_smithi047_486709_object169:head# 2024-09-18T16:05:32.625 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.649 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ab9ff358:::benchmark_data_smithi047_486709_object261:head# 2024-09-18T16:05:32.650 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.675 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ad1254a4:::benchmark_data_smithi047_486709_object92:head# 2024-09-18T16:05:32.675 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.700 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ad283ca8:::benchmark_data_smithi047_486709_object142:head# 2024-09-18T16:05:32.700 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.725 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ad529a1a:::benchmark_data_smithi047_486709_object27:head# 2024-09-18T16:05:32.725 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.750 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ad696ccc:::benchmark_data_smithi047_486709_object65:head# 2024-09-18T16:05:32.750 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.775 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ada8b08b:::benchmark_data_smithi047_486709_object168:head# 2024-09-18T16:05:32.775 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.801 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:adb2d934:::benchmark_data_smithi047_486709_object1:head# 2024-09-18T16:05:32.801 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.825 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:adc96826:::benchmark_data_smithi047_486709_object234:head# 2024-09-18T16:05:32.825 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.850 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ae847374:::benchmark_data_smithi047_486709_object289:head# 2024-09-18T16:05:32.851 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.876 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:aef68153:::benchmark_data_smithi047_486709_object388:head# 2024-09-18T16:05:32.876 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.901 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:afb67530:::benchmark_data_smithi047_486709_object35:head# 2024-09-18T16:05:32.901 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.926 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b23cee60:::benchmark_data_smithi047_486709_object176:head# 2024-09-18T16:05:32.926 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.951 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b43dea0a:::benchmark_data_smithi047_486709_object193:head# 2024-09-18T16:05:32.951 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:32.976 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b4937444:::benchmark_data_smithi047_486709_object49:head# 2024-09-18T16:05:32.976 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:33.001 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b548fc6c:::benchmark_data_smithi047_486709_object16:head# 2024-09-18T16:05:33.001 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:33.026 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b58d71e5:::benchmark_data_smithi047_486709_object243:head# 2024-09-18T16:05:33.026 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:33.051 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b66fabbe:::benchmark_data_smithi047_486709_object205:head# 2024-09-18T16:05:33.051 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:33.077 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b68f9b67:::benchmark_data_smithi047_486709_object123:head# 2024-09-18T16:05:33.077 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:33.102 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b6bb4500:::benchmark_data_smithi047_486709_object430:head# 2024-09-18T16:05:33.102 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:33.127 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b711ba00:::benchmark_data_smithi047_486709_object54:head# 2024-09-18T16:05:33.127 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:33.152 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b7aa9efd:::benchmark_data_smithi047_486709_object170:head# 2024-09-18T16:05:33.152 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:33.177 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b8001294:::benchmark_data_smithi047_486709_object211:head# 2024-09-18T16:05:33.177 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:33.204 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b8615e71:::benchmark_last_metadata:head# 2024-09-18T16:05:33.204 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:33.227 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b861831e:::benchmark_data_smithi047_486709_object377:head# 2024-09-18T16:05:33.228 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:33.294 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b86d1c91:::benchmark_data_smithi047_486709_object257:head# 2024-09-18T16:05:33.294 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:33.319 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b876a2da:::benchmark_data_smithi047_486709_object68:head# 2024-09-18T16:05:33.319 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:33.345 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b8a243fe:::benchmark_data_smithi047_486709_object62:head# 2024-09-18T16:05:33.345 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:33.369 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b8a76b51:::benchmark_data_smithi047_486709_object324:head# 2024-09-18T16:05:33.370 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:33.403 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b8aa8906:::benchmark_data_smithi047_486709_object379:head# 2024-09-18T16:05:33.403 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:33.428 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b92ada8d:::benchmark_data_smithi047_486709_object174:head# 2024-09-18T16:05:33.428 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:33.453 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:b96cca9d:::benchmark_data_smithi047_486709_object119:head# 2024-09-18T16:05:33.453 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:33.478 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:bba01b0c:::benchmark_data_smithi047_486709_object437:head# 2024-09-18T16:05:33.479 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:33.503 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:bc575efa:::benchmark_data_smithi047_486709_object191:head# 2024-09-18T16:05:33.503 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:33.529 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:be35df85:::benchmark_data_smithi047_486709_object85:head# 2024-09-18T16:05:33.529 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:33.554 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:bef8539c:::benchmark_data_smithi047_486709_object182:head# 2024-09-18T16:05:33.554 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:33.588 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:bf08fd7d:::benchmark_data_smithi047_486709_object215:head# 2024-09-18T16:05:33.589 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:33.638 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:bfebfba3:::benchmark_data_smithi047_486709_object17:head# 2024-09-18T16:05:33.638 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:33.688 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:bffd4c30:::benchmark_data_smithi047_486709_object441:head# 2024-09-18T16:05:33.688 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:33.760 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c01a9a33:::benchmark_data_smithi047_486709_object133:head# 2024-09-18T16:05:33.760 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:33.797 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c04b3582:::benchmark_data_smithi047_486709_object396:head# 2024-09-18T16:05:33.797 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:33.830 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c07e1d98:::benchmark_data_smithi047_486709_object72:head# 2024-09-18T16:05:33.830 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:33.863 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c0aeed8e:::benchmark_data_smithi047_486709_object8:head# 2024-09-18T16:05:33.864 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:33.897 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c0cb7a7f:::benchmark_data_smithi047_486709_object195:head# 2024-09-18T16:05:33.897 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:33.930 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c1154d1b:::benchmark_data_smithi047_486709_object262:head# 2024-09-18T16:05:33.931 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:33.972 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c1311f8e:::benchmark_data_smithi047_486709_object6:head# 2024-09-18T16:05:33.972 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:34.006 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c27edcf0:::benchmark_data_smithi047_486709_object14:head# 2024-09-18T16:05:34.006 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:34.039 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c2bc9557:::benchmark_data_smithi047_486709_object167:head# 2024-09-18T16:05:34.040 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:34.073 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c2cfa1ae:::benchmark_data_smithi047_486709_object122:head# 2024-09-18T16:05:34.073 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:34.106 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c2e4c850:::benchmark_data_smithi047_486709_object310:head# 2024-09-18T16:05:34.107 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:34.140 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c305b228:::benchmark_data_smithi047_486709_object18:head# 2024-09-18T16:05:34.140 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:34.173 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c39807c5:::benchmark_data_smithi047_486709_object186:head# 2024-09-18T16:05:34.173 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:34.207 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c4b18f6d:::benchmark_data_smithi047_486709_object153:head# 2024-09-18T16:05:34.207 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:34.240 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c59bd567:::benchmark_data_smithi047_486709_object147:head# 2024-09-18T16:05:34.240 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:34.273 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c611ed63:::benchmark_data_smithi047_486709_object187:head# 2024-09-18T16:05:34.274 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:34.307 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c69aa8b7:::benchmark_data_smithi047_486709_object382:head# 2024-09-18T16:05:34.307 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:34.340 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c76cb633:::benchmark_data_smithi047_486709_object98:head# 2024-09-18T16:05:34.341 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:34.374 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c77b16ab:::benchmark_data_smithi047_486709_object260:head# 2024-09-18T16:05:34.374 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:34.407 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c7a4be7e:::benchmark_data_smithi047_486709_object83:head# 2024-09-18T16:05:34.407 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:34.441 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c7bb3b1e:::benchmark_data_smithi047_486709_object314:head# 2024-09-18T16:05:34.441 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:34.474 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:c9ce43fe:::benchmark_data_smithi047_486709_object373:head# 2024-09-18T16:05:34.474 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:34.499 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cab07825:::benchmark_data_smithi047_486709_object419:head# 2024-09-18T16:05:34.499 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:34.524 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cb3c780b:::benchmark_data_smithi047_486709_object353:head# 2024-09-18T16:05:34.524 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:34.550 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cbcebd78:::benchmark_data_smithi047_486709_object67:head# 2024-09-18T16:05:34.550 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:34.575 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cc5b2aeb:::benchmark_data_smithi047_486709_object61:head# 2024-09-18T16:05:34.575 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:34.600 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ccf5e326:::benchmark_data_smithi047_486709_object319:head# 2024-09-18T16:05:34.600 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:34.625 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cdc7b216:::benchmark_data_smithi047_486709_object277:head# 2024-09-18T16:05:34.625 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:34.650 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ce245c9e:::benchmark_data_smithi047_486709_object341:head# 2024-09-18T16:05:34.650 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:34.675 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ce3a05b5:::benchmark_data_smithi047_486709_object415:head# 2024-09-18T16:05:34.675 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:34.700 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cea6f1a6:::benchmark_data_smithi047_486709_object183:head# 2024-09-18T16:05:34.700 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:34.725 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ceac51df:::benchmark_data_smithi047_486709_object208:head# 2024-09-18T16:05:34.725 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:34.750 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:cf9d71f7:::benchmark_data_smithi047_486709_object328:head# 2024-09-18T16:05:34.751 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:34.776 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d0e0ddef:::benchmark_data_smithi047_486709_object440:head# 2024-09-18T16:05:34.776 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:34.801 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d13e0997:::benchmark_data_smithi047_486709_object120:head# 2024-09-18T16:05:34.801 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:34.826 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d14c0890:::benchmark_data_smithi047_486709_object29:head# 2024-09-18T16:05:34.826 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:34.851 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d178597d:::benchmark_data_smithi047_486709_object38:head# 2024-09-18T16:05:34.851 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:34.876 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d1a87433:::benchmark_data_smithi047_486709_object101:head# 2024-09-18T16:05:34.876 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:34.901 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d231ee82:::benchmark_data_smithi047_486709_object23:head# 2024-09-18T16:05:34.901 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:34.926 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d2cc77b5:::benchmark_data_smithi047_486709_object411:head# 2024-09-18T16:05:34.926 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:34.951 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d30af311:::benchmark_data_smithi047_486709_object12:head# 2024-09-18T16:05:34.952 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:34.977 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d3689ae3:::benchmark_data_smithi047_486709_object131:head# 2024-09-18T16:05:34.977 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:35.002 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d3af68ed:::benchmark_data_smithi047_486709_object267:head# 2024-09-18T16:05:35.002 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:35.027 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d4d051bb:::benchmark_data_smithi047_486709_object442:head# 2024-09-18T16:05:35.027 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:35.052 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d4eed802:::benchmark_data_smithi047_486709_object418:head# 2024-09-18T16:05:35.052 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:35.077 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d51f7893:::benchmark_data_smithi047_486709_object203:head# 2024-09-18T16:05:35.077 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:35.102 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d55a9f5e:::benchmark_data_smithi047_486709_object282:head# 2024-09-18T16:05:35.102 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:35.127 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d6023347:::benchmark_data_smithi047_486709_object258:head# 2024-09-18T16:05:35.127 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:35.152 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d6507515:::benchmark_data_smithi047_486709_object84:head# 2024-09-18T16:05:35.152 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:35.177 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d67b3f40:::benchmark_data_smithi047_486709_object57:head# 2024-09-18T16:05:35.178 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:35.203 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d985ffbb:::benchmark_data_smithi047_486709_object221:head# 2024-09-18T16:05:35.203 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:35.228 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:d9c64963:::benchmark_data_smithi047_486709_object403:head# 2024-09-18T16:05:35.228 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:35.295 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:db5c1772:::benchmark_data_smithi047_486709_object399:head# 2024-09-18T16:05:35.295 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:35.345 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:dbc75972:::benchmark_data_smithi047_486709_object99:head# 2024-09-18T16:05:35.345 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:35.387 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:dc504976:::benchmark_data_smithi047_486709_object74:head# 2024-09-18T16:05:35.387 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:35.412 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:de0372e0:::benchmark_data_smithi047_486709_object348:head# 2024-09-18T16:05:35.412 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:35.437 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:de4ed419:::benchmark_data_smithi047_486709_object180:head# 2024-09-18T16:05:35.517 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:35.518 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:de9d26f3:::benchmark_data_smithi047_486709_object200:head# 2024-09-18T16:05:35.518 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:35.518 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:dfe26c10:::benchmark_data_smithi047_486709_object250:head# 2024-09-18T16:05:35.518 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:35.518 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:dffb606a:::benchmark_data_smithi047_486709_object86:head# 2024-09-18T16:05:35.518 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:35.538 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e00a47c6:::benchmark_data_smithi047_486709_object93:head# 2024-09-18T16:05:35.538 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:35.563 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e041d288:::benchmark_data_smithi047_486709_object295:head# 2024-09-18T16:05:35.563 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:35.621 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e1aa6946:::benchmark_data_smithi047_486709_object172:head# 2024-09-18T16:05:35.621 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:35.663 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e25d4c6c:::benchmark_data_smithi047_486709_object371:head# 2024-09-18T16:05:35.663 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:35.722 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e26ea5d9:::benchmark_data_smithi047_486709_object214:head# 2024-09-18T16:05:35.722 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:35.764 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e2d269ba:::benchmark_data_smithi047_486709_object351:head# 2024-09-18T16:05:35.764 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:35.807 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e3c63889:::benchmark_data_smithi047_486709_object247:head# 2024-09-18T16:05:35.807 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:35.839 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e436ca17:::benchmark_data_smithi047_486709_object338:head# 2024-09-18T16:05:35.839 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:35.873 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e4deba91:::benchmark_data_smithi047_486709_object28:head# 2024-09-18T16:05:35.873 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:35.906 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e53d80b3:::benchmark_data_smithi047_486709_object387:head# 2024-09-18T16:05:35.907 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:35.940 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e569c738:::benchmark_data_smithi047_486709_object241:head# 2024-09-18T16:05:35.940 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:35.973 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e60de3cc:::benchmark_data_smithi047_486709_object291:head# 2024-09-18T16:05:35.973 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:36.007 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e6a4319b:::benchmark_data_smithi047_486709_object402:head# 2024-09-18T16:05:36.007 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:36.040 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e9784ca2:::benchmark_data_smithi047_486709_object337:head# 2024-09-18T16:05:36.040 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:36.074 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e98a11e8:::benchmark_data_smithi047_486709_object438:head# 2024-09-18T16:05:36.074 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:36.141 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:e9b09430:::benchmark_data_smithi047_486709_object306:head# 2024-09-18T16:05:36.141 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:36.195 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ea1f5cdd:::benchmark_data_smithi047_486709_object80:head# 2024-09-18T16:05:36.195 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:36.241 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:eae3fd7e:::benchmark_data_smithi047_486709_object79:head# 2024-09-18T16:05:36.241 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:36.308 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ebcff5c1:::benchmark_data_smithi047_486709_object226:head# 2024-09-18T16:05:36.308 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:36.358 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ec0cd898:::benchmark_data_smithi047_486709_object253:head# 2024-09-18T16:05:36.358 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:36.417 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ec1827ae:::benchmark_data_smithi047_486709_object7:head# 2024-09-18T16:05:36.417 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:36.450 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ec3b5e98:::benchmark_data_smithi047_486709_object248:head# 2024-09-18T16:05:36.451 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:36.475 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ecdf4765:::benchmark_data_smithi047_486709_object265:head# 2024-09-18T16:05:36.476 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:36.501 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:edc3a588:::benchmark_data_smithi047_486709_object285:head# 2024-09-18T16:05:36.501 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:36.526 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:edcb7a06:::benchmark_data_smithi047_486709_object400:head# 2024-09-18T16:05:36.526 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:36.551 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ee09c109:::benchmark_data_smithi047_486709_object344:head# 2024-09-18T16:05:36.551 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:36.576 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ef289abb:::benchmark_data_smithi047_486709_object158:head# 2024-09-18T16:05:36.576 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:36.601 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ef4a0f3d:::benchmark_data_smithi047_486709_object184:head# 2024-09-18T16:05:36.601 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:36.626 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ef521e3e:::benchmark_data_smithi047_486709_object334:head# 2024-09-18T16:05:36.627 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:36.651 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:efe89b5b:::benchmark_data_smithi047_486709_object135:head# 2024-09-18T16:05:36.652 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:36.677 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f0638365:::benchmark_data_smithi047_486709_object121:head# 2024-09-18T16:05:36.677 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:36.701 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f1a51331:::benchmark_data_smithi047_486709_object102:head# 2024-09-18T16:05:36.702 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:36.727 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f26a1847:::benchmark_data_smithi047_486709_object39:head# 2024-09-18T16:05:36.727 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:36.752 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f35d757b:::benchmark_data_smithi047_486709_object278:head# 2024-09-18T16:05:36.752 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:36.777 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f3fb4ddd:::benchmark_data_smithi047_486709_object279:head# 2024-09-18T16:05:36.777 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:36.802 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f4cea92f:::benchmark_data_smithi047_486709_object424:head# 2024-09-18T16:05:36.802 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:36.827 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f4dd87be:::benchmark_data_smithi047_486709_object60:head# 2024-09-18T16:05:36.827 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:36.852 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f6597b7e:::benchmark_data_smithi047_486709_object229:head# 2024-09-18T16:05:36.852 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:36.877 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f6889cb7:::benchmark_data_smithi047_486709_object24:head# 2024-09-18T16:05:36.877 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:36.902 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f6b9845c:::benchmark_data_smithi047_486709_object130:head# 2024-09-18T16:05:36.903 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:36.928 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f7746cf1:::benchmark_data_smithi047_486709_object288:head# 2024-09-18T16:05:36.928 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:36.953 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f7a12768:::benchmark_data_smithi047_486709_object149:head# 2024-09-18T16:05:36.953 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:36.978 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f8060b73:::benchmark_data_smithi047_486709_object107:head# 2024-09-18T16:05:36.978 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:37.003 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f826a72f:::benchmark_data_smithi047_486709_object317:head# 2024-09-18T16:05:37.003 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:37.028 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f834e6c9:::benchmark_data_smithi047_486709_object204:head# 2024-09-18T16:05:37.028 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:37.053 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:f9a3718e:::benchmark_data_smithi047_486709_object230:head# 2024-09-18T16:05:37.053 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:37.080 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fa43b90e:::benchmark_data_smithi047_486709_object5:head# 2024-09-18T16:05:37.080 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:37.104 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fa71b4b0:::benchmark_data_smithi047_486709_object77:head# 2024-09-18T16:05:37.104 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:37.129 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fa9f4c57:::benchmark_data_smithi047_486709_object281:head# 2024-09-18T16:05:37.129 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:37.154 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fbe77135:::benchmark_data_smithi047_486709_object409:head# 2024-09-18T16:05:37.154 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:37.179 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fc2aef5a:::benchmark_data_smithi047_486709_object256:head# 2024-09-18T16:05:37.179 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:37.204 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fcd8d91e:::benchmark_data_smithi047_486709_object264:head# 2024-09-18T16:05:37.204 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:37.229 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fe24c144:::benchmark_data_smithi047_486709_object55:head# 2024-09-18T16:05:37.229 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:37.313 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fe3019dd:::benchmark_data_smithi047_486709_object100:head# 2024-09-18T16:05:37.313 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:37.363 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fe38134c:::benchmark_data_smithi047_486709_object246:head# 2024-09-18T16:05:37.363 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:37.388 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:fe56a8df:::benchmark_data_smithi047_486709_object175:head# 2024-09-18T16:05:37.388 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:37.413 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ff8c3386:::benchmark_data_smithi047_486709_object45:head# 2024-09-18T16:05:37.413 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:37.438 INFO:tasks.workunit.client.0.smithi047.stdout:Write #1:ffe0aa00:::benchmark_data_smithi047_486709_object287:head# 2024-09-18T16:05:37.438 INFO:tasks.workunit.client.0.smithi047.stdout:snapset 0=[]:{} 2024-09-18T16:05:37.497 INFO:tasks.workunit.client.0.smithi047.stdout:write_pg epoch 8 info 1.0( v 8'449 (0'0,8'449] local-lis/les=6/7 n=449 ec=6/6 lis/c=6/6 les/c/f=7/7/0 sis=6) 2024-09-18T16:05:37.530 INFO:tasks.workunit.client.0.smithi047.stdout:Import successful 2024-09-18T16:05:38.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:193: TEST_import_after_split: activate_osd td/pg-split-merge 0 2024-09-18T16:05:38.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-09-18T16:05:38.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T16:05:38.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-18T16:05:38.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T16:05:38.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/0 2024-09-18T16:05:38.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1a671e43-8866-44a5-abe3-c1e1f4ff765b --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-18T16:05:38.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:05:38.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:05:38.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:05:38.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-09-18T16:05:38.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-09-18T16:05:38.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T16:05:38.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T16:05:38.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-09-18T16:05:38.050 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T16:05:38.050 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:05:38.050 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:05:38.050 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:05:38.050 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:05:38.050 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.463129 2024-09-18T16:05:38.050 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.463129/$cluster-$name.asok' 2024-09-18T16:05:38.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.463129/$cluster-$name.asok' 2024-09-18T16:05:38.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:05:38.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-09-18T16:05:38.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-09-18T16:05:38.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:05:38.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:05:38.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:05:38.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:05:38.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T16:05:38.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T16:05:38.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/0 2024-09-18T16:05:38.052 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T16:05:38.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-18T16:05:38.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1a671e43-8866-44a5-abe3-c1e1f4ff765b --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.463129/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:05:38.053 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/0/whoami 2024-09-18T16:05:38.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-18T16:05:38.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T16:05:38.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T16:05:38.053 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T16:05:38.078 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:05:38.077+0000 7fa2b1d1e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:05:38.080 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:05:38.079+0000 7fa2b1d1e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:05:38.081 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:05:38.081+0000 7fa2b1d1e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:05:38.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-18T16:05:38.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:05:38.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T16:05:38.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:05:38.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:05:38.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:38.428 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:05:38.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:05:38.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:38.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:05:38.599 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:05:38.598+0000 7fa2b1d1e440 -1 Falling back to public interface 2024-09-18T16:05:38.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:05:39.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:05:39.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:39.831 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T16:05:39.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:05:39.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:39.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:05:40.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:05:41.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:05:41.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:41.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:05:41.275 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T16:05:41.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:41.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:05:41.648 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:05:41.927 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:05:41.926+0000 7fa2b1d1e440 -1 osd.0 17 log_to_monitors true 2024-09-18T16:05:42.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:05:42.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:42.649 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:05:42.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:05:42.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:42.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:05:43.076 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 20 up_thru 20 down_at 18 last_clean_interval [11,17) [v2:127.0.0.1:6802/489497,v1:127.0.0.1:6803/489497] [v2:127.0.0.1:6804/489497,v1:127.0.0.1:6805/489497] exists,up be7e9a0f-d00f-4cf4-aaef-94da8b714cda 2024-09-18T16:05:43.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:05:43.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:05:43.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:05:43.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:195: TEST_import_after_split: wait_for_clean 2024-09-18T16:05:43.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:05:43.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:05:43.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:05:43.078 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:05:43.078 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:05:43.078 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:05:43.078 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:05:43.078 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:05:43.078 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:05:43.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T16:05:43.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:05:43.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:05:43.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:05:43.167 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:05:43.167 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:05:43.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-18T16:05:43.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:05:43.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:05:43.578 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:05:43.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345923 2024-09-18T16:05:43.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345923 2024-09-18T16:05:43.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-85899345923' 2024-09-18T16:05:43.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:05:43.689 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-85899345923 2024-09-18T16:05:43.689 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:05:43.690 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:05:43.690 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-85899345923 2024-09-18T16:05:43.690 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:05:43.691 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345923 2024-09-18T16:05:43.692 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 85899345923 2024-09-18T16:05:43.692 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 85899345923' 2024-09-18T16:05:43.692 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:05:44.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 85899345923 2024-09-18T16:05:44.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:05:45.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:05:45.089 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:05:45.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345923 -lt 85899345923 2024-09-18T16:05:45.480 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:05:45.481 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:05:45.481 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:05:45.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-09-18T16:05:45.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:05:45.988 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:05:45.988 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:05:45.988 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:05:45.988 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:05:45.988 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:05:45.988 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:05:46.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-18T16:05:46.377 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:05:46.378 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:05:46.378 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:05:46.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-09-18T16:05:46.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:05:46.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:05:46.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:17: run: teardown td/pg-split-merge 2024-09-18T16:05:46.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-09-18T16:05:46.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:05:46.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-09-18T16:05:46.878 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:05:46.878 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:05:46.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:05:46.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:05:46.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:05:46.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:05:46.981 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:05:46.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:05:46.982 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:05:46.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:05:46.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:05:46.984 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:05:46.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:05:46.985 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:05:46.985 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:05:46.985 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:05:46.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:05:46.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:05:46.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-09-18T16:05:46.996 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:05:46.996 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:05:46.996 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.463129 2024-09-18T16:05:46.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.463129 2024-09-18T16:05:46.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:05:46.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:05:46.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T16:05:46.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/pg-split-merge 0 2024-09-18T16:05:46.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-09-18T16:05:46.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T16:05:46.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-09-18T16:05:46.999 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:05:46.999 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:05:46.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:05:46.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:05:46.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:05:46.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:05:47.000 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:05:47.000 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:05:47.001 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:05:47.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:05:47.002 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:05:47.002 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:05:47.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:05:47.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:05:47.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:05:47.003 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:05:47.004 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:05:47.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T16:05:47.005 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-09-18T16:05:47.006 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:05:47.006 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:05:47.007 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.463129 2024-09-18T16:05:47.007 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.463129 2024-09-18T16:05:47.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:05:47.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:05:47.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T16:05:47.009 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T16:05:47.009 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T16:05:47.073 INFO:tasks.workunit:Running workunit osd/repeer-on-acting-back.sh... 2024-09-18T16:05:47.074 DEBUG:teuthology.orchestra.run.smithi047:workunit test osd/repeer-on-acting-back.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh 2024-09-18T16:05:47.136 INFO:tasks.workunit.client.0.smithi047.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T16:05:47.142 INFO:tasks.workunit.client.0.smithi047.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T16:05:47.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T16:05:47.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T16:05:47.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T16:05:47.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T16:05:47.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T16:05:47.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T16:05:47.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T16:05:47.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T16:05:47.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/repeer-on-acting-back 2024-09-18T16:05:47.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:22: run: local dir=td/repeer-on-acting-back 2024-09-18T16:05:47.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:23: run: shift 2024-09-18T16:05:47.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:25: run: export poolname=test 2024-09-18T16:05:47.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:25: run: poolname=test 2024-09-18T16:05:47.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:26: run: export testobjects=100 2024-09-18T16:05:47.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:26: run: testobjects=100 2024-09-18T16:05:47.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:27: run: export loglen=12 2024-09-18T16:05:47.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:27: run: loglen=12 2024-09-18T16:05:47.144 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:28: run: expr 12 / 2 2024-09-18T16:05:47.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:28: run: export trim=6 2024-09-18T16:05:47.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:28: run: trim=6 2024-09-18T16:05:47.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:29: run: export CEPH_MON=127.0.0.1:7115 2024-09-18T16:05:47.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:29: run: CEPH_MON=127.0.0.1:7115 2024-09-18T16:05:47.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:30: run: export CEPH_ARGS 2024-09-18T16:05:47.144 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:31: run: uuidgen 2024-09-18T16:05:47.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:31: run: CEPH_ARGS+='--fsid=21c9ff2e-9d52-44eb-80ae-e48bd7727b72 --auth-supported=none ' 2024-09-18T16:05:47.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:32: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2024-09-18T16:05:47.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:34: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2024-09-18T16:05:47.146 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:36: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T16:05:47.146 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:38: run: set 2024-09-18T16:05:47.146 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:38: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-18T16:05:47.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:38: run: local funcs=TEST_repeer_on_down_acting_member_coming_back 2024-09-18T16:05:47.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:39: run: for func in $funcs 2024-09-18T16:05:47.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:40: run: setup td/repeer-on-acting-back 2024-09-18T16:05:47.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/repeer-on-acting-back 2024-09-18T16:05:47.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/repeer-on-acting-back 2024-09-18T16:05:47.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repeer-on-acting-back 2024-09-18T16:05:47.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:05:47.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repeer-on-acting-back KILL 2024-09-18T16:05:47.147 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:05:47.147 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:05:47.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:05:47.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:05:47.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:05:47.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:05:47.148 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:05:47.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:05:47.149 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:05:47.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:05:47.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:05:47.149 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:05:47.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:05:47.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:05:47.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:05:47.150 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:05:47.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:05:47.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:05:47.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repeer-on-acting-back 2024-09-18T16:05:47.153 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:05:47.153 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:05:47.153 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.490363 2024-09-18T16:05:47.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.490363 2024-09-18T16:05:47.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:05:47.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:05:47.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/repeer-on-acting-back 2024-09-18T16:05:47.156 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T16:05:47.156 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:05:47.156 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.490363 2024-09-18T16:05:47.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.490363 2024-09-18T16:05:47.157 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T16:05:47.157 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T16:05:47.157 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T16:05:47.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T16:05:47.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/repeer-on-acting-back 1' TERM HUP INT 2024-09-18T16:05:47.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:41: run: TEST_repeer_on_down_acting_member_coming_back td/repeer-on-acting-back 2024-09-18T16:05:47.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:48: TEST_repeer_on_down_acting_member_coming_back: local dir=td/repeer-on-acting-back 2024-09-18T16:05:47.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:49: TEST_repeer_on_down_acting_member_coming_back: local dummyfile=/etc/fstab 2024-09-18T16:05:47.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:51: TEST_repeer_on_down_acting_member_coming_back: local num_osds=6 2024-09-18T16:05:47.158 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:52: TEST_repeer_on_down_acting_member_coming_back: expr 6 - 1 2024-09-18T16:05:47.159 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:52: TEST_repeer_on_down_acting_member_coming_back: seq 0 5 2024-09-18T16:05:47.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:52: TEST_repeer_on_down_acting_member_coming_back: local 'osds=0 2024-09-18T16:05:47.159 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T16:05:47.159 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T16:05:47.160 INFO:tasks.workunit.client.0.smithi047.stderr:3 2024-09-18T16:05:47.160 INFO:tasks.workunit.client.0.smithi047.stderr:4 2024-09-18T16:05:47.160 INFO:tasks.workunit.client.0.smithi047.stderr:5' 2024-09-18T16:05:47.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:53: TEST_repeer_on_down_acting_member_coming_back: run_mon td/repeer-on-acting-back a 2024-09-18T16:05:47.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/repeer-on-acting-back 2024-09-18T16:05:47.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T16:05:47.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T16:05:47.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T16:05:47.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/repeer-on-acting-back/a 2024-09-18T16:05:47.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/repeer-on-acting-back/a --run-dir=td/repeer-on-acting-back 2024-09-18T16:05:47.487 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T16:05:47.488 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:05:47.488 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:05:47.488 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:05:47.488 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:05:47.488 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.490363 2024-09-18T16:05:47.488 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.490363/$cluster-$name.asok' 2024-09-18T16:05:47.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/repeer-on-acting-back/a '--log-file=td/repeer-on-acting-back/$name.log' '--admin-socket=/tmp/ceph-asok.490363/$cluster-$name.asok' --mon-cluster-log-file=td/repeer-on-acting-back/log --run-dir=td/repeer-on-acting-back '--pid-file=td/repeer-on-acting-back/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T16:05:47.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T16:05:47.773 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T16:05:47.773 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:05:47.773 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:05:47.773 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T16:05:47.773 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T16:05:47.773 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:05:47.774 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:05:47.774 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:05:47.774 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:05:47.774 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:05:47.774 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.490363 2024-09-18T16:05:47.774 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.490363/ceph-mon.a.asok 2024-09-18T16:05:47.774 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:05:47.774 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.490363/ceph-mon.a.asok config get fsid 2024-09-18T16:05:47.852 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T16:05:47.852 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:05:47.852 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:05:47.852 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T16:05:47.852 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T16:05:47.852 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:05:47.852 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:05:47.852 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:05:47.852 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:05:47.853 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:05:47.853 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.490363 2024-09-18T16:05:47.853 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.490363/ceph-mon.a.asok 2024-09-18T16:05:47.853 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:05:47.853 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.490363/ceph-mon.a.asok config get mon_host 2024-09-18T16:05:47.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:54: TEST_repeer_on_down_acting_member_coming_back: run_mgr td/repeer-on-acting-back x 2024-09-18T16:05:47.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/repeer-on-acting-back 2024-09-18T16:05:47.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T16:05:47.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T16:05:47.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T16:05:47.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/repeer-on-acting-back/x 2024-09-18T16:05:47.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T16:05:48.760 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T16:05:48.760 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:05:48.760 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:05:48.760 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:05:48.760 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:05:48.760 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.490363 2024-09-18T16:05:48.760 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.490363/$cluster-$name.asok' 2024-09-18T16:05:48.760 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:05:48.761 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/repeer-on-acting-back/x '--log-file=td/repeer-on-acting-back/$name.log' '--admin-socket=/tmp/ceph-asok.490363/$cluster-$name.asok' --run-dir=td/repeer-on-acting-back '--pid-file=td/repeer-on-acting-back/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:05:48.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-09-18T16:05:48.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 0 2024-09-18T16:05:48.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-09-18T16:05:48.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:05:48.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T16:05:48.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:05:48.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/0 2024-09-18T16:05:48.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=21c9ff2e-9d52-44eb-80ae-e48bd7727b72 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T16:05:48.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:05:48.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:05:48.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:05:48.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/0' 2024-09-18T16:05:48.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/0/journal' 2024-09-18T16:05:48.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:05:48.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:05:48.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-09-18T16:05:48.792 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:05:48.792 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:05:48.792 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:05:48.792 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:05:48.792 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:05:48.792 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.490363 2024-09-18T16:05:48.792 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.490363/$cluster-$name.asok' 2024-09-18T16:05:48.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.490363/$cluster-$name.asok' 2024-09-18T16:05:48.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:05:48.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:05:48.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:05:48.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-09-18T16:05:48.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-09-18T16:05:48.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:05:48.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:05:48.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:05:48.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:05:48.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:05:48.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:05:48.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/0 2024-09-18T16:05:48.793 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:05:48.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=014351de-4c9a-4973-8d37-ccaae696e8d8 2024-09-18T16:05:48.794 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 014351de-4c9a-4973-8d37-ccaae696e8d8 2024-09-18T16:05:48.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 014351de-4c9a-4973-8d37-ccaae696e8d8' 2024-09-18T16:05:48.794 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:05:48.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBc+upmHhg+MBAAQ+ilxB4qFVBJW/lvUWoHiw== 2024-09-18T16:05:48.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBc+upmHhg+MBAAQ+ilxB4qFVBJW/lvUWoHiw=="}' 2024-09-18T16:05:48.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 014351de-4c9a-4973-8d37-ccaae696e8d8 -i td/repeer-on-acting-back/0/new.json 2024-09-18T16:05:49.154 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:05:49.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/0/new.json 2024-09-18T16:05:49.164 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=21c9ff2e-9d52-44eb-80ae-e48bd7727b72 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/0 --osd-journal=td/repeer-on-acting-back/0/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.490363/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBc+upmHhg+MBAAQ+ilxB4qFVBJW/lvUWoHiw== --osd-uuid 014351de-4c9a-4973-8d37-ccaae696e8d8 2024-09-18T16:05:49.195 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:05:49.194+0000 7fb3b7d21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:05:49.198 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:05:49.197+0000 7fb3b7d21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:05:49.199 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:05:49.198+0000 7fb3b7d21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:05:49.199 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:05:49.198+0000 7fb3b7d21440 -1 bluestore(td/repeer-on-acting-back/0/block) _read_bdev_label failed to open td/repeer-on-acting-back/0/block: (2) No such file or directory 2024-09-18T16:05:49.199 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:05:49.198+0000 7fb3b7d21440 -1 bluestore(td/repeer-on-acting-back/0/block) _read_bdev_label failed to open td/repeer-on-acting-back/0/block: (2) No such file or directory 2024-09-18T16:05:49.199 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:05:49.198+0000 7fb3b7d21440 -1 bluestore(td/repeer-on-acting-back/0/block) _read_bdev_label failed to open td/repeer-on-acting-back/0/block: (2) No such file or directory 2024-09-18T16:05:49.238 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:05:49.237+0000 7fb3b7d21440 -1 bluestore(td/repeer-on-acting-back/0) _read_fsid unparsable uuid 2024-09-18T16:05:54.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/0/keyring 2024-09-18T16:05:54.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:05:54.506 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T16:05:54.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T16:05:54.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:05:55.029 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T16:05:55.029 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T16:05:55.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=21c9ff2e-9d52-44eb-80ae-e48bd7727b72 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/0 --osd-journal=td/repeer-on-acting-back/0/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.490363/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:05:55.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:05:55.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:05:55.030 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:05:55.057 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:05:55.056+0000 7f766e040440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:05:55.059 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:05:55.058+0000 7f766e040440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:05:55.060 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:05:55.059+0000 7f766e040440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:05:55.457 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T16:05:55.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:05:55.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T16:05:55.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:05:55.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:05:55.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:55.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:05:55.458 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:05:55.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:55.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:05:55.602 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:05:55.601+0000 7f766e040440 -1 Falling back to public interface 2024-09-18T16:05:55.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:05:56.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:05:56.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:56.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:05:56.869 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T16:05:56.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:56.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:05:57.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:05:58.074 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:05:58.072+0000 7f766e040440 -1 osd.0 0 log_to_monitors true 2024-09-18T16:05:58.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:05:58.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:58.253 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T16:05:58.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:05:58.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:58.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:05:58.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:05:59.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:05:59.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:05:59.670 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:05:59.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:05:59.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:05:59.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:06:00.062 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:01.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:01.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:01.064 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T16:06:01.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:06:01.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:01.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:06:01.567 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:02.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:02.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:02.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:06:02.569 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T16:06:02.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:02.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:06:03.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:04.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:04.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:04.038 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T16:06:04.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:06:04.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:04.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:06:04.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:05.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:05.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:05.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:06:05.453 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T16:06:05.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:05.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:06:05.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:06.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:06.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:06.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:06:06.871 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T16:06:06.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:06.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:06:07.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:08.294 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T16:06:08.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:08.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:08.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:06:08.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:08.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:06:08.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:09.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:09.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:09.722 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T16:06:09.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:06:09.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:09.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:06:10.419 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:11.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:11.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:11.422 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T16:06:11.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:06:11.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:11.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:06:12.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:13.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:13.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:13.382 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T16:06:13.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:06:13.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:13.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:06:14.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:15.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:15.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:15.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:06:15.126 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T16:06:15.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:15.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:06:15.910 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:16.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:16.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:16.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:06:16.912 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T16:06:16.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:16.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:06:17.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:18.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:18.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:18.455 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T16:06:18.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:06:18.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:18.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:06:18.943 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:19.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:19.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:19.945 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T16:06:19.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:06:19.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:19.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:06:20.381 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/491457,v1:127.0.0.1:6803/491457] [v2:127.0.0.1:6804/491457,v1:127.0.0.1:6805/491457] exists,up 014351de-4c9a-4973-8d37-ccaae696e8d8 2024-09-18T16:06:20.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:06:20.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:06:20.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:06:20.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-09-18T16:06:20.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 1 2024-09-18T16:06:20.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-09-18T16:06:20.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:06:20.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T16:06:20.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:06:20.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/1 2024-09-18T16:06:20.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=21c9ff2e-9d52-44eb-80ae-e48bd7727b72 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T16:06:20.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:06:20.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:06:20.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:06:20.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/1' 2024-09-18T16:06:20.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/1/journal' 2024-09-18T16:06:20.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:06:20.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:06:20.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-09-18T16:06:20.386 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:06:20.386 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:06:20.386 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:06:20.386 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:06:20.386 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:06:20.386 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.490363 2024-09-18T16:06:20.386 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.490363/$cluster-$name.asok' 2024-09-18T16:06:20.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.490363/$cluster-$name.asok' 2024-09-18T16:06:20.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:06:20.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:06:20.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:06:20.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-09-18T16:06:20.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-09-18T16:06:20.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:06:20.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:06:20.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:06:20.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:06:20.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:06:20.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:06:20.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/1 2024-09-18T16:06:20.387 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:06:20.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=262ecd85-e589-4e72-b73a-65c457f84832 2024-09-18T16:06:20.388 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 262ecd85-e589-4e72-b73a-65c457f84832 2024-09-18T16:06:20.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 262ecd85-e589-4e72-b73a-65c457f84832' 2024-09-18T16:06:20.388 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:06:20.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB8+upmeZz9FxAAM5bmrUCjsuiVkLonQ6OSTQ== 2024-09-18T16:06:20.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB8+upmeZz9FxAAM5bmrUCjsuiVkLonQ6OSTQ=="}' 2024-09-18T16:06:20.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 262ecd85-e589-4e72-b73a-65c457f84832 -i td/repeer-on-acting-back/1/new.json 2024-09-18T16:06:20.947 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T16:06:20.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/1/new.json 2024-09-18T16:06:20.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=21c9ff2e-9d52-44eb-80ae-e48bd7727b72 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/1 --osd-journal=td/repeer-on-acting-back/1/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.490363/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB8+upmeZz9FxAAM5bmrUCjsuiVkLonQ6OSTQ== --osd-uuid 262ecd85-e589-4e72-b73a-65c457f84832 2024-09-18T16:06:20.989 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:06:20.988+0000 7fe75d820440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:06:20.991 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:06:20.990+0000 7fe75d820440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:06:20.992 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:06:20.991+0000 7fe75d820440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:06:20.992 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:06:20.991+0000 7fe75d820440 -1 bluestore(td/repeer-on-acting-back/1/block) _read_bdev_label failed to open td/repeer-on-acting-back/1/block: (2) No such file or directory 2024-09-18T16:06:20.992 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:06:20.991+0000 7fe75d820440 -1 bluestore(td/repeer-on-acting-back/1/block) _read_bdev_label failed to open td/repeer-on-acting-back/1/block: (2) No such file or directory 2024-09-18T16:06:20.992 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:06:20.991+0000 7fe75d820440 -1 bluestore(td/repeer-on-acting-back/1/block) _read_bdev_label failed to open td/repeer-on-acting-back/1/block: (2) No such file or directory 2024-09-18T16:06:21.029 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:06:21.028+0000 7fe75d820440 -1 bluestore(td/repeer-on-acting-back/1) _read_fsid unparsable uuid 2024-09-18T16:06:26.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/1/keyring 2024-09-18T16:06:26.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:06:26.576 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T16:06:26.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T16:06:26.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:06:27.119 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T16:06:27.119 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T16:06:27.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=21c9ff2e-9d52-44eb-80ae-e48bd7727b72 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/1 --osd-journal=td/repeer-on-acting-back/1/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.490363/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:06:27.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:06:27.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:06:27.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:06:27.148 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:06:27.146+0000 7f8425c0d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:06:27.149 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:06:27.149+0000 7f8425c0d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:06:27.152 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:06:27.150+0000 7f8425c0d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:06:27.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T16:06:27.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:06:27.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T16:06:27.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:06:27.565 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:06:27.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:27.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:06:27.566 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:06:27.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:27.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:27.670 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:06:27.669+0000 7f8425c0d440 -1 Falling back to public interface 2024-09-18T16:06:27.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:28.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:28.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:28.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:06:28.976 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T16:06:28.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:28.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:29.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:30.174 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:06:30.172+0000 7f8425c0d440 -1 osd.1 0 log_to_monitors true 2024-09-18T16:06:30.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:30.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:30.403 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:06:30.403 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T16:06:30.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:30.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:30.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:31.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:31.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:31.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:06:31.968 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:06:31.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:31.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:32.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:33.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:33.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:33.379 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T16:06:33.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:06:33.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:33.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:33.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:34.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:34.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:34.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:06:34.988 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T16:06:34.988 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:34.988 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:35.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:36.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:36.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:36.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:06:36.442 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T16:06:36.442 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:36.443 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:36.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:37.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:37.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:37.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:06:37.908 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T16:06:37.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:37.909 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:38.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:39.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:39.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:39.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:06:39.292 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T16:06:39.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:39.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:39.699 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:40.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:40.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:40.701 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T16:06:40.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:06:40.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:40.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:41.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:42.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:42.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:42.259 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T16:06:42.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:06:42.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:42.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:43.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:44.096 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:44.096 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:44.096 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:06:44.096 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T16:06:44.096 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:44.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:45.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:46.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:46.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:46.106 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T16:06:46.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:06:46.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:46.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:46.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:47.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:47.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:47.784 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T16:06:47.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:06:47.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:47.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:48.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:49.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:49.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:49.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:06:49.501 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T16:06:49.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:49.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:50.199 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:51.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:51.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:51.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:06:51.201 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T16:06:51.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:51.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:52.022 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:06:53.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:06:53.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:06:53.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:06:53.024 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T16:06:53.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:06:53.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:06:53.423 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/493186,v1:127.0.0.1:6811/493186] [v2:127.0.0.1:6812/493186,v1:127.0.0.1:6813/493186] exists,up 262ecd85-e589-4e72-b73a-65c457f84832 2024-09-18T16:06:53.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:06:53.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:06:53.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:06:53.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-09-18T16:06:53.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 2 2024-09-18T16:06:53.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-09-18T16:06:53.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:06:53.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T16:06:53.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:06:53.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/2 2024-09-18T16:06:53.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=21c9ff2e-9d52-44eb-80ae-e48bd7727b72 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T16:06:53.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:06:53.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:06:53.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:06:53.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/2' 2024-09-18T16:06:53.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/2/journal' 2024-09-18T16:06:53.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:06:53.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:06:53.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-09-18T16:06:53.428 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:06:53.428 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:06:53.428 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:06:53.428 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:06:53.428 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:06:53.429 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.490363 2024-09-18T16:06:53.429 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.490363/$cluster-$name.asok' 2024-09-18T16:06:53.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.490363/$cluster-$name.asok' 2024-09-18T16:06:53.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:06:53.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:06:53.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:06:53.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-09-18T16:06:53.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-09-18T16:06:53.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:06:53.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:06:53.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:06:53.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:06:53.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:06:53.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:06:53.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/2 2024-09-18T16:06:53.430 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:06:53.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bcf2c02f-a273-4382-b5a4-92e83836c699 2024-09-18T16:06:53.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 bcf2c02f-a273-4382-b5a4-92e83836c699' 2024-09-18T16:06:53.431 INFO:tasks.workunit.client.0.smithi047.stdout:add osd2 bcf2c02f-a273-4382-b5a4-92e83836c699 2024-09-18T16:06:53.431 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:06:53.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCd+upmvM2CGhAAeMBqZW1RQfzyXR76Jz5/iQ== 2024-09-18T16:06:53.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCd+upmvM2CGhAAeMBqZW1RQfzyXR76Jz5/iQ=="}' 2024-09-18T16:06:53.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bcf2c02f-a273-4382-b5a4-92e83836c699 -i td/repeer-on-acting-back/2/new.json 2024-09-18T16:06:53.930 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T16:06:53.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/2/new.json 2024-09-18T16:06:53.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=21c9ff2e-9d52-44eb-80ae-e48bd7727b72 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/2 --osd-journal=td/repeer-on-acting-back/2/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.490363/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCd+upmvM2CGhAAeMBqZW1RQfzyXR76Jz5/iQ== --osd-uuid bcf2c02f-a273-4382-b5a4-92e83836c699 2024-09-18T16:06:53.972 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:06:53.971+0000 7f9bba323440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:06:53.974 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:06:53.973+0000 7f9bba323440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:06:53.975 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:06:53.974+0000 7f9bba323440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:06:53.976 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:06:53.975+0000 7f9bba323440 -1 bluestore(td/repeer-on-acting-back/2/block) _read_bdev_label failed to open td/repeer-on-acting-back/2/block: (2) No such file or directory 2024-09-18T16:06:53.976 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:06:53.975+0000 7f9bba323440 -1 bluestore(td/repeer-on-acting-back/2/block) _read_bdev_label failed to open td/repeer-on-acting-back/2/block: (2) No such file or directory 2024-09-18T16:06:53.976 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:06:53.975+0000 7f9bba323440 -1 bluestore(td/repeer-on-acting-back/2/block) _read_bdev_label failed to open td/repeer-on-acting-back/2/block: (2) No such file or directory 2024-09-18T16:06:54.020 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:06:54.019+0000 7f9bba323440 -1 bluestore(td/repeer-on-acting-back/2) _read_fsid unparsable uuid 2024-09-18T16:07:00.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/2/keyring 2024-09-18T16:07:00.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:07:00.102 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd2 key to auth repository 2024-09-18T16:07:00.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T16:07:00.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:07:00.620 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T16:07:00.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T16:07:00.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=21c9ff2e-9d52-44eb-80ae-e48bd7727b72 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/2 --osd-journal=td/repeer-on-acting-back/2/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.490363/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:07:00.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:07:00.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:07:00.621 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:07:00.650 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:07:00.649+0000 7fd22900c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:07:00.668 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:07:00.667+0000 7fd22900c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:07:00.669 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:07:00.669+0000 7fd22900c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:07:01.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T16:07:01.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:07:01.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T16:07:01.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:07:01.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:07:01.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:01.064 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:07:01.065 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:07:01.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:01.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:07:01.202 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:07:01.201+0000 7fd22900c440 -1 Falling back to public interface 2024-09-18T16:07:01.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:02.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:02.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:02.440 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T16:07:02.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:07:02.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:02.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:07:02.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:03.619 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:07:03.617+0000 7fd22900c440 -1 osd.2 0 log_to_monitors true 2024-09-18T16:07:03.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:03.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:03.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:07:03.953 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T16:07:03.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:03.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:07:04.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:05.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:05.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:05.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:07:05.383 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:07:05.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:05.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:07:05.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:06.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:06.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:06.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:07:06.834 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T16:07:06.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:06.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:07:07.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:08.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:08.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:08.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:07:08.223 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T16:07:08.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:08.224 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:07:08.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:09.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:09.730 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:09.731 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T16:07:09.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:07:09.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:09.731 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:07:10.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:11.135 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:11.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:11.136 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T16:07:11.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:07:11.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:11.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:07:11.616 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:12.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:12.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:12.618 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T16:07:12.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:07:12.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:12.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:07:13.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:14.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:14.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:14.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:07:14.078 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T16:07:14.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:14.078 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:07:14.504 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:15.505 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:15.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:15.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:07:15.506 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T16:07:15.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:15.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:07:16.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:17.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:17.153 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:17.154 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T16:07:17.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:07:17.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:17.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:07:17.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:18.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:18.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:18.980 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T16:07:18.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:07:18.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:18.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:07:19.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:20.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:20.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:20.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:07:20.736 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T16:07:20.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:20.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:07:21.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:22.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:22.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:22.480 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T16:07:22.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:07:22.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:22.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:07:22.861 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:23.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:23.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:23.863 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T16:07:23.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:07:23.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:23.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:07:24.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:25.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:25.655 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:25.655 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T16:07:25.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:07:25.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:25.656 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:07:26.241 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:27.242 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:27.242 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:27.242 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:07:27.242 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T16:07:27.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:27.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:07:27.666 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/494915,v1:127.0.0.1:6819/494915] [v2:127.0.0.1:6820/494915,v1:127.0.0.1:6821/494915] exists,up bcf2c02f-a273-4382-b5a4-92e83836c699 2024-09-18T16:07:27.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:07:27.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:07:27.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:07:27.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-09-18T16:07:27.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 3 2024-09-18T16:07:27.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-09-18T16:07:27.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:07:27.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-18T16:07:27.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:07:27.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/3 2024-09-18T16:07:27.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=21c9ff2e-9d52-44eb-80ae-e48bd7727b72 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T16:07:27.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:07:27.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:07:27.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:07:27.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/3' 2024-09-18T16:07:27.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/3/journal' 2024-09-18T16:07:27.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:07:27.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:07:27.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-09-18T16:07:27.672 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:07:27.672 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:07:27.672 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:07:27.672 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:07:27.672 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:07:27.672 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.490363 2024-09-18T16:07:27.672 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.490363/$cluster-$name.asok' 2024-09-18T16:07:27.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.490363/$cluster-$name.asok' 2024-09-18T16:07:27.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:07:27.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:07:27.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:07:27.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-09-18T16:07:27.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-09-18T16:07:27.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:07:27.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:07:27.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:07:27.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:07:27.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:07:27.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:07:27.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/3 2024-09-18T16:07:27.673 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:07:27.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=840fbf9a-6577-4496-87d4-71bb015555e5 2024-09-18T16:07:27.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 840fbf9a-6577-4496-87d4-71bb015555e5' 2024-09-18T16:07:27.674 INFO:tasks.workunit.client.0.smithi047.stdout:add osd3 840fbf9a-6577-4496-87d4-71bb015555e5 2024-09-18T16:07:27.674 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:07:27.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC/+upmimP/KBAAXIGFVE2+Z1CXkLeon5V0ew== 2024-09-18T16:07:27.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC/+upmimP/KBAAXIGFVE2+Z1CXkLeon5V0ew=="}' 2024-09-18T16:07:27.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 840fbf9a-6577-4496-87d4-71bb015555e5 -i td/repeer-on-acting-back/3/new.json 2024-09-18T16:07:28.190 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:07:28.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/3/new.json 2024-09-18T16:07:28.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=21c9ff2e-9d52-44eb-80ae-e48bd7727b72 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/3 --osd-journal=td/repeer-on-acting-back/3/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.490363/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC/+upmimP/KBAAXIGFVE2+Z1CXkLeon5V0ew== --osd-uuid 840fbf9a-6577-4496-87d4-71bb015555e5 2024-09-18T16:07:28.232 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:07:28.231+0000 7fbea9d57440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:07:28.234 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:07:28.233+0000 7fbea9d57440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:07:28.235 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:07:28.234+0000 7fbea9d57440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:07:28.235 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:07:28.234+0000 7fbea9d57440 -1 bluestore(td/repeer-on-acting-back/3/block) _read_bdev_label failed to open td/repeer-on-acting-back/3/block: (2) No such file or directory 2024-09-18T16:07:28.235 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:07:28.234+0000 7fbea9d57440 -1 bluestore(td/repeer-on-acting-back/3/block) _read_bdev_label failed to open td/repeer-on-acting-back/3/block: (2) No such file or directory 2024-09-18T16:07:28.235 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:07:28.234+0000 7fbea9d57440 -1 bluestore(td/repeer-on-acting-back/3/block) _read_bdev_label failed to open td/repeer-on-acting-back/3/block: (2) No such file or directory 2024-09-18T16:07:28.314 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:07:28.314+0000 7fbea9d57440 -1 bluestore(td/repeer-on-acting-back/3) _read_fsid unparsable uuid 2024-09-18T16:07:33.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/3/keyring 2024-09-18T16:07:33.911 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:07:33.912 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd3 key to auth repository 2024-09-18T16:07:33.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-18T16:07:33.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:07:34.423 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.3 2024-09-18T16:07:34.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-18T16:07:34.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=21c9ff2e-9d52-44eb-80ae-e48bd7727b72 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/3 --osd-journal=td/repeer-on-acting-back/3/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.490363/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:07:34.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:07:34.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:07:34.424 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:07:34.452 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:07:34.451+0000 7f05ae04f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:07:34.480 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:07:34.478+0000 7f05ae04f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:07:34.480 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:07:34.479+0000 7f05ae04f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:07:34.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-18T16:07:34.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:07:34.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T16:07:34.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:07:34.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:07:34.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:34.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:07:34.857 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:07:34.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:34.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:35.017 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:07:35.016+0000 7f05ae04f440 -1 Falling back to public interface 2024-09-18T16:07:35.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:36.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:36.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:36.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:07:36.269 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T16:07:36.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:36.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:36.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:37.593 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:07:37.591+0000 7f05ae04f440 -1 osd.3 0 log_to_monitors true 2024-09-18T16:07:37.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:37.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:37.696 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T16:07:37.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:07:37.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:37.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:38.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:39.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:39.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:39.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:07:39.238 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:07:39.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:39.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:39.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:40.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:40.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:40.697 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T16:07:40.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:07:40.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:40.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:41.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:42.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:42.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:42.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:07:42.135 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T16:07:42.135 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:42.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:42.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:43.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:43.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:43.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:07:43.547 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T16:07:43.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:43.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:43.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:44.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:44.944 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:44.944 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T16:07:44.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:07:44.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:44.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:45.434 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:46.435 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:46.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:46.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:07:46.436 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T16:07:46.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:46.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:46.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:47.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:47.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:47.891 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T16:07:47.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:07:47.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:47.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:48.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:49.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:49.333 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:49.333 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T16:07:49.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:07:49.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:49.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:49.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:50.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:50.757 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T16:07:50.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:50.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:07:50.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:50.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:51.521 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:52.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:52.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:52.523 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T16:07:52.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:07:52.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:52.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:52.981 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:53.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:53.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:53.984 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T16:07:53.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:07:53.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:53.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:54.754 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:55.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:55.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:55.755 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T16:07:55.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:07:55.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:55.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:56.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:57.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:57.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:57.354 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T16:07:57.355 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:07:57.355 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:57.355 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:58.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:07:59.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:07:59.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:07:59.204 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T16:07:59.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:07:59.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:07:59.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:07:59.660 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:00.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:00.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:00.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:08:00.662 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T16:08:00.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:00.662 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T16:08:01.051 INFO:tasks.workunit.client.0.smithi047.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/496669,v1:127.0.0.1:6827/496669] [v2:127.0.0.1:6828/496669,v1:127.0.0.1:6829/496669] exists,up 840fbf9a-6577-4496-87d4-71bb015555e5 2024-09-18T16:08:01.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:08:01.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:08:01.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:08:01.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-09-18T16:08:01.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 4 2024-09-18T16:08:01.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-09-18T16:08:01.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:08:01.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-18T16:08:01.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:08:01.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/4 2024-09-18T16:08:01.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=21c9ff2e-9d52-44eb-80ae-e48bd7727b72 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T16:08:01.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:08:01.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:08:01.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:08:01.056 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/4' 2024-09-18T16:08:01.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/4/journal' 2024-09-18T16:08:01.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:08:01.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:08:01.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-09-18T16:08:01.057 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:08:01.057 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:08:01.057 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:08:01.057 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:08:01.057 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:08:01.057 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.490363 2024-09-18T16:08:01.057 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.490363/$cluster-$name.asok' 2024-09-18T16:08:01.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.490363/$cluster-$name.asok' 2024-09-18T16:08:01.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:08:01.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:08:01.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:08:01.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-09-18T16:08:01.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-09-18T16:08:01.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:08:01.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:08:01.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:08:01.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:08:01.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:08:01.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:08:01.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/4 2024-09-18T16:08:01.059 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:08:01.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=13861686-9059-456d-973c-b7ea17a67aaa 2024-09-18T16:08:01.059 INFO:tasks.workunit.client.0.smithi047.stdout:add osd4 13861686-9059-456d-973c-b7ea17a67aaa 2024-09-18T16:08:01.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 13861686-9059-456d-973c-b7ea17a67aaa' 2024-09-18T16:08:01.059 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:08:01.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDh+upm9sxjBBAAHynKagk/hPvdZRZf3dCXKQ== 2024-09-18T16:08:01.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDh+upm9sxjBBAAHynKagk/hPvdZRZf3dCXKQ=="}' 2024-09-18T16:08:01.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 13861686-9059-456d-973c-b7ea17a67aaa -i td/repeer-on-acting-back/4/new.json 2024-09-18T16:08:01.562 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T16:08:01.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/4/new.json 2024-09-18T16:08:01.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=21c9ff2e-9d52-44eb-80ae-e48bd7727b72 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/4 --osd-journal=td/repeer-on-acting-back/4/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.490363/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDh+upm9sxjBBAAHynKagk/hPvdZRZf3dCXKQ== --osd-uuid 13861686-9059-456d-973c-b7ea17a67aaa 2024-09-18T16:08:01.605 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:08:01.604+0000 7eff7b71c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:08:01.606 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:08:01.605+0000 7eff7b71c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:08:01.607 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:08:01.606+0000 7eff7b71c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:08:01.608 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:08:01.607+0000 7eff7b71c440 -1 bluestore(td/repeer-on-acting-back/4/block) _read_bdev_label failed to open td/repeer-on-acting-back/4/block: (2) No such file or directory 2024-09-18T16:08:01.608 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:08:01.607+0000 7eff7b71c440 -1 bluestore(td/repeer-on-acting-back/4/block) _read_bdev_label failed to open td/repeer-on-acting-back/4/block: (2) No such file or directory 2024-09-18T16:08:01.608 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:08:01.607+0000 7eff7b71c440 -1 bluestore(td/repeer-on-acting-back/4/block) _read_bdev_label failed to open td/repeer-on-acting-back/4/block: (2) No such file or directory 2024-09-18T16:08:01.685 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:08:01.684+0000 7eff7b71c440 -1 bluestore(td/repeer-on-acting-back/4) _read_fsid unparsable uuid 2024-09-18T16:08:07.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/4/keyring 2024-09-18T16:08:07.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:08:07.554 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd4 key to auth repository 2024-09-18T16:08:07.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-18T16:08:07.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:08:08.075 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.4 2024-09-18T16:08:08.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-18T16:08:08.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=21c9ff2e-9d52-44eb-80ae-e48bd7727b72 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/4 --osd-journal=td/repeer-on-acting-back/4/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.490363/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:08:08.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:08:08.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:08:08.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:08:08.104 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:08:08.103+0000 7fd549012440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:08:08.129 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:08:08.128+0000 7fd549012440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:08:08.130 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:08:08.129+0000 7fd549012440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:08:08.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-18T16:08:08.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:08:08.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-18T16:08:08.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:08:08.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:08:08.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:08.488 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:08:08.488 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:08:08.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:08.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T16:08:08.667 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:08:08.666+0000 7fd549012440 -1 Falling back to public interface 2024-09-18T16:08:08.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:09.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:09.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:09.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:08:09.865 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T16:08:09.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:09.865 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T16:08:10.287 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:11.260 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:08:11.259+0000 7fd549012440 -1 osd.4 0 log_to_monitors true 2024-09-18T16:08:11.288 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:11.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:11.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:08:11.289 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T16:08:11.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:11.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T16:08:11.760 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:12.761 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:12.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:12.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:08:12.762 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:08:12.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:12.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T16:08:13.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:14.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:14.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:14.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:08:14.176 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T16:08:14.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:14.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T16:08:14.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:15.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:15.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:15.574 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T16:08:15.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:08:15.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:15.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T16:08:16.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:17.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:17.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:17.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:08:17.076 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T16:08:17.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:17.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T16:08:17.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:18.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:18.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:18.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:08:18.474 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T16:08:18.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:18.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T16:08:18.892 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:19.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:19.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:19.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:08:19.894 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T16:08:19.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:19.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T16:08:20.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:21.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:21.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:21.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:08:21.401 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T16:08:21.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:21.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T16:08:21.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:22.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:22.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:22.799 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:08:22.800 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T16:08:22.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:22.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T16:08:23.706 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:24.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:24.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:24.708 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T16:08:24.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:08:24.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:24.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T16:08:25.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:26.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:26.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:26.851 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T16:08:26.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:08:26.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:26.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T16:08:27.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:28.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:28.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:28.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:08:28.558 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T16:08:28.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:28.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T16:08:29.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:30.432 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:30.432 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:30.432 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:08:30.433 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T16:08:30.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:30.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T16:08:31.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:32.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:32.773 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:32.773 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T16:08:32.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:08:32.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:32.774 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T16:08:33.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:34.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:34.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:34.400 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T16:08:34.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:08:34.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:34.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T16:08:34.823 INFO:tasks.workunit.client.0.smithi047.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/498426,v1:127.0.0.1:6835/498426] [v2:127.0.0.1:6836/498426,v1:127.0.0.1:6837/498426] exists,up 13861686-9059-456d-973c-b7ea17a67aaa 2024-09-18T16:08:34.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:08:34.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:08:34.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:08:34.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-09-18T16:08:34.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 5 2024-09-18T16:08:34.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-09-18T16:08:34.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:08:34.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-18T16:08:34.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:08:34.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/5 2024-09-18T16:08:34.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=21c9ff2e-9d52-44eb-80ae-e48bd7727b72 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T16:08:34.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:08:34.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:08:34.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:08:34.828 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/5' 2024-09-18T16:08:34.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/5/journal' 2024-09-18T16:08:34.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:08:34.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:08:34.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-09-18T16:08:34.829 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:08:34.829 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:08:34.829 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:08:34.829 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:08:34.829 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:08:34.829 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.490363 2024-09-18T16:08:34.829 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.490363/$cluster-$name.asok' 2024-09-18T16:08:34.829 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.490363/$cluster-$name.asok' 2024-09-18T16:08:34.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:08:34.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:08:34.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:08:34.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-09-18T16:08:34.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-09-18T16:08:34.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:08:34.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:08:34.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:08:34.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:08:34.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:08:34.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:08:34.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/5 2024-09-18T16:08:34.830 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:08:34.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=18be8759-740d-446c-bb61-46135d271ad3 2024-09-18T16:08:34.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 18be8759-740d-446c-bb61-46135d271ad3' 2024-09-18T16:08:34.831 INFO:tasks.workunit.client.0.smithi047.stdout:add osd5 18be8759-740d-446c-bb61-46135d271ad3 2024-09-18T16:08:34.831 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:08:34.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAC++pm++huMhAAC2irKO6sULcI1AeMZoTQpA== 2024-09-18T16:08:34.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAC++pm++huMhAAC2irKO6sULcI1AeMZoTQpA=="}' 2024-09-18T16:08:34.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 18be8759-740d-446c-bb61-46135d271ad3 -i td/repeer-on-acting-back/5/new.json 2024-09-18T16:08:35.347 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T16:08:35.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/5/new.json 2024-09-18T16:08:35.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=21c9ff2e-9d52-44eb-80ae-e48bd7727b72 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/5 --osd-journal=td/repeer-on-acting-back/5/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.490363/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAC++pm++huMhAAC2irKO6sULcI1AeMZoTQpA== --osd-uuid 18be8759-740d-446c-bb61-46135d271ad3 2024-09-18T16:08:35.389 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:08:35.388+0000 7fe079723440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:08:35.391 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:08:35.390+0000 7fe079723440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:08:35.392 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:08:35.391+0000 7fe079723440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:08:35.392 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:08:35.391+0000 7fe079723440 -1 bluestore(td/repeer-on-acting-back/5/block) _read_bdev_label failed to open td/repeer-on-acting-back/5/block: (2) No such file or directory 2024-09-18T16:08:35.392 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:08:35.391+0000 7fe079723440 -1 bluestore(td/repeer-on-acting-back/5/block) _read_bdev_label failed to open td/repeer-on-acting-back/5/block: (2) No such file or directory 2024-09-18T16:08:35.392 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:08:35.391+0000 7fe079723440 -1 bluestore(td/repeer-on-acting-back/5/block) _read_bdev_label failed to open td/repeer-on-acting-back/5/block: (2) No such file or directory 2024-09-18T16:08:35.437 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:08:35.436+0000 7fe079723440 -1 bluestore(td/repeer-on-acting-back/5) _read_fsid unparsable uuid 2024-09-18T16:08:40.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/5/keyring 2024-09-18T16:08:40.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:08:40.973 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd5 key to auth repository 2024-09-18T16:08:40.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-18T16:08:40.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:08:41.499 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.5 2024-09-18T16:08:41.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-18T16:08:41.499 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=21c9ff2e-9d52-44eb-80ae-e48bd7727b72 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/5 --osd-journal=td/repeer-on-acting-back/5/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.490363/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:08:41.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:08:41.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:08:41.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:08:41.527 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:08:41.526+0000 7f1839521440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:08:41.530 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:08:41.529+0000 7f1839521440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:08:41.531 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:08:41.530+0000 7f1839521440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:08:41.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-18T16:08:41.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:08:41.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-18T16:08:41.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:08:41.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:08:41.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:41.890 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:08:41.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:08:41.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:41.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T16:08:42.065 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:08:42.064+0000 7f1839521440 -1 Falling back to public interface 2024-09-18T16:08:42.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:43.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:43.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:43.267 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T16:08:43.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:08:43.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:43.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T16:08:43.693 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:44.498 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:08:44.496+0000 7f1839521440 -1 osd.5 0 log_to_monitors true 2024-09-18T16:08:44.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:44.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:44.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:08:44.694 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T16:08:44.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:44.695 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T16:08:45.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:46.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:46.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:46.177 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:08:46.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:08:46.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:46.178 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T16:08:46.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:47.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:47.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:47.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:08:47.573 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T16:08:47.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:47.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T16:08:47.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:48.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:48.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:48.975 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T16:08:48.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:08:48.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:48.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T16:08:49.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:50.449 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:50.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:50.450 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T16:08:50.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:08:50.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:50.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T16:08:50.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:51.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:51.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:51.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:08:51.897 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T16:08:51.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:51.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T16:08:52.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:53.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:53.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:53.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:08:53.339 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T16:08:53.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:53.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T16:08:53.741 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:54.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:54.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:54.743 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T16:08:54.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:08:54.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:54.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T16:08:55.249 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:56.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:56.250 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:56.251 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T16:08:56.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:08:56.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:56.251 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T16:08:56.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:57.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:57.969 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T16:08:57.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:57.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:08:57.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:57.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T16:08:58.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:08:59.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:08:59.680 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T16:08:59.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:08:59.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:08:59.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:08:59.681 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T16:09:00.438 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:09:01.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:09:01.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:09:01.440 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T16:09:01.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:09:01.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:09:01.441 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T16:09:02.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:09:03.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:09:03.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:09:03.626 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T16:09:03.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:09:03.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:09:03.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T16:09:04.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:09:05.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:09:05.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:09:05.495 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T16:09:05.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:09:05.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:09:05.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T16:09:06.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:09:07.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:09:07.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:09:07.465 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T16:09:07.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:09:07.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:09:07.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T16:09:07.912 INFO:tasks.workunit.client.0.smithi047.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/500155,v1:127.0.0.1:6843/500155] [v2:127.0.0.1:6844/500155,v1:127.0.0.1:6845/500155] exists,up 18be8759-740d-446c-bb61-46135d271ad3 2024-09-18T16:09:07.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:09:07.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:09:07.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:09:07.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:60: TEST_repeer_on_down_acting_member_coming_back: create_pool test 1 1 2024-09-18T16:09:07.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-18T16:09:08.548 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test' already exists 2024-09-18T16:09:08.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T16:09:09.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:61: TEST_repeer_on_down_acting_member_coming_back: ceph osd pool set test size 3 2024-09-18T16:09:10.306 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 1 size to 3 2024-09-18T16:09:10.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:62: TEST_repeer_on_down_acting_member_coming_back: ceph osd pool set test min_size 2 2024-09-18T16:09:11.228 INFO:tasks.workunit.client.0.smithi047.stderr:set pool 1 min_size to 2 2024-09-18T16:09:11.240 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:63: TEST_repeer_on_down_acting_member_coming_back: ceph pg dump pools -f json 2024-09-18T16:09:11.240 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:63: TEST_repeer_on_down_acting_member_coming_back: jq .pool_stats 2024-09-18T16:09:11.240 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:63: TEST_repeer_on_down_acting_member_coming_back: jq '.[].poolid' 2024-09-18T16:09:11.608 INFO:tasks.workunit.client.0.smithi047.stderr:dumped pools 2024-09-18T16:09:11.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:63: TEST_repeer_on_down_acting_member_coming_back: local poolid=1 2024-09-18T16:09:11.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:64: TEST_repeer_on_down_acting_member_coming_back: local pgid=1.0 2024-09-18T16:09:11.619 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:67: TEST_repeer_on_down_acting_member_coming_back: ceph osd set-require-min-compat-client luminous 2024-09-18T16:09:12.614 INFO:tasks.workunit.client.0.smithi047.stderr:set require_min_compat_client to luminous 2024-09-18T16:09:12.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:69: TEST_repeer_on_down_acting_member_coming_back: ceph osd pg-upmap 1.0 1 2 3 2024-09-18T16:09:13.583 INFO:tasks.workunit.client.0.smithi047.stderr:set 1.0 pg_upmap mapping to [1,2,3] 2024-09-18T16:09:13.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:71: TEST_repeer_on_down_acting_member_coming_back: flush_pg_stats 2024-09-18T16:09:13.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:09:13.595 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:09:13.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:09:13.999 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T16:09:13.999 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T16:09:13.999 INFO:tasks.workunit.client.0.smithi047.stderr:3 2024-09-18T16:09:13.999 INFO:tasks.workunit.client.0.smithi047.stderr:4 2024-09-18T16:09:13.999 INFO:tasks.workunit.client.0.smithi047.stderr:5' 2024-09-18T16:09:13.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:09:13.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:09:13.999 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:09:14.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836523 2024-09-18T16:09:14.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836523 2024-09-18T16:09:14.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836523' 2024-09-18T16:09:14.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:09:14.154 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:09:14.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672998 2024-09-18T16:09:14.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672998 2024-09-18T16:09:14.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836523 1-42949672998' 2024-09-18T16:09:14.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:09:14.270 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:09:14.414 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509468 2024-09-18T16:09:14.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509468 2024-09-18T16:09:14.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836523 1-42949672998 2-64424509468' 2024-09-18T16:09:14.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:09:14.415 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T16:09:14.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345942 2024-09-18T16:09:14.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345942 2024-09-18T16:09:14.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836523 1-42949672998 2-64424509468 3-85899345942' 2024-09-18T16:09:14.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:09:14.530 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T16:09:14.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182412 2024-09-18T16:09:14.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182412 2024-09-18T16:09:14.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836523 1-42949672998 2-64424509468 3-85899345942 4-107374182412' 2024-09-18T16:09:14.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:09:14.645 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T16:09:14.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018888 2024-09-18T16:09:14.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018888 2024-09-18T16:09:14.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836523 1-42949672998 2-64424509468 3-85899345942 4-107374182412 5-128849018888' 2024-09-18T16:09:14.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:09:14.758 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836523 2024-09-18T16:09:14.758 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:09:14.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:09:14.760 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836523 2024-09-18T16:09:14.760 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:09:14.761 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836523 2024-09-18T16:09:14.761 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836523' 2024-09-18T16:09:14.762 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836523 2024-09-18T16:09:14.762 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:09:15.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836523 -lt 21474836523 2024-09-18T16:09:15.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:09:15.178 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672998 2024-09-18T16:09:15.178 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:09:15.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:09:15.179 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672998 2024-09-18T16:09:15.179 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:09:15.180 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672998 2024-09-18T16:09:15.181 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949672998 2024-09-18T16:09:15.181 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672998' 2024-09-18T16:09:15.181 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:09:15.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672998 -lt 42949672998 2024-09-18T16:09:15.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:09:15.554 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509468 2024-09-18T16:09:15.554 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:09:15.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:09:15.555 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509468 2024-09-18T16:09:15.555 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:09:15.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509468 2024-09-18T16:09:15.557 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509468 2024-09-18T16:09:15.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509468' 2024-09-18T16:09:15.557 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:09:15.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509468 -lt 64424509468 2024-09-18T16:09:15.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:09:15.945 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345942 2024-09-18T16:09:15.945 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:09:15.946 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T16:09:15.947 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345942 2024-09-18T16:09:15.947 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:09:15.948 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345942 2024-09-18T16:09:15.948 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 85899345942 2024-09-18T16:09:15.949 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345942' 2024-09-18T16:09:15.949 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T16:09:16.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345940 -lt 85899345942 2024-09-18T16:09:16.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:09:17.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:09:17.372 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T16:09:17.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345943 -lt 85899345942 2024-09-18T16:09:17.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:09:17.757 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182412 2024-09-18T16:09:17.757 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:09:17.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T16:09:17.758 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182412 2024-09-18T16:09:17.758 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:09:17.760 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182412 2024-09-18T16:09:17.760 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182412' 2024-09-18T16:09:17.760 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.4 seq 107374182412 2024-09-18T16:09:17.760 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T16:09:18.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182412 -lt 107374182412 2024-09-18T16:09:18.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:09:18.131 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018888 2024-09-18T16:09:18.132 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:09:18.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T16:09:18.133 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018888 2024-09-18T16:09:18.133 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:09:18.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018888 2024-09-18T16:09:18.134 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.5 seq 128849018888 2024-09-18T16:09:18.135 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018888' 2024-09-18T16:09:18.135 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T16:09:18.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018888 -lt 128849018888 2024-09-18T16:09:18.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:72: TEST_repeer_on_down_acting_member_coming_back: wait_for_clean 2024-09-18T16:09:18.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:09:18.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:09:18.554 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:09:18.554 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:09:18.555 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:09:18.555 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:09:18.555 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:09:18.555 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:09:18.556 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:09:18.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T16:09:18.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:09:18.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:09:18.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:09:18.644 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:09:18.644 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:09:19.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:09:19.018 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T16:09:19.018 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T16:09:19.018 INFO:tasks.workunit.client.0.smithi047.stderr:3 2024-09-18T16:09:19.018 INFO:tasks.workunit.client.0.smithi047.stderr:4 2024-09-18T16:09:19.018 INFO:tasks.workunit.client.0.smithi047.stderr:5' 2024-09-18T16:09:19.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:09:19.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:09:19.019 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:09:19.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836526 2024-09-18T16:09:19.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836526 2024-09-18T16:09:19.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836526' 2024-09-18T16:09:19.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:09:19.133 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:09:19.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673001 2024-09-18T16:09:19.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673001 2024-09-18T16:09:19.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836526 1-42949673001' 2024-09-18T16:09:19.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:09:19.247 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:09:19.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509471 2024-09-18T16:09:19.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509471 2024-09-18T16:09:19.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836526 1-42949673001 2-64424509471' 2024-09-18T16:09:19.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:09:19.361 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T16:09:19.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345945 2024-09-18T16:09:19.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345945 2024-09-18T16:09:19.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836526 1-42949673001 2-64424509471 3-85899345945' 2024-09-18T16:09:19.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:09:19.475 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T16:09:19.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182415 2024-09-18T16:09:19.589 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182415 2024-09-18T16:09:19.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836526 1-42949673001 2-64424509471 3-85899345945 4-107374182415' 2024-09-18T16:09:19.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:09:19.590 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T16:09:19.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018891 2024-09-18T16:09:19.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018891 2024-09-18T16:09:19.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836526 1-42949673001 2-64424509471 3-85899345945 4-107374182415 5-128849018891' 2024-09-18T16:09:19.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:09:19.704 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836526 2024-09-18T16:09:19.704 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:09:19.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:09:19.706 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836526 2024-09-18T16:09:19.706 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:09:19.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836526 2024-09-18T16:09:19.707 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836526 2024-09-18T16:09:19.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836526' 2024-09-18T16:09:19.707 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:09:20.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836524 -lt 21474836526 2024-09-18T16:09:20.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:09:21.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:09:21.108 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:09:21.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836527 -lt 21474836526 2024-09-18T16:09:21.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:09:21.534 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673001 2024-09-18T16:09:21.534 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:09:21.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:09:21.536 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673001 2024-09-18T16:09:21.536 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:09:21.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673001 2024-09-18T16:09:21.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673001' 2024-09-18T16:09:21.538 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949673001 2024-09-18T16:09:21.538 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:09:21.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673001 -lt 42949673001 2024-09-18T16:09:21.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:09:21.960 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509471 2024-09-18T16:09:21.960 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:09:21.961 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:09:21.962 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509471 2024-09-18T16:09:21.962 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:09:21.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509471 2024-09-18T16:09:21.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509471' 2024-09-18T16:09:21.963 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509471 2024-09-18T16:09:21.963 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:09:22.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509471 -lt 64424509471 2024-09-18T16:09:22.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:09:22.378 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345945 2024-09-18T16:09:22.378 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:09:22.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T16:09:22.380 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345945 2024-09-18T16:09:22.380 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:09:22.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345945 2024-09-18T16:09:22.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345945' 2024-09-18T16:09:22.381 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 85899345945 2024-09-18T16:09:22.381 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T16:09:22.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345946 -lt 85899345945 2024-09-18T16:09:22.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:09:22.788 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182415 2024-09-18T16:09:22.788 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:09:22.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T16:09:22.790 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182415 2024-09-18T16:09:22.790 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:09:22.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182415 2024-09-18T16:09:22.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182415' 2024-09-18T16:09:22.791 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.4 seq 107374182415 2024-09-18T16:09:22.791 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T16:09:23.231 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182415 -lt 107374182415 2024-09-18T16:09:23.231 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:09:23.231 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018891 2024-09-18T16:09:23.231 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:09:23.232 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T16:09:23.233 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018891 2024-09-18T16:09:23.233 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:09:23.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018891 2024-09-18T16:09:23.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018891' 2024-09-18T16:09:23.234 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.5 seq 128849018891 2024-09-18T16:09:23.234 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T16:09:23.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018891 -lt 128849018891 2024-09-18T16:09:23.650 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:09:23.650 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:09:23.651 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:09:24.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:09:24.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:09:24.156 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:09:24.156 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:09:24.156 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:09:24.156 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:09:24.156 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:09:24.157 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:09:24.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:09:24.555 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:09:24.555 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:09:24.556 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:09:25.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:09:25.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:09:25.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:09:25.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:74: TEST_repeer_on_down_acting_member_coming_back: echo 'writing initial objects' 2024-09-18T16:09:25.087 INFO:tasks.workunit.client.0.smithi047.stdout:writing initial objects 2024-09-18T16:09:25.087 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: seq 1 100 2024-09-18T16:09:25.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:25.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_1 /etc/fstab 2024-09-18T16:09:25.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:25.243 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_2 /etc/fstab 2024-09-18T16:09:25.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:25.411 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_3 /etc/fstab 2024-09-18T16:09:25.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:25.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_4 /etc/fstab 2024-09-18T16:09:25.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:25.680 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_5 /etc/fstab 2024-09-18T16:09:25.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:25.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_6 /etc/fstab 2024-09-18T16:09:25.948 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:25.948 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_7 /etc/fstab 2024-09-18T16:09:26.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:26.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_8 /etc/fstab 2024-09-18T16:09:26.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:26.206 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_9 /etc/fstab 2024-09-18T16:09:26.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:26.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_10 /etc/fstab 2024-09-18T16:09:26.476 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:26.477 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_11 /etc/fstab 2024-09-18T16:09:26.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:26.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_12 /etc/fstab 2024-09-18T16:09:26.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:26.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_13 /etc/fstab 2024-09-18T16:09:26.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:26.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_14 /etc/fstab 2024-09-18T16:09:27.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:27.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_15 /etc/fstab 2024-09-18T16:09:27.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:27.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_16 /etc/fstab 2024-09-18T16:09:27.265 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:27.266 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_17 /etc/fstab 2024-09-18T16:09:27.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:27.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_18 /etc/fstab 2024-09-18T16:09:27.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:27.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_19 /etc/fstab 2024-09-18T16:09:27.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:27.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_20 /etc/fstab 2024-09-18T16:09:27.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:27.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_21 /etc/fstab 2024-09-18T16:09:27.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:27.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_22 /etc/fstab 2024-09-18T16:09:28.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:28.045 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_23 /etc/fstab 2024-09-18T16:09:28.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:28.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_24 /etc/fstab 2024-09-18T16:09:28.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:28.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_25 /etc/fstab 2024-09-18T16:09:28.447 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:28.448 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_26 /etc/fstab 2024-09-18T16:09:28.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:28.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_27 /etc/fstab 2024-09-18T16:09:28.741 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:28.741 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_28 /etc/fstab 2024-09-18T16:09:28.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:28.867 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_29 /etc/fstab 2024-09-18T16:09:29.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:29.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_30 /etc/fstab 2024-09-18T16:09:29.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:29.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_31 /etc/fstab 2024-09-18T16:09:29.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:29.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_32 /etc/fstab 2024-09-18T16:09:29.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:29.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_33 /etc/fstab 2024-09-18T16:09:29.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:29.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_34 /etc/fstab 2024-09-18T16:09:29.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:29.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_35 /etc/fstab 2024-09-18T16:09:29.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:29.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_36 /etc/fstab 2024-09-18T16:09:29.949 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:29.949 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_37 /etc/fstab 2024-09-18T16:09:30.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:30.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_38 /etc/fstab 2024-09-18T16:09:30.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:30.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_39 /etc/fstab 2024-09-18T16:09:30.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:30.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_40 /etc/fstab 2024-09-18T16:09:30.469 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:30.469 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_41 /etc/fstab 2024-09-18T16:09:30.612 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:30.612 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_42 /etc/fstab 2024-09-18T16:09:30.746 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:30.746 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_43 /etc/fstab 2024-09-18T16:09:30.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:30.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_44 /etc/fstab 2024-09-18T16:09:31.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:31.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_45 /etc/fstab 2024-09-18T16:09:31.157 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:31.157 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_46 /etc/fstab 2024-09-18T16:09:31.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:31.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_47 /etc/fstab 2024-09-18T16:09:31.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:31.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_48 /etc/fstab 2024-09-18T16:09:31.551 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:31.551 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_49 /etc/fstab 2024-09-18T16:09:31.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:31.694 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_50 /etc/fstab 2024-09-18T16:09:31.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:31.820 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_51 /etc/fstab 2024-09-18T16:09:31.946 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:31.946 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_52 /etc/fstab 2024-09-18T16:09:32.080 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:32.080 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_53 /etc/fstab 2024-09-18T16:09:32.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:32.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_54 /etc/fstab 2024-09-18T16:09:32.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:32.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_55 /etc/fstab 2024-09-18T16:09:32.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:32.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_56 /etc/fstab 2024-09-18T16:09:32.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:32.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_57 /etc/fstab 2024-09-18T16:09:32.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:32.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_58 /etc/fstab 2024-09-18T16:09:32.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:32.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_59 /etc/fstab 2024-09-18T16:09:33.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:33.003 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_60 /etc/fstab 2024-09-18T16:09:33.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:33.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_61 /etc/fstab 2024-09-18T16:09:33.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:33.263 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_62 /etc/fstab 2024-09-18T16:09:33.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:33.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_63 /etc/fstab 2024-09-18T16:09:33.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:33.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_64 /etc/fstab 2024-09-18T16:09:33.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:33.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_65 /etc/fstab 2024-09-18T16:09:33.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:33.850 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_66 /etc/fstab 2024-09-18T16:09:33.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:33.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_67 /etc/fstab 2024-09-18T16:09:34.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:34.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_68 /etc/fstab 2024-09-18T16:09:34.252 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:34.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_69 /etc/fstab 2024-09-18T16:09:34.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:34.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_70 /etc/fstab 2024-09-18T16:09:34.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:34.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_71 /etc/fstab 2024-09-18T16:09:34.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:34.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_72 /etc/fstab 2024-09-18T16:09:34.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:34.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_73 /etc/fstab 2024-09-18T16:09:34.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:34.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_74 /etc/fstab 2024-09-18T16:09:35.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:35.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_75 /etc/fstab 2024-09-18T16:09:35.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:35.276 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_76 /etc/fstab 2024-09-18T16:09:35.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:35.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_77 /etc/fstab 2024-09-18T16:09:35.527 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:35.527 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_78 /etc/fstab 2024-09-18T16:09:35.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:35.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_79 /etc/fstab 2024-09-18T16:09:35.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:35.771 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_80 /etc/fstab 2024-09-18T16:09:35.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:35.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_81 /etc/fstab 2024-09-18T16:09:36.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:36.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_82 /etc/fstab 2024-09-18T16:09:36.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:36.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_83 /etc/fstab 2024-09-18T16:09:36.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:36.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_84 /etc/fstab 2024-09-18T16:09:36.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:36.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_85 /etc/fstab 2024-09-18T16:09:36.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:36.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_86 /etc/fstab 2024-09-18T16:09:36.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:36.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_87 /etc/fstab 2024-09-18T16:09:37.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:37.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_88 /etc/fstab 2024-09-18T16:09:37.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:37.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_89 /etc/fstab 2024-09-18T16:09:37.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:37.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_90 /etc/fstab 2024-09-18T16:09:37.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:37.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_91 /etc/fstab 2024-09-18T16:09:37.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:37.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_92 /etc/fstab 2024-09-18T16:09:37.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:37.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_93 /etc/fstab 2024-09-18T16:09:37.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:37.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_94 /etc/fstab 2024-09-18T16:09:38.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:38.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_95 /etc/fstab 2024-09-18T16:09:38.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:38.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_96 /etc/fstab 2024-09-18T16:09:38.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:38.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_97 /etc/fstab 2024-09-18T16:09:38.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:38.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_98 /etc/fstab 2024-09-18T16:09:38.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:38.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_99 /etc/fstab 2024-09-18T16:09:38.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-18T16:09:38.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_100 /etc/fstab 2024-09-18T16:09:38.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:81: TEST_repeer_on_down_acting_member_coming_back: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-18T16:09:38.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:81: TEST_repeer_on_down_acting_member_coming_back: wait_for_clean 2024-09-18T16:09:38.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:09:38.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:09:38.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:09:38.899 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-18T16:09:38.899 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:09:38.899 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:09:38.900 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:09:38.900 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:09:38.900 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:09:38.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-18T16:09:38.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:09:38.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:09:38.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:09:38.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:09:38.960 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:09:39.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:09:39.332 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T16:09:39.332 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T16:09:39.332 INFO:tasks.workunit.client.0.smithi047.stderr:3 2024-09-18T16:09:39.332 INFO:tasks.workunit.client.0.smithi047.stderr:4 2024-09-18T16:09:39.332 INFO:tasks.workunit.client.0.smithi047.stderr:5' 2024-09-18T16:09:39.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:09:39.332 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:09:39.333 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:09:39.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836532 2024-09-18T16:09:39.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836532 2024-09-18T16:09:39.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836532' 2024-09-18T16:09:39.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:09:39.447 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:09:39.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673007 2024-09-18T16:09:39.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673007 2024-09-18T16:09:39.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836532 1-42949673007' 2024-09-18T16:09:39.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:09:39.561 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:09:39.676 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509477 2024-09-18T16:09:39.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509477 2024-09-18T16:09:39.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836532 1-42949673007 2-64424509477' 2024-09-18T16:09:39.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:09:39.677 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T16:09:39.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345951 2024-09-18T16:09:39.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345951 2024-09-18T16:09:39.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836532 1-42949673007 2-64424509477 3-85899345951' 2024-09-18T16:09:39.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:09:39.791 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T16:09:39.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182421 2024-09-18T16:09:39.905 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182421 2024-09-18T16:09:39.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836532 1-42949673007 2-64424509477 3-85899345951 4-107374182421' 2024-09-18T16:09:39.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:09:39.906 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T16:09:40.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018897 2024-09-18T16:09:40.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018897 2024-09-18T16:09:40.019 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836532 1-42949673007 2-64424509477 3-85899345951 4-107374182421 5-128849018897' 2024-09-18T16:09:40.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:09:40.020 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836532 2024-09-18T16:09:40.020 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:09:40.021 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:09:40.021 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836532 2024-09-18T16:09:40.022 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:09:40.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836532 2024-09-18T16:09:40.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836532' 2024-09-18T16:09:40.023 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836532 2024-09-18T16:09:40.023 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:09:40.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836530 -lt 21474836532 2024-09-18T16:09:40.436 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:09:41.437 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:09:41.437 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:09:41.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836533 -lt 21474836532 2024-09-18T16:09:41.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:09:41.863 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673007 2024-09-18T16:09:41.863 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:09:41.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:09:41.865 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673007 2024-09-18T16:09:41.865 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:09:41.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673007 2024-09-18T16:09:41.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673007' 2024-09-18T16:09:41.866 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949673007 2024-09-18T16:09:41.867 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:09:42.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673007 -lt 42949673007 2024-09-18T16:09:42.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:09:42.236 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509477 2024-09-18T16:09:42.236 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:09:42.237 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:09:42.238 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509477 2024-09-18T16:09:42.238 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:09:42.238 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509477 2024-09-18T16:09:42.239 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509477' 2024-09-18T16:09:42.239 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509477 2024-09-18T16:09:42.239 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:09:42.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509478 -lt 64424509477 2024-09-18T16:09:42.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:09:42.658 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345951 2024-09-18T16:09:42.658 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:09:42.659 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T16:09:42.659 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345951 2024-09-18T16:09:42.659 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:09:42.660 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345951 2024-09-18T16:09:42.660 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345951' 2024-09-18T16:09:42.661 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 85899345951 2024-09-18T16:09:42.661 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T16:09:43.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345952 -lt 85899345951 2024-09-18T16:09:43.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:09:43.060 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182421 2024-09-18T16:09:43.060 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:09:43.061 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T16:09:43.062 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182421 2024-09-18T16:09:43.062 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:09:43.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182421 2024-09-18T16:09:43.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182421' 2024-09-18T16:09:43.063 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.4 seq 107374182421 2024-09-18T16:09:43.064 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T16:09:43.469 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182421 -lt 107374182421 2024-09-18T16:09:43.469 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:09:43.470 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018897 2024-09-18T16:09:43.470 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:09:43.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T16:09:43.471 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018897 2024-09-18T16:09:43.471 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:09:43.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018897 2024-09-18T16:09:43.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018897' 2024-09-18T16:09:43.473 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.5 seq 128849018897 2024-09-18T16:09:43.473 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T16:09:43.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018897 -lt 128849018897 2024-09-18T16:09:43.871 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:09:43.871 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:09:43.871 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:09:44.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:09:44.386 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:09:44.387 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:09:44.387 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:09:44.387 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:09:44.387 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:09:44.388 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:09:44.388 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:09:44.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:09:44.767 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:09:44.767 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:09:44.767 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:09:45.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:09:45.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:09:45.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:09:45.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:84: TEST_repeer_on_down_acting_member_coming_back: ceph osd pg-upmap 1.0 1 4 5 2024-09-18T16:09:46.203 INFO:tasks.workunit.client.0.smithi047.stderr:set 1.0 pg_upmap mapping to [1,4,5] 2024-09-18T16:09:46.216 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:87: TEST_repeer_on_down_acting_member_coming_back: sleep 2 2024-09-18T16:09:48.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:90: TEST_repeer_on_down_acting_member_coming_back: ceph pg 1.0 query 2024-09-18T16:09:48.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:90: TEST_repeer_on_down_acting_member_coming_back: jq .acting 2024-09-18T16:09:48.218 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:90: TEST_repeer_on_down_acting_member_coming_back: grep 2 2024-09-18T16:09:48.350 INFO:tasks.workunit.client.0.smithi047.stdout: 2, 2024-09-18T16:09:48.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:93: TEST_repeer_on_down_acting_member_coming_back: kill_daemons td/repeer-on-acting-back KILL osd.2 2024-09-18T16:09:48.351 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:09:48.351 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:09:48.351 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:09:48.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:09:48.352 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:09:48.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:09:48.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:94: TEST_repeer_on_down_acting_member_coming_back: ceph osd down osd.2 2024-09-18T16:09:48.872 INFO:tasks.workunit.client.0.smithi047.stderr:osd.2 is already down. 2024-09-18T16:09:48.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:97: TEST_repeer_on_down_acting_member_coming_back: sleep 2 2024-09-18T16:09:50.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:100: TEST_repeer_on_down_acting_member_coming_back: ceph pg 1.0 query 2024-09-18T16:09:50.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:100: TEST_repeer_on_down_acting_member_coming_back: jq .acting 2024-09-18T16:09:50.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:100: TEST_repeer_on_down_acting_member_coming_back: grep 2 2024-09-18T16:09:51.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:103: TEST_repeer_on_down_acting_member_coming_back: activate_osd td/repeer-on-acting-back 2 2024-09-18T16:09:51.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/repeer-on-acting-back 2024-09-18T16:09:51.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T16:09:51.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T16:09:51.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T16:09:51.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/repeer-on-acting-back/2 2024-09-18T16:09:51.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=21c9ff2e-9d52-44eb-80ae-e48bd7727b72 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-18T16:09:51.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:09:51.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:09:51.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:09:51.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/2' 2024-09-18T16:09:51.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/2/journal' 2024-09-18T16:09:51.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T16:09:51.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T16:09:51.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-09-18T16:09:51.106 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T16:09:51.106 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:09:51.106 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:09:51.106 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:09:51.106 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:09:51.106 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.490363 2024-09-18T16:09:51.106 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.490363/$cluster-$name.asok' 2024-09-18T16:09:51.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.490363/$cluster-$name.asok' 2024-09-18T16:09:51.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:09:51.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-09-18T16:09:51.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-09-18T16:09:51.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:09:51.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:09:51.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:09:51.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:09:51.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T16:09:51.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T16:09:51.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/repeer-on-acting-back/2 2024-09-18T16:09:51.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T16:09:51.108 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T16:09:51.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=21c9ff2e-9d52-44eb-80ae-e48bd7727b72 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/2 --osd-journal=td/repeer-on-acting-back/2/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.490363/$cluster-$name.asok' --debug-osd=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:09:51.109 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/repeer-on-acting-back/2/whoami 2024-09-18T16:09:51.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T16:09:51.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T16:09:51.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T16:09:51.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T16:09:51.133 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:09:51.132+0000 7f8824184440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:09:51.135 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:09:51.134+0000 7f8824184440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:09:51.136 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:09:51.135+0000 7f8824184440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:09:51.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T16:09:51.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:09:51.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T16:09:51.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:09:51.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:09:51.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:09:51.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:09:51.487 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:09:51.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:09:51.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:09:51.662 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:09:51.661+0000 7f8824184440 -1 Falling back to public interface 2024-09-18T16:09:52.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:09:53.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:09:53.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:09:53.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:09:53.072 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T16:09:53.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:09:53.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:09:53.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:09:54.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:09:54.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:09:54.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:09:54.458 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T16:09:54.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:09:54.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:09:54.938 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:09:55.658 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:09:55.658+0000 7f8824184440 -1 osd.2 45 log_to_monitors true 2024-09-18T16:09:55.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:09:55.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:09:55.939 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:09:55.939 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:09:55.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:09:55.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:09:56.593 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 49 up_thru 0 down_at 46 last_clean_interval [15,45) [v2:127.0.0.1:6818/504623,v1:127.0.0.1:6819/504623] [v2:127.0.0.1:6820/504623,v1:127.0.0.1:6821/504623] exists,up bcf2c02f-a273-4382-b5a4-92e83836c699 2024-09-18T16:09:56.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:09:56.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:09:56.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:09:56.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:104: TEST_repeer_on_down_acting_member_coming_back: wait_for_osd up 2 2024-09-18T16:09:56.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:09:56.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T16:09:56.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:09:56.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:09:56.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:09:56.595 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:09:56.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:09:56.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:09:56.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:09:56.990 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 49 up_thru 0 down_at 46 last_clean_interval [15,45) [v2:127.0.0.1:6818/504623,v1:127.0.0.1:6819/504623] [v2:127.0.0.1:6820/504623,v1:127.0.0.1:6821/504623] exists,up bcf2c02f-a273-4382-b5a4-92e83836c699 2024-09-18T16:09:56.991 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:09:56.991 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:09:56.991 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:09:56.991 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:107: TEST_repeer_on_down_acting_member_coming_back: sleep 2 2024-09-18T16:09:58.992 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:110: TEST_repeer_on_down_acting_member_coming_back: ceph pg 1.0 query 2024-09-18T16:09:58.992 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:110: TEST_repeer_on_down_acting_member_coming_back: jq .acting 2024-09-18T16:09:58.992 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:110: TEST_repeer_on_down_acting_member_coming_back: grep 2 2024-09-18T16:09:59.202 INFO:tasks.workunit.client.0.smithi047.stdout: 2 2024-09-18T16:09:59.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:112: TEST_repeer_on_down_acting_member_coming_back: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-18T16:09:59.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:112: TEST_repeer_on_down_acting_member_coming_back: wait_for_clean 2024-09-18T16:09:59.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:09:59.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:09:59.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:09:59.203 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-18T16:09:59.203 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:09:59.204 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:09:59.204 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:09:59.204 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:09:59.204 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:09:59.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-18T16:09:59.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:09:59.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:09:59.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:09:59.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:09:59.260 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:09:59.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:09:59.647 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T16:09:59.647 INFO:tasks.workunit.client.0.smithi047.stderr:2 2024-09-18T16:09:59.647 INFO:tasks.workunit.client.0.smithi047.stderr:3 2024-09-18T16:09:59.647 INFO:tasks.workunit.client.0.smithi047.stderr:4 2024-09-18T16:09:59.647 INFO:tasks.workunit.client.0.smithi047.stderr:5' 2024-09-18T16:09:59.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:09:59.647 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:09:59.647 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:09:59.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836538 2024-09-18T16:09:59.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836538 2024-09-18T16:09:59.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836538' 2024-09-18T16:09:59.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:09:59.798 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:09:59.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673013 2024-09-18T16:09:59.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673013 2024-09-18T16:09:59.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836538 1-42949673013' 2024-09-18T16:09:59.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:09:59.912 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:10:00.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=210453397507 2024-09-18T16:10:00.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 210453397507 2024-09-18T16:10:00.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836538 1-42949673013 2-210453397507' 2024-09-18T16:10:00.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:10:00.084 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T16:10:00.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345958 2024-09-18T16:10:00.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345958 2024-09-18T16:10:00.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836538 1-42949673013 2-210453397507 3-85899345958' 2024-09-18T16:10:00.198 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:10:00.198 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T16:10:00.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182427 2024-09-18T16:10:00.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182427 2024-09-18T16:10:00.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836538 1-42949673013 2-210453397507 3-85899345958 4-107374182427' 2024-09-18T16:10:00.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:10:00.313 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T16:10:00.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018903 2024-09-18T16:10:00.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018903 2024-09-18T16:10:00.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836538 1-42949673013 2-210453397507 3-85899345958 4-107374182427 5-128849018903' 2024-09-18T16:10:00.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:10:00.429 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836538 2024-09-18T16:10:00.429 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:10:00.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:10:00.429 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836538 2024-09-18T16:10:00.430 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:10:00.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836538 2024-09-18T16:10:00.431 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836538 2024-09-18T16:10:00.431 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836538' 2024-09-18T16:10:00.431 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:10:00.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836539 -lt 21474836538 2024-09-18T16:10:00.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:10:00.803 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673013 2024-09-18T16:10:00.803 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:10:00.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:10:00.805 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673013 2024-09-18T16:10:00.805 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:10:00.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673013 2024-09-18T16:10:00.806 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 42949673013 2024-09-18T16:10:00.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673013' 2024-09-18T16:10:00.806 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:10:01.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673013 -lt 42949673013 2024-09-18T16:10:01.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:10:01.340 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-210453397507 2024-09-18T16:10:01.340 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:10:01.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:10:01.341 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-210453397507 2024-09-18T16:10:01.341 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:10:01.342 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=210453397507 2024-09-18T16:10:01.342 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 210453397507 2024-09-18T16:10:01.343 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 210453397507' 2024-09-18T16:10:01.343 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:10:01.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 210453397507 -lt 210453397507 2024-09-18T16:10:01.723 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:10:01.723 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345958 2024-09-18T16:10:01.723 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:10:01.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T16:10:01.725 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345958 2024-09-18T16:10:01.725 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:10:01.725 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345958 2024-09-18T16:10:01.726 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.3 seq 85899345958 2024-09-18T16:10:01.726 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345958' 2024-09-18T16:10:01.726 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T16:10:02.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345958 -lt 85899345958 2024-09-18T16:10:02.202 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:10:02.202 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182427 2024-09-18T16:10:02.202 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:10:02.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T16:10:02.204 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182427 2024-09-18T16:10:02.204 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:10:02.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182427 2024-09-18T16:10:02.205 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182427' 2024-09-18T16:10:02.205 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.4 seq 107374182427 2024-09-18T16:10:02.205 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T16:10:02.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182427 -lt 107374182427 2024-09-18T16:10:02.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:10:02.766 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018903 2024-09-18T16:10:02.766 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:10:02.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T16:10:02.767 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018903 2024-09-18T16:10:02.768 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:10:02.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018903 2024-09-18T16:10:02.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018903' 2024-09-18T16:10:02.769 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.5 seq 128849018903 2024-09-18T16:10:02.769 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T16:10:03.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018903 -lt 128849018903 2024-09-18T16:10:03.310 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:10:03.310 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:10:03.310 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:10:03.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:10:03.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:10:03.998 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:10:03.998 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:10:03.998 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:10:03.998 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:10:03.998 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:10:03.998 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:10:04.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T16:10:04.367 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:10:04.367 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:10:04.367 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:10:04.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T16:10:04.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-18T16:10:04.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T16:10:04.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-18T16:10:04.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T16:10:04.912 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T16:10:05.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T16:10:05.013 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:10:05.014 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:10:05.014 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:10:05.014 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:10:05.014 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:10:05.014 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:10:05.014 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:10:05.446 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T16:10:05.446 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:10:05.446 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:10:05.446 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:10:06.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T16:10:06.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T16:10:06.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T16:10:06.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T16:10:06.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T16:10:06.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T16:10:06.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T16:10:06.195 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T16:10:06.195 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T16:10:06.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1632 2024-09-18T16:10:06.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1632 '!=' null 2024-09-18T16:10:06.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T16:10:06.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T16:10:06.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T16:10:06.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T16:10:06.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:10:06.858 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:10:06.858 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:10:06.858 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:10:06.858 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:10:06.858 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:10:06.858 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:10:07.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-18T16:10:07.222 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:10:07.223 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:10:07.223 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:10:07.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-18T16:10:07.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-18T16:10:07.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T16:10:07.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T16:10:07.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T16:10:07.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T16:10:07.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T16:10:07.896 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T16:10:07.896 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T16:10:08.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1632 2024-09-18T16:10:08.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1632 '!=' null 2024-09-18T16:10:08.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T16:10:08.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T16:10:08.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T16:10:08.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T16:10:08.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:10:08.490 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:10:08.490 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:10:08.490 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:10:08.490 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:10:08.490 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:10:08.490 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:10:08.880 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:10:08.880 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:10:08.881 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:10:08.881 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:10:09.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:10:09.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:10:09.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:10:09.416 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:114: TEST_repeer_on_down_acting_member_coming_back: find td/repeer-on-acting-back -name '*osd*log' 2024-09-18T16:10:09.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:114: TEST_repeer_on_down_acting_member_coming_back: grep -q 'Active: got notify from previous acting member.*, requesting pg_temp change' td/repeer-on-acting-back/osd.0.log td/repeer-on-acting-back/osd.1.log td/repeer-on-acting-back/osd.5.log td/repeer-on-acting-back/osd.3.log td/repeer-on-acting-back/osd.4.log td/repeer-on-acting-back/osd.2.log 2024-09-18T16:10:09.422 INFO:tasks.workunit.client.0.smithi047.stdout:success 2024-09-18T16:10:09.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:119: TEST_repeer_on_down_acting_member_coming_back: echo success 2024-09-18T16:10:09.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:121: TEST_repeer_on_down_acting_member_coming_back: delete_pool test 2024-09-18T16:10:09.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-18T16:10:09.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-18T16:10:09.981 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test' does not exist 2024-09-18T16:10:09.991 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:122: TEST_repeer_on_down_acting_member_coming_back: kill_daemons td/repeer-on-acting-back 2024-09-18T16:10:09.991 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:10:09.992 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:10:09.992 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:10:09.992 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:10:09.992 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:10:15.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:10:15.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:42: run: teardown td/repeer-on-acting-back 2024-09-18T16:10:15.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repeer-on-acting-back 2024-09-18T16:10:15.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:10:15.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repeer-on-acting-back KILL 2024-09-18T16:10:15.310 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:10:15.310 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:10:15.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:10:15.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:10:15.310 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:10:15.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:10:15.318 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:10:15.319 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:10:15.320 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:10:15.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:10:15.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:10:15.321 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:10:15.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:10:15.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:10:15.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:10:15.323 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:10:15.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:10:15.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:10:15.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repeer-on-acting-back 2024-09-18T16:10:15.362 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:10:15.362 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:10:15.362 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.490363 2024-09-18T16:10:15.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.490363 2024-09-18T16:10:15.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:10:15.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:10:15.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T16:10:15.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/repeer-on-acting-back 0 2024-09-18T16:10:15.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repeer-on-acting-back 2024-09-18T16:10:15.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T16:10:15.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repeer-on-acting-back KILL 2024-09-18T16:10:15.364 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:10:15.364 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:10:15.364 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:10:15.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:10:15.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:10:15.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:10:15.366 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:10:15.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:10:15.367 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:10:15.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:10:15.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:10:15.369 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:10:15.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:10:15.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:10:15.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:10:15.370 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:10:15.370 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:10:15.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T16:10:15.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repeer-on-acting-back 2024-09-18T16:10:15.373 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:10:15.373 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:10:15.373 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.490363 2024-09-18T16:10:15.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.490363 2024-09-18T16:10:15.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:10:15.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:10:15.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T16:10:15.376 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T16:10:15.376 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T16:10:15.403 INFO:tasks.workunit:Running workunit osd/repro_long_log.sh... 2024-09-18T16:10:15.403 DEBUG:teuthology.orchestra.run.smithi047:workunit test osd/repro_long_log.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh 2024-09-18T16:10:15.468 INFO:tasks.workunit.client.0.smithi047.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T16:10:15.474 INFO:tasks.workunit.client.0.smithi047.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T16:10:15.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T16:10:15.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-18T16:10:15.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T16:10:15.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T16:10:15.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T16:10:15.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T16:10:15.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T16:10:15.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T16:10:15.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/repro-long-log 2024-09-18T16:10:15.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:23: run: local dir=td/repro-long-log 2024-09-18T16:10:15.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:24: run: shift 2024-09-18T16:10:15.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:26: run: export CEPH_MON=127.0.0.1:7100 2024-09-18T16:10:15.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:26: run: CEPH_MON=127.0.0.1:7100 2024-09-18T16:10:15.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:27: run: export CEPH_ARGS 2024-09-18T16:10:15.475 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:28: run: uuidgen 2024-09-18T16:10:15.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:28: run: CEPH_ARGS+='--fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none ' 2024-09-18T16:10:15.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:29: run: CEPH_ARGS+='--mon-host=127.0.0.1:7100 ' 2024-09-18T16:10:15.476 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:31: run: set 2024-09-18T16:10:15.477 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-18T16:10:15.477 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:31: run: local 'funcs=TEST_repro_long_log1 2024-09-18T16:10:15.477 INFO:tasks.workunit.client.0.smithi047.stderr:TEST_repro_long_log2 2024-09-18T16:10:15.477 INFO:tasks.workunit.client.0.smithi047.stderr:TEST_trim_max_entries 2024-09-18T16:10:15.477 INFO:tasks.workunit.client.0.smithi047.stderr:TEST_trim_max_entries_with_dups' 2024-09-18T16:10:15.477 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:32: run: for func in $funcs 2024-09-18T16:10:15.477 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:33: run: setup td/repro-long-log 2024-09-18T16:10:15.477 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/repro-long-log 2024-09-18T16:10:15.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/repro-long-log 2024-09-18T16:10:15.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-09-18T16:10:15.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:10:15.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-09-18T16:10:15.478 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:10:15.478 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:10:15.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:10:15.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:10:15.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:10:15.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:10:15.479 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:10:15.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:10:15.479 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:10:15.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:10:15.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:10:15.480 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:10:15.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:10:15.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:10:15.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:10:15.481 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:10:15.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:10:15.483 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:10:15.483 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-09-18T16:10:15.485 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:10:15.485 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:10:15.485 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:10:15.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.506073 2024-09-18T16:10:15.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:10:15.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:10:15.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/repro-long-log 2024-09-18T16:10:15.487 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T16:10:15.487 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:10:15.487 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:10:15.487 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.506073 2024-09-18T16:10:15.489 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T16:10:15.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T16:10:15.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T16:10:15.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T16:10:15.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/repro-long-log 1' TERM HUP INT 2024-09-18T16:10:15.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:34: run: TEST_repro_long_log1 td/repro-long-log 2024-09-18T16:10:15.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:97: TEST_repro_long_log1: local dir=td/repro-long-log 2024-09-18T16:10:15.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:99: TEST_repro_long_log1: setup_log_test td/repro-long-log 2024-09-18T16:10:15.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:55: setup_log_test: local dir=td/repro-long-log 2024-09-18T16:10:15.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:56: setup_log_test: local which= 2024-09-18T16:10:15.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:58: setup_log_test: run_mon td/repro-long-log a 2024-09-18T16:10:15.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/repro-long-log 2024-09-18T16:10:15.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T16:10:15.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T16:10:15.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T16:10:15.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/repro-long-log/a 2024-09-18T16:10:15.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/repro-long-log/a --run-dir=td/repro-long-log 2024-09-18T16:10:15.860 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T16:10:15.860 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:10:15.860 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:10:15.860 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:10:15.860 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:10:15.860 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:10:15.860 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.506073/$cluster-$name.asok' 2024-09-18T16:10:15.860 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/repro-long-log/a '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' --mon-cluster-log-file=td/repro-long-log/log --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T16:10:16.095 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T16:10:16.096 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T16:10:16.096 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:10:16.096 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:10:16.096 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T16:10:16.096 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T16:10:16.097 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:10:16.097 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:10:16.097 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:10:16.097 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:10:16.097 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:10:16.098 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:10:16.098 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.506073/ceph-mon.a.asok 2024-09-18T16:10:16.098 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:10:16.098 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.506073/ceph-mon.a.asok config get fsid 2024-09-18T16:10:16.180 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T16:10:16.180 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:10:16.180 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:10:16.181 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T16:10:16.181 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T16:10:16.181 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:10:16.181 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:10:16.181 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:10:16.181 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:10:16.181 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:10:16.181 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:10:16.181 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.506073/ceph-mon.a.asok 2024-09-18T16:10:16.181 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:10:16.181 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.506073/ceph-mon.a.asok config get mon_host 2024-09-18T16:10:16.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:59: setup_log_test: run_mgr td/repro-long-log x 2024-09-18T16:10:16.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/repro-long-log 2024-09-18T16:10:16.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T16:10:16.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T16:10:16.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T16:10:16.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/repro-long-log/x 2024-09-18T16:10:16.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T16:10:17.101 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T16:10:17.101 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:10:17.101 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:10:17.101 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:10:17.101 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:10:17.101 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:10:17.101 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.506073/$cluster-$name.asok' 2024-09-18T16:10:17.101 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:10:17.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/repro-long-log/x '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:10:17.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:60: setup_log_test: run_osd td/repro-long-log 0 2024-09-18T16:10:17.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-18T16:10:17.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:10:17.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T16:10:17.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:10:17.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/0 2024-09-18T16:10:17.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-18T16:10:17.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:10:17.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:10:17.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:10:17.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/0' 2024-09-18T16:10:17.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/0/journal' 2024-09-18T16:10:17.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:10:17.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:10:17.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-18T16:10:17.128 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:10:17.128 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:10:17.129 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:10:17.129 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:10:17.129 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:10:17.129 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:10:17.129 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.506073/$cluster-$name.asok' 2024-09-18T16:10:17.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' 2024-09-18T16:10:17.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:10:17.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:10:17.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:10:17.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-18T16:10:17.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-18T16:10:17.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:10:17.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:10:17.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:10:17.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:10:17.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:10:17.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:10:17.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/0 2024-09-18T16:10:17.130 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:10:17.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3e81d76c-49dc-4f21-92bc-e249825f2df4 2024-09-18T16:10:17.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3e81d76c-49dc-4f21-92bc-e249825f2df4' 2024-09-18T16:10:17.131 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 3e81d76c-49dc-4f21-92bc-e249825f2df4 2024-09-18T16:10:17.131 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:10:17.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBp++pm8MFgCBAA/Y5TxDGZdeRP0Q9Sf7r+WA== 2024-09-18T16:10:17.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBp++pm8MFgCBAA/Y5TxDGZdeRP0Q9Sf7r+WA=="}' 2024-09-18T16:10:17.141 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3e81d76c-49dc-4f21-92bc-e249825f2df4 -i td/repro-long-log/0/new.json 2024-09-18T16:10:17.449 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:10:17.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/0/new.json 2024-09-18T16:10:17.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBp++pm8MFgCBAA/Y5TxDGZdeRP0Q9Sf7r+WA== --osd-uuid 3e81d76c-49dc-4f21-92bc-e249825f2df4 2024-09-18T16:10:17.482 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:10:17.482+0000 7ff656d21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:10:17.494 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:10:17.494+0000 7ff656d21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:10:17.496 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:10:17.495+0000 7ff656d21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:10:17.496 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:10:17.495+0000 7ff656d21440 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-18T16:10:17.496 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:10:17.495+0000 7ff656d21440 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-18T16:10:17.496 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:10:17.495+0000 7ff656d21440 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-18T16:10:17.535 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:10:17.534+0000 7ff656d21440 -1 bluestore(td/repro-long-log/0) _read_fsid unparsable uuid 2024-09-18T16:10:22.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/0/keyring 2024-09-18T16:10:22.898 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:10:22.899 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T16:10:22.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T16:10:22.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:10:23.401 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T16:10:23.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T16:10:23.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:10:23.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:10:23.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:10:23.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:10:23.422 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:10:23.421+0000 7feab6a71440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:10:23.425 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:10:23.423+0000 7feab6a71440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:10:23.426 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:10:23.425+0000 7feab6a71440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:10:23.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T16:10:23.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:10:23.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T16:10:23.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:10:23.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:10:23.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:23.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:10:23.848 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:10:23.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:23.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:23.958 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:10:23.957+0000 7feab6a71440 -1 Falling back to public interface 2024-09-18T16:10:24.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:10:25.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:10:25.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:25.258 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T16:10:25.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:10:25.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:25.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:25.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:10:26.438 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:10:26.437+0000 7feab6a71440 -1 osd.0 0 log_to_monitors true 2024-09-18T16:10:26.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:10:26.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:26.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:10:26.653 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T16:10:26.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:26.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:27.154 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:10:28.155 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:10:28.155 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:28.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:10:28.156 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:10:28.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:28.156 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:28.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:10:29.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:10:29.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:29.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:10:29.535 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T16:10:29.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:29.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:29.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:10:30.958 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:10:30.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:30.959 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T16:10:30.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:10:30.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:30.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:31.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:10:32.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:10:32.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:32.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:10:32.427 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T16:10:32.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:32.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:32.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:10:33.805 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:10:33.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:33.806 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T16:10:33.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:10:33.806 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:33.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:34.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:10:35.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:10:35.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:35.293 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T16:10:35.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:10:35.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:35.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:35.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:10:36.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:10:36.672 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:36.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:10:36.673 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T16:10:36.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:36.673 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:37.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:10:38.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:10:38.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:38.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:10:38.108 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T16:10:38.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:38.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:38.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:10:39.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:10:39.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:39.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:10:39.818 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T16:10:39.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:39.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:40.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:10:41.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:10:41.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:41.474 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T16:10:41.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:10:41.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:41.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:42.580 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:10:43.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:10:43.582 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T16:10:43.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:43.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:10:43.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:43.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:44.316 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:10:45.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:10:45.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:45.319 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T16:10:45.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:10:45.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:45.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:45.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:10:46.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:10:46.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:46.872 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T16:10:46.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:10:46.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:46.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:47.367 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:10:48.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:10:48.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:48.368 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T16:10:48.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:10:48.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:48.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:48.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:10:49.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:10:49.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:49.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:10:49.972 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T16:10:49.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:49.972 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:10:50.371 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/507164,v1:127.0.0.1:6803/507164] [v2:127.0.0.1:6804/507164,v1:127.0.0.1:6805/507164] exists,up 3e81d76c-49dc-4f21-92bc-e249825f2df4 2024-09-18T16:10:50.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:10:50.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:10:50.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:10:50.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:61: setup_log_test: run_osd td/repro-long-log 1 2024-09-18T16:10:50.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-18T16:10:50.375 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:10:50.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T16:10:50.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:10:50.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/1 2024-09-18T16:10:50.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-18T16:10:50.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:10:50.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:10:50.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:10:50.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/1' 2024-09-18T16:10:50.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/1/journal' 2024-09-18T16:10:50.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:10:50.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:10:50.376 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-18T16:10:50.376 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:10:50.376 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:10:50.377 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:10:50.377 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:10:50.377 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:10:50.377 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:10:50.377 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.506073/$cluster-$name.asok' 2024-09-18T16:10:50.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' 2024-09-18T16:10:50.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:10:50.379 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:10:50.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:10:50.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-18T16:10:50.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-18T16:10:50.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:10:50.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:10:50.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:10:50.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:10:50.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:10:50.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:10:50.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/1 2024-09-18T16:10:50.380 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:10:50.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=47de5d01-a908-4ad4-8ba2-3ba96697c151 2024-09-18T16:10:50.380 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 47de5d01-a908-4ad4-8ba2-3ba96697c151' 2024-09-18T16:10:50.381 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 47de5d01-a908-4ad4-8ba2-3ba96697c151 2024-09-18T16:10:50.381 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:10:50.387 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCK++pmfXITFxAAv1ps2P9smIzcEoKFR/ezOw== 2024-09-18T16:10:50.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCK++pmfXITFxAAv1ps2P9smIzcEoKFR/ezOw=="}' 2024-09-18T16:10:50.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 47de5d01-a908-4ad4-8ba2-3ba96697c151 -i td/repro-long-log/1/new.json 2024-09-18T16:10:50.895 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T16:10:50.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/1/new.json 2024-09-18T16:10:50.907 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCK++pmfXITFxAAv1ps2P9smIzcEoKFR/ezOw== --osd-uuid 47de5d01-a908-4ad4-8ba2-3ba96697c151 2024-09-18T16:10:50.929 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:10:50.928+0000 7f654f045440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:10:50.953 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:10:50.952+0000 7f654f045440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:10:50.954 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:10:50.954+0000 7f654f045440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:10:50.955 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:10:50.954+0000 7f654f045440 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-18T16:10:50.955 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:10:50.954+0000 7f654f045440 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-18T16:10:50.955 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:10:50.954+0000 7f654f045440 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-18T16:10:51.038 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:10:51.036+0000 7f654f045440 -1 bluestore(td/repro-long-log/1) _read_fsid unparsable uuid 2024-09-18T16:10:56.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/1/keyring 2024-09-18T16:10:56.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:10:56.625 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T16:10:56.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T16:10:56.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:10:57.142 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T16:10:57.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T16:10:57.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:10:57.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:10:57.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:10:57.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:10:57.163 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:10:57.162+0000 7fe662409440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:10:57.191 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:10:57.191+0000 7fe662409440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:10:57.193 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:10:57.192+0000 7fe662409440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:10:57.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T16:10:57.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:10:57.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T16:10:57.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:10:57.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:10:57.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:57.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:10:57.572 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:10:57.572 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:57.573 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:10:57.712 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:10:57.711+0000 7fe662409440 -1 Falling back to public interface 2024-09-18T16:10:57.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:10:58.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:10:58.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:10:58.982 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T16:10:58.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:10:58.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:10:58.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:10:59.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:00.196 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:11:00.195+0000 7fe662409440 -1 osd.1 0 log_to_monitors true 2024-09-18T16:11:00.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:00.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:00.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:11:00.396 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T16:11:00.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:00.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:11:00.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:01.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:01.868 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:01.869 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:11:01.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:11:01.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:01.869 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:11:02.354 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:03.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:03.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:03.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:11:03.357 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T16:11:03.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:03.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:11:03.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:04.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:04.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:04.788 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T16:11:04.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:11:04.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:04.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:11:05.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:06.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:06.190 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:06.190 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T16:11:06.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:11:06.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:06.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:11:06.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:07.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:07.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:07.581 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T16:11:07.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:11:07.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:07.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:11:08.095 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:09.096 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:09.096 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:09.097 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T16:11:09.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:11:09.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:09.097 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:11:09.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:10.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:10.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:10.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:11:10.537 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T16:11:10.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:10.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:11:10.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:11.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:11.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:11.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:11:11.953 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T16:11:11.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:11.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:11:12.661 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:13.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:13.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:13.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:11:13.663 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T16:11:13.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:13.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:11:14.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:15.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:15.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:15.527 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T16:11:15.527 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:11:15.527 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:15.527 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:11:16.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:17.159 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:17.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:17.160 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T16:11:17.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:11:17.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:17.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:11:18.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:19.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:19.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:19.186 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:11:19.186 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T16:11:19.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:19.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:11:19.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:20.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:20.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:20.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:11:20.978 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T16:11:20.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:20.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:11:21.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:22.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:22.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:22.955 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T16:11:22.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:11:22.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:22.955 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:11:23.331 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/508918,v1:127.0.0.1:6811/508918] [v2:127.0.0.1:6812/508918,v1:127.0.0.1:6813/508918] exists,up 47de5d01-a908-4ad4-8ba2-3ba96697c151 2024-09-18T16:11:23.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:11:23.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:11:23.334 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:11:23.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:62: setup_log_test: run_osd td/repro-long-log 2 2024-09-18T16:11:23.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-18T16:11:23.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:11:23.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T16:11:23.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:11:23.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/2 2024-09-18T16:11:23.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-18T16:11:23.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:11:23.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:11:23.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:11:23.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/2' 2024-09-18T16:11:23.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/2/journal' 2024-09-18T16:11:23.335 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:11:23.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:11:23.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-18T16:11:23.336 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:11:23.336 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:11:23.336 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:11:23.336 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:11:23.336 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:11:23.336 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:11:23.336 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.506073/$cluster-$name.asok' 2024-09-18T16:11:23.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' 2024-09-18T16:11:23.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:11:23.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:11:23.336 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:11:23.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-18T16:11:23.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-18T16:11:23.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:11:23.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:11:23.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:11:23.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:11:23.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:11:23.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:11:23.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/2 2024-09-18T16:11:23.337 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:11:23.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ef15750d-943a-43e3-8334-799190d2c0f2 2024-09-18T16:11:23.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 ef15750d-943a-43e3-8334-799190d2c0f2' 2024-09-18T16:11:23.338 INFO:tasks.workunit.client.0.smithi047.stdout:add osd2 ef15750d-943a-43e3-8334-799190d2c0f2 2024-09-18T16:11:23.338 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:11:23.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCr++pmpLK0FBAACkHNldVXjwjvsb/dNF6c1w== 2024-09-18T16:11:23.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCr++pmpLK0FBAACkHNldVXjwjvsb/dNF6c1w=="}' 2024-09-18T16:11:23.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ef15750d-943a-43e3-8334-799190d2c0f2 -i td/repro-long-log/2/new.json 2024-09-18T16:11:23.823 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T16:11:23.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/2/new.json 2024-09-18T16:11:23.835 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCr++pmpLK0FBAACkHNldVXjwjvsb/dNF6c1w== --osd-uuid ef15750d-943a-43e3-8334-799190d2c0f2 2024-09-18T16:11:23.857 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:11:23.856+0000 7f52ba62f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:11:23.859 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:11:23.858+0000 7f52ba62f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:11:23.860 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:11:23.859+0000 7f52ba62f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:11:23.860 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:11:23.859+0000 7f52ba62f440 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-18T16:11:23.861 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:11:23.859+0000 7f52ba62f440 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-18T16:11:23.861 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:11:23.859+0000 7f52ba62f440 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-18T16:11:23.939 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:11:23.938+0000 7f52ba62f440 -1 bluestore(td/repro-long-log/2) _read_fsid unparsable uuid 2024-09-18T16:11:29.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/2/keyring 2024-09-18T16:11:29.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:11:29.803 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd2 key to auth repository 2024-09-18T16:11:29.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T16:11:29.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:11:30.308 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T16:11:30.308 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T16:11:30.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:11:30.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:11:30.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:11:30.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:11:30.329 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:11:30.329+0000 7f6d0777b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:11:30.331 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:11:30.331+0000 7f6d0777b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:11:30.332 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:11:30.332+0000 7f6d0777b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:11:30.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T16:11:30.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:11:30.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T16:11:30.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:11:30.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:11:30.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:30.717 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:11:30.718 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:11:30.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:30.718 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:11:30.846 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:11:30.845+0000 7f6d0777b440 -1 Falling back to public interface 2024-09-18T16:11:31.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:32.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:32.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:32.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:11:32.088 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T16:11:32.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:32.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:11:32.464 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:33.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:33.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:33.466 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T16:11:33.466 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:11:33.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:33.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:11:33.484 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:11:33.483+0000 7f6d0777b440 -1 osd.2 0 log_to_monitors true 2024-09-18T16:11:33.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:34.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:34.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:34.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:11:34.885 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:11:34.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:34.885 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:11:35.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:36.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:36.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:36.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:11:36.291 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T16:11:36.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:36.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:11:36.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:37.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:37.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:37.698 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T16:11:37.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:11:37.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:37.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:11:38.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:39.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:39.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:39.104 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T16:11:39.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:11:39.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:39.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:11:39.568 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:40.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:40.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:40.570 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T16:11:40.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:11:40.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:40.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:11:40.948 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:41.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:41.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:41.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:11:41.950 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T16:11:41.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:41.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:11:42.420 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:43.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:43.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:43.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:11:43.422 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T16:11:43.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:43.422 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:11:43.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:44.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:44.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:44.874 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T16:11:44.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:11:44.875 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:44.875 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:11:45.871 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:46.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:46.872 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:46.872 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T16:11:46.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:11:46.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:46.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:11:47.628 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:48.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:48.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:48.630 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T16:11:48.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:11:48.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:48.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:11:49.055 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:50.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:50.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:50.057 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T16:11:50.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:11:50.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:50.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:11:50.856 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:51.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:51.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:51.858 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T16:11:51.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:11:51.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:51.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:11:52.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:53.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:53.531 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T16:11:53.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:53.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:11:53.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:53.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:11:54.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:11:55.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:11:55.524 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:11:55.524 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T16:11:55.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:11:55.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:11:55.525 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:11:55.944 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/510647,v1:127.0.0.1:6819/510647] [v2:127.0.0.1:6820/510647,v1:127.0.0.1:6821/510647] exists,up ef15750d-943a-43e3-8334-799190d2c0f2 2024-09-18T16:11:55.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:11:55.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:11:55.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:11:55.945 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:64: setup_log_test: ceph osd pool create test 1 1 2024-09-18T16:11:56.439 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test' already exists 2024-09-18T16:11:56.450 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: ceph osd dump --format json 2024-09-18T16:11:56.450 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: jq '.pools[] | select(.pool_name == "test") | .pool' 2024-09-18T16:11:56.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: POOL_ID=1 2024-09-18T16:11:56.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:66: setup_log_test: PGID=1.0 2024-09-18T16:11:56.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:69: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 20 2024-09-18T16:11:57.091 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: {} 2024-09-18T16:11:57.118 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: {} 2024-09-18T16:11:57.143 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: {} 2024-09-18T16:11:57.152 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:70: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 20 2024-09-18T16:11:57.284 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: {} 2024-09-18T16:11:57.310 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: {} 2024-09-18T16:11:57.335 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: {} 2024-09-18T16:11:57.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:71: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-max-pg-log-entries 30 2024-09-18T16:11:57.477 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: {} 2024-09-18T16:11:57.502 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: {} 2024-09-18T16:11:57.527 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: {} 2024-09-18T16:11:57.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:72: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 10 2024-09-18T16:11:57.661 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: {} 2024-09-18T16:11:57.688 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: {} 2024-09-18T16:11:57.713 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: {} 2024-09-18T16:11:57.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:73: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 20 2024-09-18T16:11:57.846 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: {} 2024-09-18T16:11:57.872 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: {} 2024-09-18T16:11:57.897 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: {} 2024-09-18T16:11:57.906 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:75: setup_log_test: touch td/repro-long-log/foo 2024-09-18T16:11:57.907 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: seq 1 20 2024-09-18T16:11:57.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:11:57.908 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:11:58.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:11:58.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:11:58.247 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:11:58.247 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:11:58.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:11:58.347 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:11:58.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:11:58.456 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:11:58.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:11:58.548 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:11:58.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:11:58.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:11:58.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:11:58.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:11:58.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:11:58.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:11:59.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:11:59.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:11:59.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:11:59.134 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:11:59.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:11:59.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:11:59.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:11:59.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:11:59.419 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:11:59.419 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:11:59.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:11:59.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:11:59.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:11:59.629 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:11:59.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:11:59.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:11:59.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:11:59.855 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:11:59.947 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:11:59.947 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:12:00.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:12:00.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:12:00.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:81: setup_log_test: test_log_size 1.0 20 2024-09-18T16:12:00.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-18T16:12:00.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=20 2024-09-18T16:12:00.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-18T16:12:00.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-18T16:12:00.229 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: 21474836500 2024-09-18T16:12:00.252 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: 42949672975 2024-09-18T16:12:00.273 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: 64424509448 2024-09-18T16:12:00.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-18T16:12:03.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-18T16:12:03.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-18T16:12:03.406 INFO:tasks.workunit.client.0.smithi047.stdout:20 2024-09-18T16:12:03.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-18T16:12:03.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-18T16:12:03.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 20 2024-09-18T16:12:03.519 INFO:tasks.workunit.client.0.smithi047.stdout:20 2024-09-18T16:12:03.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-18T16:12:03.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:12:03.630 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:12:03.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-18T16:12:03.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:12:03.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-18T16:12:03.740 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:12:03.741 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:83: setup_log_test: rados -p test rm foo 2024-09-18T16:12:03.826 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: seq 1 20 2024-09-18T16:12:03.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:12:03.827 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:12:03.916 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:12:03.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:12:03.917 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:12:04.016 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:12:04.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:12:04.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:12:04.108 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:12:04.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:12:04.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:12:04.234 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:12:04.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:12:04.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:12:04.327 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:12:04.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:12:04.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:12:04.419 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:12:04.420 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:12:04.420 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:12:04.511 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:12:04.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:12:04.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:12:04.603 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:12:04.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:12:04.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:12:04.695 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:12:04.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:12:04.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:12:04.821 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:12:04.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:12:04.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:12:04.930 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:12:04.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:12:04.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:12:05.014 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:12:05.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:12:05.015 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:12:05.097 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:12:05.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:12:05.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:12:05.198 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:12:05.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:12:05.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:12:05.315 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:12:05.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:12:05.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:12:05.408 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:12:05.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:12:05.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:12:05.500 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:12:05.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:12:05.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:12:05.609 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:12:05.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:12:05.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:12:05.717 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:12:05.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:12:05.719 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:12:05.829 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:12:05.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:92: setup_log_test: test_log_size 1.0 21 2024-09-18T16:12:05.831 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-18T16:12:05.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=21 2024-09-18T16:12:05.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-18T16:12:05.832 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-18T16:12:05.929 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: 21474836503 2024-09-18T16:12:05.951 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: 42949672978 2024-09-18T16:12:05.973 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: 64424509452 2024-09-18T16:12:05.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-18T16:12:08.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-18T16:12:08.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-18T16:12:09.106 INFO:tasks.workunit.client.0.smithi047.stdout:21 2024-09-18T16:12:09.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-18T16:12:09.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-18T16:12:09.108 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 21 2024-09-18T16:12:09.218 INFO:tasks.workunit.client.0.smithi047.stdout:21 2024-09-18T16:12:09.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-18T16:12:09.219 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:12:09.329 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:12:09.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-18T16:12:09.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:12:09.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-18T16:12:09.440 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:12:09.440 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:101: TEST_repro_long_log1: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:12:09.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:102: TEST_repro_long_log1: test_log_size 1.0 22 2024-09-18T16:12:09.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-18T16:12:09.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=22 2024-09-18T16:12:09.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-18T16:12:09.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-18T16:12:09.633 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: 21474836505 2024-09-18T16:12:09.652 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: 42949672981 2024-09-18T16:12:09.674 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: 64424509454 2024-09-18T16:12:09.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-18T16:12:12.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-18T16:12:12.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-18T16:12:12.809 INFO:tasks.workunit.client.0.smithi047.stdout:22 2024-09-18T16:12:12.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-18T16:12:12.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-18T16:12:12.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 22 2024-09-18T16:12:12.938 INFO:tasks.workunit.client.0.smithi047.stdout:22 2024-09-18T16:12:12.938 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-18T16:12:12.938 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:12:13.049 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:12:13.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-18T16:12:13.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:12:13.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-18T16:12:13.160 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:12:13.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:35: run: teardown td/repro-long-log 2024-09-18T16:12:13.161 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-09-18T16:12:13.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:12:13.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-09-18T16:12:13.162 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:12:13.162 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:12:13.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:12:13.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:12:13.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:12:13.269 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:12:13.269 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:12:13.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:12:13.270 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:12:13.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:12:13.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:12:13.271 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:12:13.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:12:13.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:12:13.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:12:13.272 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:12:13.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:12:13.274 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:12:13.275 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-09-18T16:12:13.289 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:12:13.289 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:12:13.289 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:12:13.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.506073 2024-09-18T16:12:13.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:12:13.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:12:13.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:32: run: for func in $funcs 2024-09-18T16:12:13.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:33: run: setup td/repro-long-log 2024-09-18T16:12:13.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/repro-long-log 2024-09-18T16:12:13.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/repro-long-log 2024-09-18T16:12:13.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-09-18T16:12:13.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:12:13.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-09-18T16:12:13.291 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:12:13.291 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:12:13.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:12:13.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:12:13.291 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:12:13.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:12:13.292 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:12:13.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:12:13.293 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:12:13.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:12:13.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:12:13.294 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:12:13.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:12:13.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:12:13.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:12:13.296 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:12:13.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:12:13.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:12:13.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-09-18T16:12:13.298 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:12:13.299 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:12:13.299 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:12:13.299 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.506073 2024-09-18T16:12:13.299 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:12:13.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:12:13.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/repro-long-log 2024-09-18T16:12:13.301 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T16:12:13.301 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:12:13.301 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:12:13.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.506073 2024-09-18T16:12:13.303 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T16:12:13.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T16:12:13.303 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T16:12:13.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/repro-long-log 1' TERM HUP INT 2024-09-18T16:12:13.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:34: run: TEST_repro_long_log2 td/repro-long-log 2024-09-18T16:12:13.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:107: TEST_repro_long_log2: local dir=td/repro-long-log 2024-09-18T16:12:13.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:109: TEST_repro_long_log2: setup_log_test td/repro-long-log 2024-09-18T16:12:13.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:55: setup_log_test: local dir=td/repro-long-log 2024-09-18T16:12:13.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:56: setup_log_test: local which= 2024-09-18T16:12:13.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:58: setup_log_test: run_mon td/repro-long-log a 2024-09-18T16:12:13.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/repro-long-log 2024-09-18T16:12:13.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T16:12:13.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T16:12:13.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T16:12:13.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/repro-long-log/a 2024-09-18T16:12:13.304 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/repro-long-log/a --run-dir=td/repro-long-log 2024-09-18T16:12:13.649 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T16:12:13.649 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:12:13.649 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:12:13.649 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:12:13.649 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:12:13.649 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:12:13.650 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.506073/$cluster-$name.asok' 2024-09-18T16:12:13.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/repro-long-log/a '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' --mon-cluster-log-file=td/repro-long-log/log --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T16:12:13.942 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T16:12:13.943 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T16:12:13.943 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:12:13.943 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:12:13.943 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T16:12:13.944 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T16:12:13.944 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:12:13.944 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:12:13.944 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:12:13.944 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:12:13.944 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:12:13.945 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:12:13.945 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.506073/ceph-mon.a.asok 2024-09-18T16:12:13.945 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:12:13.945 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.506073/ceph-mon.a.asok config get fsid 2024-09-18T16:12:14.026 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T16:12:14.026 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:12:14.026 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:12:14.027 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T16:12:14.027 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T16:12:14.027 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:12:14.027 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:12:14.027 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:12:14.027 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:12:14.027 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:12:14.027 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:12:14.027 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.506073/ceph-mon.a.asok 2024-09-18T16:12:14.028 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:12:14.028 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.506073/ceph-mon.a.asok config get mon_host 2024-09-18T16:12:14.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:59: setup_log_test: run_mgr td/repro-long-log x 2024-09-18T16:12:14.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/repro-long-log 2024-09-18T16:12:14.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T16:12:14.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T16:12:14.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T16:12:14.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/repro-long-log/x 2024-09-18T16:12:14.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T16:12:14.955 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T16:12:14.955 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:12:14.955 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:12:14.955 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:12:14.955 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:12:14.956 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:12:14.956 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.506073/$cluster-$name.asok' 2024-09-18T16:12:14.956 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:12:14.956 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/repro-long-log/x '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:12:14.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:60: setup_log_test: run_osd td/repro-long-log 0 2024-09-18T16:12:14.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-18T16:12:14.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:12:14.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T16:12:14.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:12:14.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/0 2024-09-18T16:12:14.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-18T16:12:14.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:12:14.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:12:14.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:12:14.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/0' 2024-09-18T16:12:14.987 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/0/journal' 2024-09-18T16:12:14.988 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:12:14.988 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:12:14.988 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-18T16:12:14.988 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:12:14.988 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:12:14.989 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:12:14.989 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:12:14.989 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:12:14.989 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:12:14.989 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.506073/$cluster-$name.asok' 2024-09-18T16:12:14.989 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' 2024-09-18T16:12:14.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:12:14.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:12:14.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:12:14.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-18T16:12:14.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-18T16:12:14.990 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:12:14.991 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:12:14.991 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:12:14.991 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:12:14.991 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:12:14.991 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:12:14.991 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/0 2024-09-18T16:12:14.992 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:12:14.992 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f623d4d0-dca3-4d28-af43-437d81a41258 2024-09-18T16:12:14.992 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f623d4d0-dca3-4d28-af43-437d81a41258' 2024-09-18T16:12:14.992 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 f623d4d0-dca3-4d28-af43-437d81a41258 2024-09-18T16:12:14.993 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:12:14.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDe++pmJIFYOxAAsr9LtrGG2lRVznpjfTPldg== 2024-09-18T16:12:14.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDe++pmJIFYOxAAsr9LtrGG2lRVznpjfTPldg=="}' 2024-09-18T16:12:14.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f623d4d0-dca3-4d28-af43-437d81a41258 -i td/repro-long-log/0/new.json 2024-09-18T16:12:15.280 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:12:15.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/0/new.json 2024-09-18T16:12:15.290 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDe++pmJIFYOxAAsr9LtrGG2lRVznpjfTPldg== --osd-uuid f623d4d0-dca3-4d28-af43-437d81a41258 2024-09-18T16:12:15.313 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:12:15.312+0000 7ffa3608d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:12:15.325 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:12:15.324+0000 7ffa3608d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:12:15.326 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:12:15.325+0000 7ffa3608d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:12:15.327 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:12:15.326+0000 7ffa3608d440 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-18T16:12:15.327 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:12:15.326+0000 7ffa3608d440 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-18T16:12:15.327 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:12:15.326+0000 7ffa3608d440 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-18T16:12:15.366 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:12:15.365+0000 7ffa3608d440 -1 bluestore(td/repro-long-log/0) _read_fsid unparsable uuid 2024-09-18T16:12:20.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/0/keyring 2024-09-18T16:12:20.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:12:20.765 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T16:12:20.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T16:12:20.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:12:21.313 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T16:12:21.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T16:12:21.313 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:12:21.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:12:21.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:12:21.314 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:12:21.334 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:12:21.333+0000 7efc9668d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:12:21.355 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:12:21.354+0000 7efc9668d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:12:21.356 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:12:21.355+0000 7efc9668d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:12:21.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T16:12:21.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:12:21.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T16:12:21.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:12:21.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:12:21.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:12:21.705 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:12:21.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:12:21.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:12:21.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:12:21.885 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:12:21.884+0000 7efc9668d440 -1 Falling back to public interface 2024-09-18T16:12:22.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:12:23.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:12:23.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:12:23.112 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T16:12:23.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:12:23.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:12:23.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:12:23.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:12:24.350 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:12:24.349+0000 7efc9668d440 -1 osd.0 0 log_to_monitors true 2024-09-18T16:12:24.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:12:24.522 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:12:24.523 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T16:12:24.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:12:24.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:12:24.523 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:12:24.919 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:12:25.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:12:25.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:12:25.921 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:12:25.921 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:12:25.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:12:25.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:12:26.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:12:27.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:12:27.365 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:12:27.365 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T16:12:27.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:12:27.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:12:27.366 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:12:27.740 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:12:28.741 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:12:28.741 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:12:28.742 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T16:12:28.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:12:28.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:12:28.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:12:29.175 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:12:30.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:12:30.176 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:12:30.177 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T16:12:30.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:12:30.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:12:30.177 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:12:30.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:12:31.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:12:31.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:12:31.557 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:12:31.557 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T16:12:31.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:12:31.558 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:12:32.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:12:33.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:12:33.037 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:12:33.038 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T16:12:33.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:12:33.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:12:33.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:12:33.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:12:34.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:12:34.452 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:12:34.452 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T16:12:34.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:12:34.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:12:34.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:12:34.861 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:12:35.862 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:12:35.862 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:12:35.863 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T16:12:35.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:12:35.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:12:35.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:12:36.686 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:12:37.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:12:37.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:12:37.688 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T16:12:37.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:12:37.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:12:37.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:12:38.170 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:12:39.171 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:12:39.172 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:12:39.172 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:12:39.172 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T16:12:39.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:12:39.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:12:39.611 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:12:40.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:12:40.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:12:40.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:12:40.613 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T16:12:40.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:12:40.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:12:41.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:12:42.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:12:42.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:12:42.138 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T16:12:42.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:12:42.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:12:42.139 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:12:42.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:12:43.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:12:43.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:12:43.797 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T16:12:43.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:12:43.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:12:43.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:12:44.893 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:12:45.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:12:45.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:12:45.894 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:12:45.895 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T16:12:45.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:12:45.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:12:46.288 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/514084,v1:127.0.0.1:6803/514084] [v2:127.0.0.1:6804/514084,v1:127.0.0.1:6805/514084] exists,up f623d4d0-dca3-4d28-af43-437d81a41258 2024-09-18T16:12:46.292 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:12:46.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:12:46.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:12:46.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:61: setup_log_test: run_osd td/repro-long-log 1 2024-09-18T16:12:46.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-18T16:12:46.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:12:46.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T16:12:46.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:12:46.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/1 2024-09-18T16:12:46.293 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-18T16:12:46.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:12:46.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:12:46.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:12:46.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/1' 2024-09-18T16:12:46.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/1/journal' 2024-09-18T16:12:46.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:12:46.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:12:46.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-18T16:12:46.294 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:12:46.294 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:12:46.294 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:12:46.294 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:12:46.294 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:12:46.294 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:12:46.295 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.506073/$cluster-$name.asok' 2024-09-18T16:12:46.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' 2024-09-18T16:12:46.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:12:46.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:12:46.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:12:46.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-18T16:12:46.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-18T16:12:46.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:12:46.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:12:46.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:12:46.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:12:46.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:12:46.295 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:12:46.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/1 2024-09-18T16:12:46.296 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:12:46.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4ed4fe30-a6c1-4c05-b879-04ff2ae92aa2 2024-09-18T16:12:46.296 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 4ed4fe30-a6c1-4c05-b879-04ff2ae92aa2' 2024-09-18T16:12:46.296 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 4ed4fe30-a6c1-4c05-b879-04ff2ae92aa2 2024-09-18T16:12:46.296 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:12:46.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD+++pmvTswEhAAj7XDWK+FYlRUbt3xvUrf3w== 2024-09-18T16:12:46.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD+++pmvTswEhAAj7XDWK+FYlRUbt3xvUrf3w=="}' 2024-09-18T16:12:46.306 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4ed4fe30-a6c1-4c05-b879-04ff2ae92aa2 -i td/repro-long-log/1/new.json 2024-09-18T16:12:46.888 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T16:12:46.899 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/1/new.json 2024-09-18T16:12:46.900 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD+++pmvTswEhAAj7XDWK+FYlRUbt3xvUrf3w== --osd-uuid 4ed4fe30-a6c1-4c05-b879-04ff2ae92aa2 2024-09-18T16:12:46.924 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:12:46.923+0000 7f192cc0e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:12:46.925 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:12:46.925+0000 7f192cc0e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:12:46.926 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:12:46.926+0000 7f192cc0e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:12:46.927 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:12:46.926+0000 7f192cc0e440 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-18T16:12:46.927 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:12:46.926+0000 7f192cc0e440 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-18T16:12:46.927 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:12:46.926+0000 7f192cc0e440 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-18T16:12:47.054 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:12:47.053+0000 7f192cc0e440 -1 bluestore(td/repro-long-log/1) _read_fsid unparsable uuid 2024-09-18T16:12:52.666 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/1/keyring 2024-09-18T16:12:52.666 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:12:52.667 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T16:12:52.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T16:12:52.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:12:53.173 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T16:12:53.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T16:12:53.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:12:53.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:12:53.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:12:53.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:12:53.195 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:12:53.194+0000 7f079528b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:12:53.209 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:12:53.208+0000 7f079528b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:12:53.210 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:12:53.209+0000 7f079528b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:12:53.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T16:12:53.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:12:53.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T16:12:53.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:12:53.570 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:12:53.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:12:53.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:12:53.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:12:53.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:12:53.571 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:12:53.744 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:12:53.743+0000 7f079528b440 -1 Falling back to public interface 2024-09-18T16:12:53.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:12:54.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:12:54.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:12:54.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:12:54.975 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T16:12:54.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:12:54.976 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:12:55.400 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:12:56.214 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:12:56.213+0000 7f079528b440 -1 osd.1 0 log_to_monitors true 2024-09-18T16:12:56.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:12:56.401 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:12:56.402 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T16:12:56.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:12:56.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:12:56.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:12:56.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:12:57.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:12:57.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:12:57.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:12:57.933 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:12:57.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:12:57.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:12:58.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:12:59.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:12:59.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:12:59.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:12:59.329 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T16:12:59.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:12:59.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:12:59.761 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:00.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:00.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:00.763 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T16:13:00.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:13:00.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:00.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:13:01.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:02.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:02.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:02.256 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T16:13:02.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:13:02.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:02.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:13:02.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:03.688 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:03.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:03.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:13:03.689 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T16:13:03.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:03.689 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:13:04.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:05.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:05.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:05.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:13:05.103 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T16:13:05.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:05.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:13:05.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:06.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:06.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:06.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:13:06.556 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T16:13:06.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:06.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:13:06.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:07.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:07.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:07.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:13:07.935 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T16:13:07.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:07.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:13:08.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:09.361 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:09.361 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T16:13:09.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:09.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:13:09.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:09.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:13:10.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:11.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:11.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:11.384 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T16:13:11.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:13:11.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:11.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:13:12.101 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:13.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:13.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:13.103 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T16:13:13.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:13:13.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:13.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:13:13.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:14.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:14.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:14.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:13:14.881 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T16:13:14.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:14.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:13:15.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:16.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:16.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:16.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:13:16.634 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T16:13:16.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:16.634 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:13:17.574 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:18.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:18.575 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:18.575 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T16:13:18.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:13:18.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:18.576 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:13:19.065 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/515813,v1:127.0.0.1:6811/515813] [v2:127.0.0.1:6812/515813,v1:127.0.0.1:6813/515813] exists,up 4ed4fe30-a6c1-4c05-b879-04ff2ae92aa2 2024-09-18T16:13:19.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:13:19.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:13:19.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:13:19.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:62: setup_log_test: run_osd td/repro-long-log 2 2024-09-18T16:13:19.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-18T16:13:19.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:13:19.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T16:13:19.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:13:19.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/2 2024-09-18T16:13:19.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-18T16:13:19.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:13:19.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:13:19.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:13:19.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/2' 2024-09-18T16:13:19.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/2/journal' 2024-09-18T16:13:19.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:13:19.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:13:19.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-18T16:13:19.069 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:13:19.070 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:13:19.070 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:13:19.070 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:13:19.070 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:13:19.070 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:13:19.070 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.506073/$cluster-$name.asok' 2024-09-18T16:13:19.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' 2024-09-18T16:13:19.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:13:19.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:13:19.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:13:19.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-18T16:13:19.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-18T16:13:19.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:13:19.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:13:19.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:13:19.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:13:19.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:13:19.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:13:19.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/2 2024-09-18T16:13:19.071 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:13:19.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2b2f97bd-7033-4128-94d9-b2b6abf18d96 2024-09-18T16:13:19.071 INFO:tasks.workunit.client.0.smithi047.stdout:add osd2 2b2f97bd-7033-4128-94d9-b2b6abf18d96 2024-09-18T16:13:19.072 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 2b2f97bd-7033-4128-94d9-b2b6abf18d96' 2024-09-18T16:13:19.072 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:13:19.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAf/OpmRTDTBBAAReDpzy0P0X45BJ/P2xmGqQ== 2024-09-18T16:13:19.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAf/OpmRTDTBBAAReDpzy0P0X45BJ/P2xmGqQ=="}' 2024-09-18T16:13:19.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2b2f97bd-7033-4128-94d9-b2b6abf18d96 -i td/repro-long-log/2/new.json 2024-09-18T16:13:19.690 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T16:13:19.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/2/new.json 2024-09-18T16:13:19.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAf/OpmRTDTBBAAReDpzy0P0X45BJ/P2xmGqQ== --osd-uuid 2b2f97bd-7033-4128-94d9-b2b6abf18d96 2024-09-18T16:13:19.725 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:13:19.724+0000 7f4c1d50f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:13:19.727 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:13:19.726+0000 7f4c1d50f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:13:19.728 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:13:19.727+0000 7f4c1d50f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:13:19.728 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:13:19.727+0000 7f4c1d50f440 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-18T16:13:19.728 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:13:19.727+0000 7f4c1d50f440 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-18T16:13:19.728 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:13:19.727+0000 7f4c1d50f440 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-18T16:13:19.805 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:13:19.804+0000 7f4c1d50f440 -1 bluestore(td/repro-long-log/2) _read_fsid unparsable uuid 2024-09-18T16:13:25.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/2/keyring 2024-09-18T16:13:25.369 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:13:25.370 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd2 key to auth repository 2024-09-18T16:13:25.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T16:13:25.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:13:25.881 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T16:13:25.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T16:13:25.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:13:25.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:13:25.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:13:25.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:13:25.903 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:13:25.902+0000 7f5bc200d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:13:25.921 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:13:25.920+0000 7f5bc200d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:13:25.922 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:13:25.921+0000 7f5bc200d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:13:26.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T16:13:26.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:13:26.317 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T16:13:26.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:13:26.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:13:26.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:26.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:13:26.318 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:13:26.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:26.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:13:26.459 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:13:26.458+0000 7f5bc200d440 -1 Falling back to public interface 2024-09-18T16:13:26.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:27.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:27.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:27.684 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T16:13:27.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:13:27.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:27.685 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:13:28.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:28.959 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:13:28.958+0000 7f5bc200d440 -1 osd.2 0 log_to_monitors true 2024-09-18T16:13:29.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:29.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:29.060 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T16:13:29.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:13:29.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:29.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:13:29.594 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:30.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:30.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:30.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:13:30.596 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:13:30.596 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:30.597 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:13:31.023 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:32.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:32.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:32.025 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T16:13:32.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:13:32.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:32.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:13:32.404 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:33.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:33.405 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:33.405 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T16:13:33.406 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:13:33.406 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:33.406 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:13:33.877 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:34.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:34.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:34.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:13:34.879 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T16:13:34.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:34.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:13:35.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:36.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:36.278 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:36.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:13:36.279 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T16:13:36.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:36.279 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:13:36.700 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:37.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:37.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:37.702 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T16:13:37.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:13:37.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:37.702 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:13:38.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:39.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:39.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:39.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:13:39.132 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T16:13:39.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:39.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:13:39.696 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:40.697 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:40.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:40.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:13:40.698 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T16:13:40.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:40.698 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:13:41.666 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:42.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:42.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:42.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:13:42.668 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T16:13:42.668 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:42.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:13:43.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:44.194 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:44.194 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:44.194 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T16:13:44.194 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:13:44.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:44.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:13:44.873 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:45.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:45.874 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:45.875 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T16:13:45.875 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:13:45.875 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:45.875 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:13:46.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:47.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:47.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:47.603 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T16:13:47.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:13:47.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:47.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:13:48.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:49.415 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:49.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:49.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:13:49.416 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T16:13:49.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:49.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:13:50.490 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:13:51.491 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:13:51.492 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:13:51.492 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:13:51.492 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T16:13:51.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:13:51.493 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:13:51.934 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/517542,v1:127.0.0.1:6819/517542] [v2:127.0.0.1:6820/517542,v1:127.0.0.1:6821/517542] exists,up 2b2f97bd-7033-4128-94d9-b2b6abf18d96 2024-09-18T16:13:51.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:13:51.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:13:51.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:13:51.935 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:64: setup_log_test: ceph osd pool create test 1 1 2024-09-18T16:13:52.567 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test' already exists 2024-09-18T16:13:52.577 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: ceph osd dump --format json 2024-09-18T16:13:52.578 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: jq '.pools[] | select(.pool_name == "test") | .pool' 2024-09-18T16:13:52.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: POOL_ID=1 2024-09-18T16:13:52.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:66: setup_log_test: PGID=1.0 2024-09-18T16:13:52.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:69: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 20 2024-09-18T16:13:53.052 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: {} 2024-09-18T16:13:53.078 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: {} 2024-09-18T16:13:53.103 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: {} 2024-09-18T16:13:53.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:70: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 20 2024-09-18T16:13:53.212 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: {} 2024-09-18T16:13:53.238 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: {} 2024-09-18T16:13:53.264 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: {} 2024-09-18T16:13:53.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:71: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-max-pg-log-entries 30 2024-09-18T16:13:53.373 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: {} 2024-09-18T16:13:53.399 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: {} 2024-09-18T16:13:53.424 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: {} 2024-09-18T16:13:53.433 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:72: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 10 2024-09-18T16:13:53.534 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: {} 2024-09-18T16:13:53.560 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: {} 2024-09-18T16:13:53.586 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: {} 2024-09-18T16:13:53.595 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:73: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 20 2024-09-18T16:13:53.720 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: {} 2024-09-18T16:13:53.747 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: {} 2024-09-18T16:13:53.772 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: {} 2024-09-18T16:13:53.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:75: setup_log_test: touch td/repro-long-log/foo 2024-09-18T16:13:53.782 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: seq 1 20 2024-09-18T16:13:53.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:13:53.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:13:54.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:13:54.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:13:55.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:13:55.158 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:13:55.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:13:55.309 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:13:55.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:13:55.410 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:13:55.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:13:55.526 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:13:55.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:13:55.627 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:13:55.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:13:55.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:13:55.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:13:55.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:13:55.953 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:13:55.954 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:13:56.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:13:56.071 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:13:56.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:13:56.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:13:56.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:13:56.322 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:13:56.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:13:56.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:13:56.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:13:56.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:13:56.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:13:56.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:13:56.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:13:56.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:13:56.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:13:56.866 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:13:56.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:13:56.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:13:57.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:13:57.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:13:57.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:81: setup_log_test: test_log_size 1.0 20 2024-09-18T16:13:57.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-18T16:13:57.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=20 2024-09-18T16:13:57.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-18T16:13:57.235 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-18T16:13:57.336 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: 21474836500 2024-09-18T16:13:57.358 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: 42949672973 2024-09-18T16:13:57.379 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: 64424509449 2024-09-18T16:13:57.388 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-18T16:14:00.390 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-18T16:14:00.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-18T16:14:00.512 INFO:tasks.workunit.client.0.smithi047.stdout:20 2024-09-18T16:14:00.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-18T16:14:00.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-18T16:14:00.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 20 2024-09-18T16:14:00.626 INFO:tasks.workunit.client.0.smithi047.stdout:20 2024-09-18T16:14:00.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-18T16:14:00.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:14:00.737 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:14:00.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-18T16:14:00.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:14:00.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-18T16:14:00.849 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:14:00.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:83: setup_log_test: rados -p test rm foo 2024-09-18T16:14:00.979 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: seq 1 20 2024-09-18T16:14:00.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:14:00.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:14:01.085 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:14:01.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:14:01.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:14:01.186 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:14:01.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:14:01.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:14:01.278 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:14:01.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:14:01.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:14:01.379 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:14:01.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:14:01.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:14:01.471 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:14:01.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:14:01.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:14:01.588 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:14:01.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:14:01.590 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:14:01.722 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:14:01.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:14:01.724 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:14:01.823 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:14:01.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:14:01.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:14:01.923 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:14:01.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:14:01.925 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:14:02.041 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:14:02.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:14:02.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:14:02.158 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:14:02.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:14:02.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:14:02.258 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:14:02.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:14:02.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:14:02.375 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:14:02.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:14:02.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:14:02.476 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:14:02.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:14:02.478 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:14:02.576 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:14:02.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:14:02.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:14:02.685 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:14:02.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:14:02.687 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:14:02.803 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:14:02.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:14:02.804 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:14:02.895 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:14:02.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:14:02.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:14:02.987 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:14:02.989 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:14:02.989 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:14:03.130 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:14:03.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:92: setup_log_test: test_log_size 1.0 21 2024-09-18T16:14:03.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-18T16:14:03.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=21 2024-09-18T16:14:03.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-18T16:14:03.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-18T16:14:03.230 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: 21474836503 2024-09-18T16:14:03.252 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: 42949672976 2024-09-18T16:14:03.273 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: 64424509453 2024-09-18T16:14:03.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-18T16:14:06.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-18T16:14:06.284 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-18T16:14:06.407 INFO:tasks.workunit.client.0.smithi047.stdout:21 2024-09-18T16:14:06.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-18T16:14:06.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-18T16:14:06.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 21 2024-09-18T16:14:06.518 INFO:tasks.workunit.client.0.smithi047.stdout:21 2024-09-18T16:14:06.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-18T16:14:06.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:14:06.630 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:14:06.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-18T16:14:06.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:14:06.631 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-18T16:14:06.741 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:14:06.742 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:110: TEST_repro_long_log2: ceph pg 1.0 query 2024-09-18T16:14:06.742 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:110: TEST_repro_long_log2: jq .info.stats.up_primary 2024-09-18T16:14:06.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:110: TEST_repro_long_log2: local PRIMARY=1 2024-09-18T16:14:06.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:111: TEST_repro_long_log2: kill_daemons td/repro-long-log TERM osd.1 2024-09-18T16:14:06.854 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:14:06.854 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:14:06.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:14:06.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:14:06.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:14:07.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:14:07.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:112: TEST_repro_long_log2: CEPH_ARGS='--osd-max-pg-log-entries=2 --osd-pg-log-dups-tracked=3 --no-mon-config' 2024-09-18T16:14:07.160 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:112: TEST_repro_long_log2: ceph-objectstore-tool --data-path td/repro-long-log/1 --pgid 1.0 --op trim-pg-log 2024-09-18T16:14:08.393 INFO:tasks.workunit.client.0.smithi047.stderr:Log bounds are: (19'20,19'41] 2024-09-18T16:14:08.394 INFO:tasks.workunit.client.0.smithi047.stdout:Removing keys 0000000019.00000000000000000021 - 0000000019.00000000000000000039 2024-09-18T16:14:08.423 INFO:tasks.workunit.client.0.smithi047.stderr:Finished trimming, now compacting... 2024-09-18T16:14:08.423 INFO:tasks.workunit.client.0.smithi047.stdout:Finished trimming pg log 2024-09-18T16:14:09.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:113: TEST_repro_long_log2: activate_osd td/repro-long-log 1 2024-09-18T16:14:09.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/repro-long-log 2024-09-18T16:14:09.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T16:14:09.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T16:14:09.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T16:14:09.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/repro-long-log/1 2024-09-18T16:14:09.663 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-18T16:14:09.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:14:09.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:14:09.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:14:09.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/repro-long-log/1' 2024-09-18T16:14:09.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/repro-long-log/1/journal' 2024-09-18T16:14:09.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T16:14:09.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T16:14:09.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-18T16:14:09.664 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T16:14:09.664 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:14:09.664 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:14:09.664 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:14:09.664 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:14:09.665 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:14:09.665 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.506073/$cluster-$name.asok' 2024-09-18T16:14:09.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' 2024-09-18T16:14:09.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:14:09.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-18T16:14:09.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-18T16:14:09.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:14:09.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:14:09.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:14:09.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:14:09.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T16:14:09.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T16:14:09.665 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/repro-long-log/1 2024-09-18T16:14:09.666 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T16:14:09.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T16:14:09.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:14:09.667 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/repro-long-log/1/whoami 2024-09-18T16:14:09.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T16:14:09.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T16:14:09.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T16:14:09.667 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T16:14:09.688 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:14:09.687+0000 7fbb80275440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:14:09.690 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:14:09.689+0000 7fbb80275440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:14:09.691 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:14:09.690+0000 7fbb80275440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:14:10.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T16:14:10.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:14:10.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T16:14:10.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:14:10.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:14:10.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:10.041 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:14:10.042 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:14:10.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:10.042 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:14:10.211 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:14:10.210+0000 7fbb80275440 -1 Falling back to public interface 2024-09-18T16:14:10.406 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:11.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:11.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:11.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:14:11.408 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T16:14:11.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:11.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:14:11.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:12.493 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:14:12.492+0000 7fbb80275440 -1 osd.1 19 log_to_monitors true 2024-09-18T16:14:12.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:12.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:12.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:14:12.821 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T16:14:12.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:12.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:14:13.292 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 23 up_thru 23 down_at 20 last_clean_interval [10,19) [v2:127.0.0.1:6810/520140,v1:127.0.0.1:6811/520140] [v2:127.0.0.1:6812/520140,v1:127.0.0.1:6813/520140] exists,up 4ed4fe30-a6c1-4c05-b879-04ff2ae92aa2 2024-09-18T16:14:13.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:14:13.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:14:13.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:14:13.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:114: TEST_repro_long_log2: wait_for_clean 2024-09-18T16:14:13.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T16:14:13.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T16:14:13.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T16:14:13.295 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T16:14:13.295 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T16:14:13.295 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T16:14:13.295 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T16:14:13.295 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T16:14:13.295 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T16:14:13.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-18T16:14:13.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T16:14:13.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T16:14:13.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T16:14:13.382 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T16:14:13.382 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T16:14:13.745 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T16:14:13.745 INFO:tasks.workunit.client.0.smithi047.stderr:1 2024-09-18T16:14:13.745 INFO:tasks.workunit.client.0.smithi047.stderr:2' 2024-09-18T16:14:13.746 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T16:14:13.746 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:14:13.746 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T16:14:13.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836507 2024-09-18T16:14:13.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836507 2024-09-18T16:14:13.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836507' 2024-09-18T16:14:13.853 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:14:13.853 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T16:14:13.968 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=98784247811 2024-09-18T16:14:13.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 98784247811 2024-09-18T16:14:13.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836507 1-98784247811' 2024-09-18T16:14:13.969 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T16:14:13.969 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T16:14:14.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509457 2024-09-18T16:14:14.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509457 2024-09-18T16:14:14.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836507 1-98784247811 2-64424509457' 2024-09-18T16:14:14.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:14:14.075 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836507 2024-09-18T16:14:14.075 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:14:14.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T16:14:14.076 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836507 2024-09-18T16:14:14.076 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:14:14.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836507 2024-09-18T16:14:14.077 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.0 seq 21474836507 2024-09-18T16:14:14.077 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-09-18T16:14:14.078 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:14:14.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836505 -lt 21474836507 2024-09-18T16:14:14.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T16:14:15.445 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T16:14:15.446 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T16:14:15.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836507 -lt 21474836507 2024-09-18T16:14:15.821 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:14:15.821 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-98784247811 2024-09-18T16:14:15.821 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:14:15.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T16:14:15.823 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-98784247811 2024-09-18T16:14:15.823 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:14:15.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=98784247811 2024-09-18T16:14:15.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 98784247811' 2024-09-18T16:14:15.824 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.1 seq 98784247811 2024-09-18T16:14:15.824 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T16:14:16.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 98784247811 -lt 98784247811 2024-09-18T16:14:16.193 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T16:14:16.193 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509457 2024-09-18T16:14:16.193 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T16:14:16.194 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T16:14:16.194 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509457 2024-09-18T16:14:16.194 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T16:14:16.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509457 2024-09-18T16:14:16.195 INFO:tasks.workunit.client.0.smithi047.stdout:waiting osd.2 seq 64424509457 2024-09-18T16:14:16.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-09-18T16:14:16.196 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T16:14:16.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509457 -lt 64424509457 2024-09-18T16:14:16.601 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T16:14:16.601 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:14:16.602 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:14:17.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-18T16:14:17.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T16:14:17.107 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T16:14:17.107 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T16:14:17.107 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T16:14:17.107 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T16:14:17.107 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T16:14:17.107 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T16:14:17.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-18T16:14:17.480 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T16:14:17.481 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T16:14:17.481 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T16:14:17.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-18T16:14:17.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T16:14:17.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T16:14:17.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:115: TEST_repro_long_log2: test_log_size 1.0 21 18 2024-09-18T16:14:17.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-18T16:14:17.970 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=21 2024-09-18T16:14:17.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=18 2024-09-18T16:14:17.971 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-18T16:14:18.110 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: 21474836510 2024-09-18T16:14:18.132 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: 98784247814 2024-09-18T16:14:18.154 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: 64424509460 2024-09-18T16:14:18.163 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-18T16:14:21.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-18T16:14:21.165 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-18T16:14:21.288 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T16:14:21.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-18T16:14:21.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-18T16:14:21.289 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 21 2024-09-18T16:14:21.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-18T16:14:21.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:14:21.512 INFO:tasks.workunit.client.0.smithi047.stdout:18 2024-09-18T16:14:21.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-18T16:14:21.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:14:21.513 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 18 2024-09-18T16:14:21.624 INFO:tasks.workunit.client.0.smithi047.stdout:18 2024-09-18T16:14:21.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:35: run: teardown td/repro-long-log 2024-09-18T16:14:21.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-09-18T16:14:21.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:14:21.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-09-18T16:14:21.626 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:14:21.626 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:14:21.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:14:21.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:14:21.626 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:14:21.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:14:21.733 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:14:21.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:14:21.733 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:14:21.734 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:14:21.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:14:21.735 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:14:21.735 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:14:21.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:14:21.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:14:21.736 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:14:21.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:14:21.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:14:21.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-09-18T16:14:21.753 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:14:21.753 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:14:21.753 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:14:21.753 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.506073 2024-09-18T16:14:21.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:14:21.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:14:21.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:32: run: for func in $funcs 2024-09-18T16:14:21.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:33: run: setup td/repro-long-log 2024-09-18T16:14:21.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/repro-long-log 2024-09-18T16:14:21.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/repro-long-log 2024-09-18T16:14:21.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-09-18T16:14:21.755 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:14:21.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-09-18T16:14:21.756 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:14:21.756 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:14:21.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:14:21.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:14:21.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:14:21.756 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:14:21.757 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:14:21.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:14:21.758 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:14:21.758 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:14:21.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:14:21.759 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:14:21.759 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:14:21.760 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:14:21.760 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:14:21.760 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:14:21.761 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:14:21.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:14:21.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-09-18T16:14:21.763 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:14:21.763 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:14:21.763 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:14:21.763 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.506073 2024-09-18T16:14:21.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:14:21.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:14:21.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/repro-long-log 2024-09-18T16:14:21.765 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T16:14:21.765 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:14:21.765 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:14:21.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.506073 2024-09-18T16:14:21.767 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T16:14:21.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T16:14:21.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T16:14:21.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/repro-long-log 1' TERM HUP INT 2024-09-18T16:14:21.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:34: run: TEST_trim_max_entries td/repro-long-log 2024-09-18T16:14:21.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:120: TEST_trim_max_entries: local dir=td/repro-long-log 2024-09-18T16:14:21.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:122: TEST_trim_max_entries: setup_log_test td/repro-long-log 2024-09-18T16:14:21.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:55: setup_log_test: local dir=td/repro-long-log 2024-09-18T16:14:21.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:56: setup_log_test: local which= 2024-09-18T16:14:21.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:58: setup_log_test: run_mon td/repro-long-log a 2024-09-18T16:14:21.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/repro-long-log 2024-09-18T16:14:21.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T16:14:21.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T16:14:21.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T16:14:21.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/repro-long-log/a 2024-09-18T16:14:21.769 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/repro-long-log/a --run-dir=td/repro-long-log 2024-09-18T16:14:22.062 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T16:14:22.063 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:14:22.063 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:14:22.063 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:14:22.063 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:14:22.063 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:14:22.063 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.506073/$cluster-$name.asok' 2024-09-18T16:14:22.063 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/repro-long-log/a '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' --mon-cluster-log-file=td/repro-long-log/log --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T16:14:22.294 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T16:14:22.295 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T16:14:22.295 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:14:22.295 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:14:22.296 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T16:14:22.296 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T16:14:22.296 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:14:22.296 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:14:22.296 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:14:22.297 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:14:22.297 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:14:22.297 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:14:22.297 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.506073/ceph-mon.a.asok 2024-09-18T16:14:22.297 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:14:22.297 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.506073/ceph-mon.a.asok config get fsid 2024-09-18T16:14:22.375 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T16:14:22.375 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:14:22.375 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:14:22.375 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T16:14:22.376 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T16:14:22.376 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:14:22.376 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:14:22.376 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:14:22.376 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:14:22.376 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:14:22.376 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:14:22.376 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.506073/ceph-mon.a.asok 2024-09-18T16:14:22.376 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:14:22.376 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.506073/ceph-mon.a.asok config get mon_host 2024-09-18T16:14:22.450 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:59: setup_log_test: run_mgr td/repro-long-log x 2024-09-18T16:14:22.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/repro-long-log 2024-09-18T16:14:22.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T16:14:22.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T16:14:22.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T16:14:22.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/repro-long-log/x 2024-09-18T16:14:22.451 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T16:14:23.300 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T16:14:23.300 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:14:23.300 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:14:23.300 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:14:23.301 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:14:23.301 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:14:23.301 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.506073/$cluster-$name.asok' 2024-09-18T16:14:23.301 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:14:23.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/repro-long-log/x '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:14:23.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:60: setup_log_test: run_osd td/repro-long-log 0 2024-09-18T16:14:23.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-18T16:14:23.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:14:23.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T16:14:23.327 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:14:23.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/0 2024-09-18T16:14:23.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-18T16:14:23.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:14:23.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:14:23.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:14:23.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/0' 2024-09-18T16:14:23.328 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/0/journal' 2024-09-18T16:14:23.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:14:23.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:14:23.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-18T16:14:23.329 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:14:23.329 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:14:23.329 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:14:23.329 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:14:23.329 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:14:23.329 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:14:23.329 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.506073/$cluster-$name.asok' 2024-09-18T16:14:23.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' 2024-09-18T16:14:23.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:14:23.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:14:23.329 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:14:23.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-18T16:14:23.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-18T16:14:23.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:14:23.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:14:23.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:14:23.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:14:23.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:14:23.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:14:23.330 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/0 2024-09-18T16:14:23.330 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:14:23.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d0c66aad-854f-4e3b-bc78-d4eea726e353 2024-09-18T16:14:23.331 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d0c66aad-854f-4e3b-bc78-d4eea726e353' 2024-09-18T16:14:23.331 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 d0c66aad-854f-4e3b-bc78-d4eea726e353 2024-09-18T16:14:23.331 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:14:23.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBf/Opm95BOFBAACexDbxOxd5qNfNrc9t7JBg== 2024-09-18T16:14:23.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBf/Opm95BOFBAACexDbxOxd5qNfNrc9t7JBg=="}' 2024-09-18T16:14:23.341 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d0c66aad-854f-4e3b-bc78-d4eea726e353 -i td/repro-long-log/0/new.json 2024-09-18T16:14:23.641 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:14:23.650 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/0/new.json 2024-09-18T16:14:23.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBf/Opm95BOFBAACexDbxOxd5qNfNrc9t7JBg== --osd-uuid d0c66aad-854f-4e3b-bc78-d4eea726e353 2024-09-18T16:14:23.674 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:14:23.673+0000 7f0deec3b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:14:23.720 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:14:23.719+0000 7f0deec3b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:14:23.722 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:14:23.720+0000 7f0deec3b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:14:23.722 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:14:23.720+0000 7f0deec3b440 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-18T16:14:23.722 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:14:23.720+0000 7f0deec3b440 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-18T16:14:23.722 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:14:23.720+0000 7f0deec3b440 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-18T16:14:23.760 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:14:23.759+0000 7f0deec3b440 -1 bluestore(td/repro-long-log/0) _read_fsid unparsable uuid 2024-09-18T16:14:28.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/0/keyring 2024-09-18T16:14:28.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:14:28.798 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T16:14:28.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T16:14:28.798 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:14:29.345 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T16:14:29.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T16:14:29.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:14:29.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:14:29.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:14:29.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:14:29.365 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:14:29.364+0000 7ff205d26440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:14:29.368 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:14:29.367+0000 7ff205d26440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:14:29.369 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:14:29.368+0000 7ff205d26440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:14:29.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T16:14:29.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:14:29.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T16:14:29.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:14:29.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:14:29.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:29.711 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:14:29.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:14:29.711 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:29.712 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:14:29.886 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:14:29.885+0000 7ff205d26440 -1 Falling back to public interface 2024-09-18T16:14:30.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:31.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:31.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:31.083 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T16:14:31.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:14:31.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:31.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:14:31.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:32.261 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:14:32.260+0000 7ff205d26440 -1 osd.0 0 log_to_monitors true 2024-09-18T16:14:32.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:32.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:32.500 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T16:14:32.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:14:32.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:32.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:14:32.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:33.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:33.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:33.979 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:14:33.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:14:33.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:33.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:14:34.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:35.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:35.362 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:35.362 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T16:14:35.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:14:35.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:35.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:14:35.741 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:36.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:36.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:36.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:14:36.743 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T16:14:36.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:36.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:14:37.256 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:38.257 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:38.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:38.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:14:38.258 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T16:14:38.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:38.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:14:38.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:39.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:39.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:39.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:14:39.635 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T16:14:39.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:39.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:14:40.067 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:41.068 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:41.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:41.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:14:41.069 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T16:14:41.069 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:41.070 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:14:41.453 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:42.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:42.454 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:42.455 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T16:14:42.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:14:42.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:42.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:14:42.962 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:43.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:43.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:43.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:14:43.964 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T16:14:43.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:43.964 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:14:44.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:45.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:45.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:45.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:14:45.884 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T16:14:45.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:45.884 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:14:46.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:47.420 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:47.420 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:47.420 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:14:47.420 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T16:14:47.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:47.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:14:48.599 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:49.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:49.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:49.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:14:49.601 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T16:14:49.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:49.601 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:14:49.975 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:50.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:50.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:50.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:14:50.977 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T16:14:50.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:50.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:14:51.484 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:52.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:52.485 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:52.485 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T16:14:52.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:14:52.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:52.486 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:14:53.213 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:54.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:54.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:54.214 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:14:54.215 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T16:14:54.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:54.215 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:14:54.707 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:14:55.708 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:14:55.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:14:55.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:14:55.709 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T16:14:55.709 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:14:55.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:14:56.120 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/522256,v1:127.0.0.1:6803/522256] [v2:127.0.0.1:6804/522256,v1:127.0.0.1:6805/522256] exists,up d0c66aad-854f-4e3b-bc78-d4eea726e353 2024-09-18T16:14:56.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:14:56.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:14:56.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:14:56.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:61: setup_log_test: run_osd td/repro-long-log 1 2024-09-18T16:14:56.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-18T16:14:56.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:14:56.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T16:14:56.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:14:56.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/1 2024-09-18T16:14:56.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-18T16:14:56.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:14:56.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:14:56.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:14:56.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/1' 2024-09-18T16:14:56.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/1/journal' 2024-09-18T16:14:56.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:14:56.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:14:56.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-18T16:14:56.126 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:14:56.126 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:14:56.126 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:14:56.126 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:14:56.126 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:14:56.126 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:14:56.126 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.506073/$cluster-$name.asok' 2024-09-18T16:14:56.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' 2024-09-18T16:14:56.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:14:56.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:14:56.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:14:56.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-18T16:14:56.126 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-18T16:14:56.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:14:56.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:14:56.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:14:56.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:14:56.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:14:56.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:14:56.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/1 2024-09-18T16:14:56.127 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:14:56.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=31ca937e-8977-409c-a428-2260296d0e8d 2024-09-18T16:14:56.127 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 31ca937e-8977-409c-a428-2260296d0e8d 2024-09-18T16:14:56.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 31ca937e-8977-409c-a428-2260296d0e8d' 2024-09-18T16:14:56.128 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:14:56.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCA/Opmj60gCBAA/8SMnE1Y8tSwsqsaZTXaqw== 2024-09-18T16:14:56.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCA/Opmj60gCBAA/8SMnE1Y8tSwsqsaZTXaqw=="}' 2024-09-18T16:14:56.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 31ca937e-8977-409c-a428-2260296d0e8d -i td/repro-long-log/1/new.json 2024-09-18T16:14:56.628 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T16:14:56.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/1/new.json 2024-09-18T16:14:56.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCA/Opmj60gCBAA/8SMnE1Y8tSwsqsaZTXaqw== --osd-uuid 31ca937e-8977-409c-a428-2260296d0e8d 2024-09-18T16:14:56.663 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:14:56.662+0000 7f370a015440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:14:56.665 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:14:56.664+0000 7f370a015440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:14:56.666 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:14:56.665+0000 7f370a015440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:14:56.666 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:14:56.665+0000 7f370a015440 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-18T16:14:56.666 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:14:56.665+0000 7f370a015440 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-18T16:14:56.666 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:14:56.665+0000 7f370a015440 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-18T16:14:56.710 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:14:56.709+0000 7f370a015440 -1 bluestore(td/repro-long-log/1) _read_fsid unparsable uuid 2024-09-18T16:15:02.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/1/keyring 2024-09-18T16:15:02.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:15:02.516 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T16:15:02.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T16:15:02.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:15:03.024 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T16:15:03.024 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T16:15:03.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:15:03.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:15:03.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:15:03.025 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:15:03.045 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:15:03.044+0000 7f30bc72d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:15:03.047 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:15:03.046+0000 7f30bc72d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:15:03.048 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:15:03.047+0000 7f30bc72d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:15:03.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T16:15:03.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:15:03.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T16:15:03.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:15:03.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:15:03.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:03.397 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:15:03.398 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:15:03.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:03.398 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:15:03.562 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:15:03.561+0000 7f30bc72d440 -1 Falling back to public interface 2024-09-18T16:15:03.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:04.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:04.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:04.791 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T16:15:04.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:15:04.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:04.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:15:05.207 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:06.021 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:15:06.020+0000 7f30bc72d440 -1 osd.1 0 log_to_monitors true 2024-09-18T16:15:06.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:06.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:06.208 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:15:06.209 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T16:15:06.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:06.209 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:15:06.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:07.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:07.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:07.683 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:15:07.684 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:15:07.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:07.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:15:08.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:09.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:09.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:09.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:15:09.104 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T16:15:09.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:09.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:15:09.479 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:10.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:10.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:10.480 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:15:10.481 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T16:15:10.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:10.481 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:15:10.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:11.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:11.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:11.882 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:15:11.883 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T16:15:11.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:11.883 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:15:12.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:13.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:13.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:13.348 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:15:13.348 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T16:15:13.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:13.349 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:15:13.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:14.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:14.751 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:14.752 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T16:15:14.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:15:14.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:14.752 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:15:15.128 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:16.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:16.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:16.130 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:15:16.130 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T16:15:16.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:16.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:15:16.507 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:17.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:17.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:17.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:15:17.509 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T16:15:17.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:17.509 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:15:18.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:19.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:19.320 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:19.320 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T16:15:19.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:15:19.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:19.321 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:15:19.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:20.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:20.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:20.793 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T16:15:20.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:15:20.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:20.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:15:21.473 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:22.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:22.474 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:22.475 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T16:15:22.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:15:22.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:22.475 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:15:23.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:24.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:24.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:24.086 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T16:15:24.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:15:24.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:24.087 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:15:24.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:25.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:25.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:25.764 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:15:25.764 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T16:15:25.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:25.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:15:26.544 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:27.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:27.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:27.546 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T16:15:27.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:15:27.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:27.547 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:15:28.080 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:29.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:29.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:29.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:15:29.082 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T16:15:29.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:29.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:15:29.492 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/524010,v1:127.0.0.1:6811/524010] [v2:127.0.0.1:6812/524010,v1:127.0.0.1:6813/524010] exists,up 31ca937e-8977-409c-a428-2260296d0e8d 2024-09-18T16:15:29.495 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:15:29.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:15:29.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:15:29.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:62: setup_log_test: run_osd td/repro-long-log 2 2024-09-18T16:15:29.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-18T16:15:29.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:15:29.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T16:15:29.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:15:29.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/2 2024-09-18T16:15:29.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-18T16:15:29.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:15:29.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:15:29.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:15:29.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/2' 2024-09-18T16:15:29.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/2/journal' 2024-09-18T16:15:29.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:15:29.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:15:29.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-18T16:15:29.497 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:15:29.497 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:15:29.497 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:15:29.497 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:15:29.497 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:15:29.497 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:15:29.497 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.506073/$cluster-$name.asok' 2024-09-18T16:15:29.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' 2024-09-18T16:15:29.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:15:29.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:15:29.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:15:29.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-18T16:15:29.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-18T16:15:29.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:15:29.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:15:29.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:15:29.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:15:29.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:15:29.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:15:29.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/2 2024-09-18T16:15:29.498 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:15:29.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4f29ef03-b954-4169-9be1-b0a44228c4ba 2024-09-18T16:15:29.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 4f29ef03-b954-4169-9be1-b0a44228c4ba' 2024-09-18T16:15:29.499 INFO:tasks.workunit.client.0.smithi047.stdout:add osd2 4f29ef03-b954-4169-9be1-b0a44228c4ba 2024-09-18T16:15:29.499 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:15:29.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCh/OpmZgVEHhAAuBQ3KRLWzGiQye8Z5fhRZw== 2024-09-18T16:15:29.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCh/OpmZgVEHhAAuBQ3KRLWzGiQye8Z5fhRZw=="}' 2024-09-18T16:15:29.508 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4f29ef03-b954-4169-9be1-b0a44228c4ba -i td/repro-long-log/2/new.json 2024-09-18T16:15:29.983 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T16:15:29.993 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/2/new.json 2024-09-18T16:15:29.994 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCh/OpmZgVEHhAAuBQ3KRLWzGiQye8Z5fhRZw== --osd-uuid 4f29ef03-b954-4169-9be1-b0a44228c4ba 2024-09-18T16:15:30.017 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:15:30.016+0000 7f335e00e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:15:30.019 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:15:30.018+0000 7f335e00e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:15:30.020 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:15:30.019+0000 7f335e00e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:15:30.020 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:15:30.019+0000 7f335e00e440 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-18T16:15:30.021 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:15:30.019+0000 7f335e00e440 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-18T16:15:30.021 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:15:30.019+0000 7f335e00e440 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-18T16:15:30.056 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:15:30.055+0000 7f335e00e440 -1 bluestore(td/repro-long-log/2) _read_fsid unparsable uuid 2024-09-18T16:15:35.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/2/keyring 2024-09-18T16:15:35.608 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:15:35.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T16:15:35.609 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd2 key to auth repository 2024-09-18T16:15:35.609 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:15:36.117 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T16:15:36.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T16:15:36.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:15:36.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:15:36.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:15:36.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:15:36.138 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:15:36.137+0000 7f1750d2f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:15:36.141 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:15:36.140+0000 7f1750d2f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:15:36.142 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:15:36.141+0000 7f1750d2f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:15:36.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T16:15:36.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:15:36.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T16:15:36.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:15:36.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:15:36.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:36.511 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:15:36.511 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:15:36.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:36.512 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:15:36.657 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:15:36.655+0000 7f1750d2f440 -1 Falling back to public interface 2024-09-18T16:15:36.936 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:37.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:37.937 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:37.938 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T16:15:37.938 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:15:37.938 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:37.938 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:15:38.355 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:39.136 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:15:39.134+0000 7f1750d2f440 -1 osd.2 0 log_to_monitors true 2024-09-18T16:15:39.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:39.356 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:39.356 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T16:15:39.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:15:39.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:39.357 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:15:39.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:40.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:40.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:40.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:15:40.789 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:15:40.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:40.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:15:41.225 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:42.226 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:42.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:42.227 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T16:15:42.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:15:42.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:42.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:15:42.639 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:43.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:43.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:43.641 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T16:15:43.642 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:15:43.642 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:43.642 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:15:44.172 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:45.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:45.173 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:45.174 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T16:15:45.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:15:45.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:45.174 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:15:45.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:46.648 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:46.648 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:46.648 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T16:15:46.648 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:15:46.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:46.649 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:15:47.201 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:48.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:48.203 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:48.203 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T16:15:48.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:15:48.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:48.204 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:15:48.577 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:49.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:49.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:49.579 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T16:15:49.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:15:49.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:49.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:15:50.036 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:51.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:51.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:51.038 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T16:15:51.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:15:51.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:51.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:15:51.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:52.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:52.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:52.517 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T16:15:52.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:15:52.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:52.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:15:53.280 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:54.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:54.282 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:54.282 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T16:15:54.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:15:54.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:54.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:15:54.996 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:55.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:55.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:55.998 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T16:15:55.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:15:55.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:55.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:15:56.781 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:57.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:57.783 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:57.783 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T16:15:57.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:15:57.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:57.784 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:15:58.407 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:15:59.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:15:59.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:15:59.408 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:15:59.409 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T16:15:59.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:15:59.409 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:16:00.092 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:16:01.093 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:16:01.093 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T16:16:01.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:16:01.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:16:01.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:16:01.094 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:16:01.527 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:16:02.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:16:02.529 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:16:02.529 INFO:tasks.workunit.client.0.smithi047.stdout:17 2024-09-18T16:16:02.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T16:16:02.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:16:02.530 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:16:02.932 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/525764,v1:127.0.0.1:6819/525764] [v2:127.0.0.1:6820/525764,v1:127.0.0.1:6821/525764] exists,up 4f29ef03-b954-4169-9be1-b0a44228c4ba 2024-09-18T16:16:02.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:16:02.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:16:02.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:16:02.933 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:64: setup_log_test: ceph osd pool create test 1 1 2024-09-18T16:16:03.457 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test' already exists 2024-09-18T16:16:03.469 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: ceph osd dump --format json 2024-09-18T16:16:03.469 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: jq '.pools[] | select(.pool_name == "test") | .pool' 2024-09-18T16:16:03.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: POOL_ID=1 2024-09-18T16:16:03.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:66: setup_log_test: PGID=1.0 2024-09-18T16:16:03.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:69: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 20 2024-09-18T16:16:04.006 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: {} 2024-09-18T16:16:04.032 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: {} 2024-09-18T16:16:04.058 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: {} 2024-09-18T16:16:04.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:70: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 20 2024-09-18T16:16:04.167 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: {} 2024-09-18T16:16:04.193 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: {} 2024-09-18T16:16:04.219 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: {} 2024-09-18T16:16:04.227 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:71: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-max-pg-log-entries 30 2024-09-18T16:16:04.341 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: {} 2024-09-18T16:16:04.368 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: {} 2024-09-18T16:16:04.393 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: {} 2024-09-18T16:16:04.402 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:72: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 10 2024-09-18T16:16:04.508 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: {} 2024-09-18T16:16:04.535 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: {} 2024-09-18T16:16:04.560 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: {} 2024-09-18T16:16:04.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:73: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 20 2024-09-18T16:16:04.726 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: {} 2024-09-18T16:16:04.753 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: {} 2024-09-18T16:16:04.778 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: {} 2024-09-18T16:16:04.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:75: setup_log_test: touch td/repro-long-log/foo 2024-09-18T16:16:04.788 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: seq 1 20 2024-09-18T16:16:04.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:04.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:16:05.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:05.026 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:16:05.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:05.162 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:16:05.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:05.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:16:05.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:05.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:16:05.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:05.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:16:05.632 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:05.633 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:16:05.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:05.733 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:16:05.833 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:05.834 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:16:05.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:05.934 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:16:06.057 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:06.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:16:06.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:06.223 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:16:06.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:06.277 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:16:06.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:06.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:16:06.501 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:06.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:16:06.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:06.613 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:16:06.704 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:06.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:16:06.796 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:06.797 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:16:06.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:06.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:16:07.032 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:07.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:16:07.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:81: setup_log_test: test_log_size 1.0 20 2024-09-18T16:16:07.150 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-18T16:16:07.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=20 2024-09-18T16:16:07.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-18T16:16:07.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-18T16:16:07.247 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: 21474836501 2024-09-18T16:16:07.270 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: 42949672976 2024-09-18T16:16:07.291 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: 64424509448 2024-09-18T16:16:07.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-18T16:16:10.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:10.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-18T16:16:10.425 INFO:tasks.workunit.client.0.smithi047.stdout:20 2024-09-18T16:16:10.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:10.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-18T16:16:10.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 20 2024-09-18T16:16:10.540 INFO:tasks.workunit.client.0.smithi047.stdout:20 2024-09-18T16:16:10.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:10.540 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:16:10.650 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:16:10.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:10.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:16:10.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-18T16:16:10.761 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:16:10.761 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:83: setup_log_test: rados -p test rm foo 2024-09-18T16:16:10.869 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: seq 1 20 2024-09-18T16:16:10.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:10.870 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:16:10.984 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:16:10.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:10.986 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:16:11.084 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:16:11.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:11.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:16:11.177 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:16:11.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:11.179 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:16:11.269 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:16:11.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:11.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:16:11.361 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:16:11.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:11.363 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:16:11.470 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:16:11.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:11.472 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:16:11.638 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:16:11.640 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:11.641 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:16:11.730 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:16:11.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:11.732 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:16:11.839 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:16:11.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:11.841 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:16:11.939 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:16:11.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:11.941 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:16:12.032 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:16:12.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:12.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:16:12.140 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:16:12.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:12.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:16:12.258 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:16:12.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:12.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:16:12.376 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:16:12.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:12.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:16:12.475 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:16:12.477 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:12.477 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:16:12.576 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:16:12.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:12.578 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:16:12.668 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:16:12.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:12.670 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:16:12.760 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:16:12.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:12.762 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:16:12.861 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:16:12.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:16:12.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:16:12.995 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:16:12.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:92: setup_log_test: test_log_size 1.0 21 2024-09-18T16:16:12.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-18T16:16:12.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=21 2024-09-18T16:16:12.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-18T16:16:12.998 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-18T16:16:13.095 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: 21474836504 2024-09-18T16:16:13.117 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: 42949672979 2024-09-18T16:16:13.139 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: 64424509452 2024-09-18T16:16:13.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-18T16:16:16.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:16.149 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-18T16:16:16.272 INFO:tasks.workunit.client.0.smithi047.stdout:21 2024-09-18T16:16:16.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:16.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-18T16:16:16.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 21 2024-09-18T16:16:16.385 INFO:tasks.workunit.client.0.smithi047.stdout:21 2024-09-18T16:16:16.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:16.385 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:16:16.496 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:16:16.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:16.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:16:16.496 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-18T16:16:16.606 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:16:16.607 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:124: TEST_trim_max_entries: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 2 2024-09-18T16:16:16.707 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: {} 2024-09-18T16:16:16.733 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: {} 2024-09-18T16:16:16.758 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: {} 2024-09-18T16:16:16.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:125: TEST_trim_max_entries: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 2 2024-09-18T16:16:16.869 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: {} 2024-09-18T16:16:16.895 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: {} 2024-09-18T16:16:16.921 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: {} 2024-09-18T16:16:16.929 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:126: TEST_trim_max_entries: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 2 2024-09-18T16:16:17.030 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: {} 2024-09-18T16:16:17.056 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: {} 2024-09-18T16:16:17.081 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: {} 2024-09-18T16:16:17.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:127: TEST_trim_max_entries: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-max 4 2024-09-18T16:16:17.195 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: {} 2024-09-18T16:16:17.220 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: {} 2024-09-18T16:16:17.246 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: {} 2024-09-18T16:16:17.254 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:128: TEST_trim_max_entries: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 0 2024-09-18T16:16:17.354 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: {} 2024-09-18T16:16:17.380 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: {} 2024-09-18T16:16:17.404 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: {} 2024-09-18T16:16:17.413 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:131: TEST_trim_max_entries: rados -p test rm foo 2024-09-18T16:16:17.517 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:16:17.519 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:132: TEST_trim_max_entries: test_log_size 1.0 18 2024-09-18T16:16:17.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-18T16:16:17.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=18 2024-09-18T16:16:17.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-18T16:16:17.520 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-18T16:16:17.617 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: 21474836507 2024-09-18T16:16:17.639 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: 42949672982 2024-09-18T16:16:17.661 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: 64424509455 2024-09-18T16:16:17.669 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-18T16:16:20.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:20.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-18T16:16:20.794 INFO:tasks.workunit.client.0.smithi047.stdout:18 2024-09-18T16:16:20.794 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:20.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-18T16:16:20.795 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 18 2024-09-18T16:16:20.906 INFO:tasks.workunit.client.0.smithi047.stdout:18 2024-09-18T16:16:20.907 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:20.907 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:16:21.017 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:16:21.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:21.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:16:21.018 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-18T16:16:21.129 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:16:21.129 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:133: TEST_trim_max_entries: rados -p test rm foo 2024-09-18T16:16:21.228 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:16:21.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:134: TEST_trim_max_entries: test_log_size 1.0 15 2024-09-18T16:16:21.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-18T16:16:21.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=15 2024-09-18T16:16:21.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-18T16:16:21.230 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-18T16:16:21.328 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: 21474836510 2024-09-18T16:16:21.350 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: 42949672985 2024-09-18T16:16:21.372 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: 64424509457 2024-09-18T16:16:21.381 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-18T16:16:24.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:24.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-18T16:16:24.505 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T16:16:24.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:24.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-18T16:16:24.506 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 15 2024-09-18T16:16:24.617 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T16:16:24.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:24.618 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:16:24.729 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:16:24.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:24.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:16:24.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-18T16:16:24.840 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:16:24.840 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:135: TEST_trim_max_entries: rados -p test rm foo 2024-09-18T16:16:24.963 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:16:24.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:136: TEST_trim_max_entries: test_log_size 1.0 12 2024-09-18T16:16:24.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-18T16:16:24.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=12 2024-09-18T16:16:24.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-18T16:16:24.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-18T16:16:25.080 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: 21474836512 2024-09-18T16:16:25.103 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: 42949672988 2024-09-18T16:16:25.124 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: 64424509460 2024-09-18T16:16:25.133 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-18T16:16:28.135 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:28.135 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-18T16:16:28.258 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T16:16:28.258 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:28.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-18T16:16:28.259 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 12 2024-09-18T16:16:28.370 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T16:16:28.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:28.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:16:28.481 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:16:28.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:28.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:16:28.482 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-18T16:16:28.592 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:16:28.593 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:137: TEST_trim_max_entries: rados -p test rm foo 2024-09-18T16:16:28.699 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:16:28.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:138: TEST_trim_max_entries: test_log_size 1.0 9 2024-09-18T16:16:28.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-18T16:16:28.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=9 2024-09-18T16:16:28.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-18T16:16:28.701 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-18T16:16:28.799 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: 21474836515 2024-09-18T16:16:28.821 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: 42949672991 2024-09-18T16:16:28.843 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: 64424509463 2024-09-18T16:16:28.851 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-18T16:16:31.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:31.854 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-18T16:16:31.978 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T16:16:31.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:31.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-18T16:16:31.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 9 2024-09-18T16:16:32.089 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T16:16:32.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:32.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:16:32.199 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:16:32.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:32.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:16:32.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-18T16:16:32.310 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:16:32.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:139: TEST_trim_max_entries: rados -p test rm foo 2024-09-18T16:16:32.426 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:16:32.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:140: TEST_trim_max_entries: test_log_size 1.0 6 2024-09-18T16:16:32.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-18T16:16:32.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=6 2024-09-18T16:16:32.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-18T16:16:32.429 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-18T16:16:32.526 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: 21474836518 2024-09-18T16:16:32.549 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: 42949672993 2024-09-18T16:16:32.571 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: 64424509466 2024-09-18T16:16:32.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-18T16:16:35.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:35.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-18T16:16:35.704 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T16:16:35.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:35.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-18T16:16:35.705 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 6 2024-09-18T16:16:35.816 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T16:16:35.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:35.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:16:35.927 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:16:35.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:35.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:16:35.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-18T16:16:36.038 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:16:36.038 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:141: TEST_trim_max_entries: rados -p test rm foo 2024-09-18T16:16:36.146 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:16:36.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:142: TEST_trim_max_entries: test_log_size 1.0 3 2024-09-18T16:16:36.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-18T16:16:36.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-09-18T16:16:36.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-18T16:16:36.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-18T16:16:36.245 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: 21474836521 2024-09-18T16:16:36.267 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: 42949672996 2024-09-18T16:16:36.289 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: 64424509468 2024-09-18T16:16:36.297 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-18T16:16:39.299 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:39.299 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-18T16:16:39.422 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:16:39.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:39.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-18T16:16:39.423 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-09-18T16:16:39.535 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:16:39.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:39.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:16:39.645 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:16:39.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:39.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:16:39.646 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-18T16:16:39.757 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:16:39.757 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:145: TEST_trim_max_entries: rados -p test rm foo 2024-09-18T16:16:39.856 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:16:39.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:146: TEST_trim_max_entries: test_log_size 1.0 4 2024-09-18T16:16:39.858 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-18T16:16:39.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=4 2024-09-18T16:16:39.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-18T16:16:39.859 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-18T16:16:39.956 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: 21474836523 2024-09-18T16:16:39.979 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: 42949672999 2024-09-18T16:16:40.001 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: 64424509471 2024-09-18T16:16:40.010 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-18T16:16:43.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:43.012 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-18T16:16:43.135 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T16:16:43.135 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:43.135 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-18T16:16:43.135 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 4 2024-09-18T16:16:43.246 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T16:16:43.247 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:43.247 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:16:43.357 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:16:43.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:43.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:16:43.358 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-18T16:16:43.468 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:16:43.468 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:147: TEST_trim_max_entries: rados -p test rm foo 2024-09-18T16:16:43.567 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:16:43.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:148: TEST_trim_max_entries: test_log_size 1.0 3 2024-09-18T16:16:43.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-18T16:16:43.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-09-18T16:16:43.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-18T16:16:43.570 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-18T16:16:43.667 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: 21474836526 2024-09-18T16:16:43.689 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: 42949673002 2024-09-18T16:16:43.711 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: 64424509474 2024-09-18T16:16:43.720 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-18T16:16:46.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:46.722 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-18T16:16:46.846 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:16:46.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:46.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-18T16:16:46.847 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-09-18T16:16:46.958 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:16:46.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:46.959 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:16:47.099 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:16:47.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:47.099 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:16:47.100 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-18T16:16:47.210 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:16:47.210 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:149: TEST_trim_max_entries: rados -p test rm foo 2024-09-18T16:16:47.336 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:16:47.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:150: TEST_trim_max_entries: test_log_size 1.0 4 2024-09-18T16:16:47.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-18T16:16:47.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=4 2024-09-18T16:16:47.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-18T16:16:47.338 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-18T16:16:47.445 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: 21474836529 2024-09-18T16:16:47.468 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: 42949673004 2024-09-18T16:16:47.491 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: 64424509476 2024-09-18T16:16:47.500 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-18T16:16:50.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:50.502 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-18T16:16:50.624 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T16:16:50.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:50.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-18T16:16:50.625 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 4 2024-09-18T16:16:50.737 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T16:16:50.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:50.738 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:16:50.848 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:16:50.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:50.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:16:50.848 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-18T16:16:50.959 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:16:50.960 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:151: TEST_trim_max_entries: rados -p test rm foo 2024-09-18T16:16:51.063 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:16:51.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:152: TEST_trim_max_entries: test_log_size 1.0 3 2024-09-18T16:16:51.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-18T16:16:51.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-09-18T16:16:51.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-18T16:16:51.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-18T16:16:51.164 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: 21474836532 2024-09-18T16:16:51.187 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: 42949673007 2024-09-18T16:16:51.208 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: 64424509479 2024-09-18T16:16:51.217 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-18T16:16:54.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:54.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-18T16:16:54.342 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:16:54.343 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:54.343 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-18T16:16:54.343 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-09-18T16:16:54.454 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:16:54.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:54.455 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:16:54.565 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:16:54.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-18T16:16:54.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:16:54.566 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-18T16:16:54.676 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:16:54.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:35: run: teardown td/repro-long-log 2024-09-18T16:16:54.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-09-18T16:16:54.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:16:54.677 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-09-18T16:16:54.677 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:16:54.678 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:16:54.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:16:54.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:16:54.678 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:16:54.785 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:16:54.785 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:16:54.786 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:16:54.786 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:16:54.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:16:54.787 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:16:54.787 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:16:54.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:16:54.788 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:16:54.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:16:54.789 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:16:54.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:16:54.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:16:54.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-09-18T16:16:54.806 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:16:54.806 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:16:54.806 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:16:54.807 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.506073 2024-09-18T16:16:54.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:16:54.808 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:16:54.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:32: run: for func in $funcs 2024-09-18T16:16:54.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:33: run: setup td/repro-long-log 2024-09-18T16:16:54.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/repro-long-log 2024-09-18T16:16:54.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/repro-long-log 2024-09-18T16:16:54.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-09-18T16:16:54.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:16:54.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-09-18T16:16:54.809 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:16:54.809 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:16:54.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:16:54.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:16:54.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:16:54.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:16:54.810 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:16:54.810 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:16:54.811 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:16:54.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:16:54.812 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:16:54.812 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:16:54.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:16:54.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:16:54.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:16:54.814 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:16:54.814 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:16:54.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:16:54.815 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-09-18T16:16:54.817 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:16:54.817 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:16:54.817 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:16:54.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.506073 2024-09-18T16:16:54.817 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:16:54.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:16:54.818 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/repro-long-log 2024-09-18T16:16:54.819 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T16:16:54.819 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:16:54.819 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:16:54.819 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.506073 2024-09-18T16:16:54.821 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T16:16:54.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T16:16:54.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T16:16:54.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/repro-long-log 1' TERM HUP INT 2024-09-18T16:16:54.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:34: run: TEST_trim_max_entries_with_dups td/repro-long-log 2024-09-18T16:16:54.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:157: TEST_trim_max_entries_with_dups: local dir=td/repro-long-log 2024-09-18T16:16:54.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:159: TEST_trim_max_entries_with_dups: setup_log_test td/repro-long-log 2024-09-18T16:16:54.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:55: setup_log_test: local dir=td/repro-long-log 2024-09-18T16:16:54.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:56: setup_log_test: local which= 2024-09-18T16:16:54.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:58: setup_log_test: run_mon td/repro-long-log a 2024-09-18T16:16:54.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/repro-long-log 2024-09-18T16:16:54.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T16:16:54.822 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T16:16:54.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T16:16:54.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/repro-long-log/a 2024-09-18T16:16:54.823 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/repro-long-log/a --run-dir=td/repro-long-log 2024-09-18T16:16:55.157 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T16:16:55.157 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:16:55.157 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:16:55.157 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:16:55.157 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:16:55.157 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:16:55.157 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.506073/$cluster-$name.asok' 2024-09-18T16:16:55.157 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/repro-long-log/a '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' --mon-cluster-log-file=td/repro-long-log/log --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T16:16:55.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T16:16:55.400 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T16:16:55.400 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:16:55.400 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:16:55.401 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T16:16:55.401 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T16:16:55.401 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:16:55.401 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:16:55.401 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:16:55.402 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:16:55.402 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:16:55.402 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:16:55.402 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.506073/ceph-mon.a.asok 2024-09-18T16:16:55.402 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:16:55.402 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.506073/ceph-mon.a.asok config get fsid 2024-09-18T16:16:55.480 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T16:16:55.480 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T16:16:55.480 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T16:16:55.480 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T16:16:55.480 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T16:16:55.480 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T16:16:55.480 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T16:16:55.480 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T16:16:55.480 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T16:16:55.480 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:16:55.481 INFO:tasks.workunit.client.0.smithi047.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:16:55.481 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.506073/ceph-mon.a.asok 2024-09-18T16:16:55.481 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T16:16:55.481 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.506073/ceph-mon.a.asok config get mon_host 2024-09-18T16:16:55.555 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:59: setup_log_test: run_mgr td/repro-long-log x 2024-09-18T16:16:55.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/repro-long-log 2024-09-18T16:16:55.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T16:16:55.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T16:16:55.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T16:16:55.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/repro-long-log/x 2024-09-18T16:16:55.556 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T16:16:56.398 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T16:16:56.398 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:16:56.398 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:16:56.398 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:16:56.398 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:16:56.398 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:16:56.398 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.506073/$cluster-$name.asok' 2024-09-18T16:16:56.398 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:16:56.399 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/repro-long-log/x '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T16:16:56.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:60: setup_log_test: run_osd td/repro-long-log 0 2024-09-18T16:16:56.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-18T16:16:56.425 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:16:56.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T16:16:56.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:16:56.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/0 2024-09-18T16:16:56.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-18T16:16:56.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:16:56.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:16:56.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:16:56.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/0' 2024-09-18T16:16:56.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/0/journal' 2024-09-18T16:16:56.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:16:56.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:16:56.426 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-18T16:16:56.426 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:16:56.427 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:16:56.427 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:16:56.427 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:16:56.427 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:16:56.427 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:16:56.427 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.506073/$cluster-$name.asok' 2024-09-18T16:16:56.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' 2024-09-18T16:16:56.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:16:56.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:16:56.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:16:56.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-18T16:16:56.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-18T16:16:56.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:16:56.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:16:56.427 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:16:56.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:16:56.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:16:56.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:16:56.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/0 2024-09-18T16:16:56.428 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:16:56.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4365f80b-2353-4b4a-a29c-8cf0512c36ce 2024-09-18T16:16:56.428 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4365f80b-2353-4b4a-a29c-8cf0512c36ce' 2024-09-18T16:16:56.429 INFO:tasks.workunit.client.0.smithi047.stdout:add osd0 4365f80b-2353-4b4a-a29c-8cf0512c36ce 2024-09-18T16:16:56.429 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:16:56.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD4/OpmvZ8hGhAA4rMGG0FFwsyKRKhkYm6kGA== 2024-09-18T16:16:56.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD4/OpmvZ8hGhAA4rMGG0FFwsyKRKhkYm6kGA=="}' 2024-09-18T16:16:56.439 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4365f80b-2353-4b4a-a29c-8cf0512c36ce -i td/repro-long-log/0/new.json 2024-09-18T16:16:56.739 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:16:56.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/0/new.json 2024-09-18T16:16:56.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD4/OpmvZ8hGhAA4rMGG0FFwsyKRKhkYm6kGA== --osd-uuid 4365f80b-2353-4b4a-a29c-8cf0512c36ce 2024-09-18T16:16:56.772 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:16:56.771+0000 7fed3731f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:16:56.818 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:16:56.817+0000 7fed3731f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:16:56.819 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:16:56.818+0000 7fed3731f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:16:56.819 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:16:56.818+0000 7fed3731f440 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-18T16:16:56.820 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:16:56.818+0000 7fed3731f440 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-18T16:16:56.820 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:16:56.818+0000 7fed3731f440 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-18T16:16:56.866 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:16:56.865+0000 7fed3731f440 -1 bluestore(td/repro-long-log/0) _read_fsid unparsable uuid 2024-09-18T16:17:02.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/0/keyring 2024-09-18T16:17:02.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:17:02.050 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd0 key to auth repository 2024-09-18T16:17:02.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T16:17:02.051 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:17:02.568 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.0 2024-09-18T16:17:02.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T16:17:02.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:17:02.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:17:02.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:17:02.569 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:17:02.589 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:17:02.588+0000 7fec52016440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:17:02.592 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:17:02.591+0000 7fec52016440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:17:02.593 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:17:02.592+0000 7fec52016440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:17:02.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T16:17:02.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:17:02.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T16:17:02.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:17:02.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:17:02.931 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:17:02.931 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:17:02.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:17:02.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:17:02.932 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:17:03.116 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:17:03.115+0000 7fec52016440 -1 Falling back to public interface 2024-09-18T16:17:03.337 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:17:04.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:17:04.339 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:17:04.339 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T16:17:04.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:17:04.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:17:04.340 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:17:04.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:17:05.589 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:17:05.588+0000 7fec52016440 -1 osd.0 0 log_to_monitors true 2024-09-18T16:17:05.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:17:05.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:17:05.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:17:05.767 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T16:17:05.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:17:05.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:17:06.194 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:17:07.195 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:17:07.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:17:07.196 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:17:07.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:17:07.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:17:07.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:17:07.582 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:17:08.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:17:08.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:17:08.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:17:08.584 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T16:17:08.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:17:08.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:17:08.977 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:17:09.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:17:09.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:17:09.979 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:17:09.979 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T16:17:09.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:17:09.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:17:10.393 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:17:11.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:17:11.394 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:17:11.395 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T16:17:11.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:17:11.395 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:17:11.396 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:17:11.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:17:12.791 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:17:12.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:17:12.792 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T16:17:12.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:17:12.792 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:17:12.793 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:17:13.265 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:17:14.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:17:14.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:17:14.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:17:14.267 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T16:17:14.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:17:14.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:17:14.645 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:17:15.657 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:17:15.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:17:15.914 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T16:17:15.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:17:15.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:17:15.915 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:17:16.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:17:17.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:17:17.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:17:17.059 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:17:17.060 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T16:17:17.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:17:17.060 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:17:17.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:17:18.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:17:18.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:17:18.927 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T16:17:18.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:17:18.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:17:18.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:17:19.536 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:17:20.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:17:20.538 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:17:20.538 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T16:17:20.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:17:20.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:17:20.539 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:17:21.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:17:22.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:17:22.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:17:22.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:17:22.562 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T16:17:22.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:17:22.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:17:23.323 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:17:24.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:17:24.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:17:24.325 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T16:17:24.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:17:24.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:17:24.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:17:25.299 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:17:26.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:17:26.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:17:26.300 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:17:26.301 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T16:17:26.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:17:26.301 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:17:26.714 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:17:27.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:17:27.715 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:17:27.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:17:27.716 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T16:17:27.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:17:27.716 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T16:17:28.114 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/530717,v1:127.0.0.1:6803/530717] [v2:127.0.0.1:6804/530717,v1:127.0.0.1:6805/530717] exists,up 4365f80b-2353-4b4a-a29c-8cf0512c36ce 2024-09-18T16:17:28.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:17:28.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:17:28.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:17:28.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:61: setup_log_test: run_osd td/repro-long-log 1 2024-09-18T16:17:28.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-18T16:17:28.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:17:28.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T16:17:28.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:17:28.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/1 2024-09-18T16:17:28.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-18T16:17:28.119 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:17:28.119 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:17:28.119 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:17:28.119 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/1' 2024-09-18T16:17:28.119 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/1/journal' 2024-09-18T16:17:28.119 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:17:28.119 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:17:28.119 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-18T16:17:28.119 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:17:28.119 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:17:28.119 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:17:28.119 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:17:28.119 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:17:28.119 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:17:28.120 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.506073/$cluster-$name.asok' 2024-09-18T16:17:28.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' 2024-09-18T16:17:28.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:17:28.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:17:28.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:17:28.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-18T16:17:28.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-18T16:17:28.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:17:28.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:17:28.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:17:28.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:17:28.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:17:28.120 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:17:28.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/1 2024-09-18T16:17:28.121 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:17:28.121 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=38eaf2fd-9586-44b7-b535-b0aaa388500c 2024-09-18T16:17:28.121 INFO:tasks.workunit.client.0.smithi047.stdout:add osd1 38eaf2fd-9586-44b7-b535-b0aaa388500c 2024-09-18T16:17:28.122 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 38eaf2fd-9586-44b7-b535-b0aaa388500c' 2024-09-18T16:17:28.122 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:17:28.131 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAY/epm8cLMBxAA9SXPf5a2MJZjNr2l59QMsA== 2024-09-18T16:17:28.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAY/epm8cLMBxAA9SXPf5a2MJZjNr2l59QMsA=="}' 2024-09-18T16:17:28.132 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 38eaf2fd-9586-44b7-b535-b0aaa388500c -i td/repro-long-log/1/new.json 2024-09-18T16:17:28.631 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T16:17:28.642 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/1/new.json 2024-09-18T16:17:28.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAY/epm8cLMBxAA9SXPf5a2MJZjNr2l59QMsA== --osd-uuid 38eaf2fd-9586-44b7-b535-b0aaa388500c 2024-09-18T16:17:28.665 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:17:28.664+0000 7ff424521440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:17:28.667 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:17:28.666+0000 7ff424521440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:17:28.668 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:17:28.667+0000 7ff424521440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:17:28.668 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:17:28.667+0000 7ff424521440 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-18T16:17:28.668 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:17:28.667+0000 7ff424521440 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-18T16:17:28.668 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:17:28.667+0000 7ff424521440 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-18T16:17:28.740 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:17:28.738+0000 7ff424521440 -1 bluestore(td/repro-long-log/1) _read_fsid unparsable uuid 2024-09-18T16:17:34.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/1/keyring 2024-09-18T16:17:34.271 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:17:34.272 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd1 key to auth repository 2024-09-18T16:17:34.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T16:17:34.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:17:34.788 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.1 2024-09-18T16:17:34.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T16:17:34.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:17:34.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:17:34.789 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:17:34.790 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:17:34.810 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:17:34.809+0000 7fb76211f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:17:34.812 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:17:34.811+0000 7fb76211f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:17:34.813 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:17:34.812+0000 7fb76211f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:17:35.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T16:17:35.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:17:35.187 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T16:17:35.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:17:35.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:17:35.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:17:35.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:17:35.188 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:17:35.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:17:35.188 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:17:35.339 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:17:35.338+0000 7fb76211f440 -1 Falling back to public interface 2024-09-18T16:17:35.559 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:17:36.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:17:36.560 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:17:36.560 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T16:17:36.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:17:36.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:17:36.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:17:36.982 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:17:37.804 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:17:37.802+0000 7fb76211f440 -1 osd.1 0 log_to_monitors true 2024-09-18T16:17:37.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:17:37.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:17:37.983 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T16:17:37.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:17:37.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:17:37.984 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:17:38.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:17:39.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:17:39.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:17:39.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:17:39.515 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:17:39.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:17:39.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:17:39.889 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:17:40.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:17:40.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:17:40.890 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T16:17:40.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:17:40.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:17:40.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:17:41.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:17:42.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:17:42.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:17:42.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:17:42.346 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T16:17:42.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:17:42.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:17:42.747 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:17:43.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:17:43.748 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:17:43.748 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T16:17:43.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:17:43.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:17:43.749 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:17:44.136 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:17:45.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:17:45.137 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:17:45.137 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T16:17:45.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:17:45.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:17:45.138 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:17:45.603 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:17:46.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:17:46.604 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:17:46.605 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T16:17:46.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:17:46.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:17:46.605 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:17:47.074 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:17:48.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:17:48.075 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:17:48.075 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T16:17:48.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:17:48.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:17:48.076 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:17:48.581 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:17:49.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:17:49.583 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:17:49.584 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T16:17:49.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:17:49.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:17:49.584 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:17:50.459 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:17:51.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:17:51.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:17:51.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:17:51.461 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T16:17:51.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:17:51.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:17:52.145 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:17:53.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:17:53.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:17:53.147 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:17:53.147 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T16:17:53.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:17:53.148 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:17:53.949 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:17:54.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:17:54.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:17:54.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:17:54.951 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T16:17:54.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:17:54.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:17:55.886 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:17:56.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:17:56.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:17:56.890 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:17:56.890 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T16:17:56.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:17:56.891 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:17:57.531 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:17:58.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:17:58.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:17:58.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:17:58.533 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T16:17:58.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:17:58.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:17:59.343 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:18:00.344 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:18:00.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:18:00.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:18:00.345 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T16:18:00.345 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:18:00.346 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T16:18:00.761 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/532446,v1:127.0.0.1:6811/532446] [v2:127.0.0.1:6812/532446,v1:127.0.0.1:6813/532446] exists,up 38eaf2fd-9586-44b7-b535-b0aaa388500c 2024-09-18T16:18:00.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:18:00.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:18:00.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:18:00.765 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:62: setup_log_test: run_osd td/repro-long-log 2 2024-09-18T16:18:00.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-18T16:18:00.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T16:18:00.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T16:18:00.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T16:18:00.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/2 2024-09-18T16:18:00.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-18T16:18:00.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T16:18:00.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T16:18:00.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T16:18:00.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/2' 2024-09-18T16:18:00.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/2/journal' 2024-09-18T16:18:00.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T16:18:00.766 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T16:18:00.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-18T16:18:00.767 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T16:18:00.767 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T16:18:00.767 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T16:18:00.767 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T16:18:00.767 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:18:00.767 INFO:tasks.workunit.client.0.smithi047.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:18:00.767 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.506073/$cluster-$name.asok' 2024-09-18T16:18:00.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' 2024-09-18T16:18:00.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T16:18:00.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T16:18:00.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T16:18:00.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-18T16:18:00.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-18T16:18:00.767 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T16:18:00.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T16:18:00.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T16:18:00.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T16:18:00.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T16:18:00.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T16:18:00.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/2 2024-09-18T16:18:00.768 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T16:18:00.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1d3a69e4-2c58-4f5a-82bc-efcc20c30ae3 2024-09-18T16:18:00.768 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 1d3a69e4-2c58-4f5a-82bc-efcc20c30ae3' 2024-09-18T16:18:00.768 INFO:tasks.workunit.client.0.smithi047.stdout:add osd2 1d3a69e4-2c58-4f5a-82bc-efcc20c30ae3 2024-09-18T16:18:00.769 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T16:18:00.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA4/epmtUddLhAARpYs7ywJixNvRtf2ZVJKRw== 2024-09-18T16:18:00.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA4/epmtUddLhAARpYs7ywJixNvRtf2ZVJKRw=="}' 2024-09-18T16:18:00.778 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1d3a69e4-2c58-4f5a-82bc-efcc20c30ae3 -i td/repro-long-log/2/new.json 2024-09-18T16:18:01.313 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T16:18:01.324 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/2/new.json 2024-09-18T16:18:01.325 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA4/epmtUddLhAARpYs7ywJixNvRtf2ZVJKRw== --osd-uuid 1d3a69e4-2c58-4f5a-82bc-efcc20c30ae3 2024-09-18T16:18:01.348 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:18:01.346+0000 7f30c062e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:18:01.349 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:18:01.349+0000 7f30c062e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:18:01.350 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:18:01.350+0000 7f30c062e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:18:01.351 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:18:01.350+0000 7f30c062e440 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-18T16:18:01.351 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:18:01.350+0000 7f30c062e440 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-18T16:18:01.351 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:18:01.350+0000 7f30c062e440 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-18T16:18:01.420 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:18:01.419+0000 7f30c062e440 -1 bluestore(td/repro-long-log/2) _read_fsid unparsable uuid 2024-09-18T16:18:07.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/2/keyring 2024-09-18T16:18:07.008 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T16:18:07.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T16:18:07.009 INFO:tasks.workunit.client.0.smithi047.stdout:adding osd2 key to auth repository 2024-09-18T16:18:07.009 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T16:18:07.514 INFO:tasks.workunit.client.0.smithi047.stdout:start osd.2 2024-09-18T16:18:07.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T16:18:07.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=878df9c3-e8a8-4c44-908d-40b3c182b1f9 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.506073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T16:18:07.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T16:18:07.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T16:18:07.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T16:18:07.535 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:18:07.534+0000 7fa2f108b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:18:07.537 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:18:07.536+0000 7fa2f108b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:18:07.538 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:18:07.537+0000 7fa2f108b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T16:18:07.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T16:18:07.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T16:18:07.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T16:18:07.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T16:18:07.878 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T16:18:07.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:18:07.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T16:18:07.879 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:18:07.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:18:07.879 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:18:08.058 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:18:08.057+0000 7fa2f108b440 -1 Falling back to public interface 2024-09-18T16:18:08.244 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:18:09.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:18:09.245 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:18:09.245 INFO:tasks.workunit.client.0.smithi047.stdout:1 2024-09-18T16:18:09.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T16:18:09.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:18:09.246 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:18:09.651 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:18:10.622 INFO:tasks.workunit.client.0.smithi047.stderr:2024-09-18T16:18:10.621+0000 7fa2f108b440 -1 osd.2 0 log_to_monitors true 2024-09-18T16:18:10.652 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:18:10.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:18:10.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T16:18:10.653 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T16:18:10.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:18:10.653 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:18:11.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:18:12.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:18:12.124 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:18:12.124 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:18:12.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T16:18:12.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:18:12.125 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:18:12.612 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:18:13.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:18:13.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:18:13.614 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T16:18:13.614 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T16:18:13.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:18:13.615 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:18:13.989 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:18:14.991 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:18:14.991 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:18:14.991 INFO:tasks.workunit.client.0.smithi047.stdout:5 2024-09-18T16:18:14.992 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T16:18:14.992 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:18:14.992 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:18:15.458 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:18:16.460 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:18:16.460 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T16:18:16.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:18:16.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T16:18:16.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:18:16.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:18:16.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:18:17.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:18:17.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:18:17.923 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T16:18:17.924 INFO:tasks.workunit.client.0.smithi047.stdout:7 2024-09-18T16:18:17.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:18:17.924 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:18:18.515 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:18:19.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:18:19.516 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:18:19.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T16:18:19.517 INFO:tasks.workunit.client.0.smithi047.stdout:8 2024-09-18T16:18:19.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:18:19.517 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:18:19.895 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:18:20.896 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:18:20.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:18:20.897 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T16:18:20.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T16:18:20.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:18:20.897 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:18:21.270 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:18:22.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:18:22.272 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:18:22.272 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T16:18:22.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T16:18:22.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:18:22.273 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:18:23.048 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:18:24.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:18:24.049 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:18:24.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T16:18:24.050 INFO:tasks.workunit.client.0.smithi047.stdout:11 2024-09-18T16:18:24.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:18:24.050 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:18:24.740 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:18:25.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:18:25.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:18:25.742 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T16:18:25.742 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T16:18:25.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:18:25.743 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:18:26.416 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:18:27.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:18:27.417 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:18:27.418 INFO:tasks.workunit.client.0.smithi047.stdout:13 2024-09-18T16:18:27.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T16:18:27.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:18:27.418 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:18:28.116 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:18:29.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:18:29.117 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:18:29.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T16:18:29.118 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T16:18:29.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:18:29.118 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:18:29.727 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:18:30.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:18:30.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:18:30.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T16:18:30.728 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T16:18:30.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:18:30.729 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:18:31.543 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T16:18:32.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T16:18:32.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T16:18:32.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T16:18:32.545 INFO:tasks.workunit.client.0.smithi047.stdout:16 2024-09-18T16:18:32.545 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T16:18:32.546 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T16:18:32.950 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/534175,v1:127.0.0.1:6819/534175] [v2:127.0.0.1:6820/534175,v1:127.0.0.1:6821/534175] exists,up 1d3a69e4-2c58-4f5a-82bc-efcc20c30ae3 2024-09-18T16:18:32.950 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T16:18:32.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T16:18:32.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T16:18:32.951 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:64: setup_log_test: ceph osd pool create test 1 1 2024-09-18T16:18:33.492 INFO:tasks.workunit.client.0.smithi047.stderr:pool 'test' already exists 2024-09-18T16:18:33.503 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: ceph osd dump --format json 2024-09-18T16:18:33.504 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: jq '.pools[] | select(.pool_name == "test") | .pool' 2024-09-18T16:18:33.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: POOL_ID=1 2024-09-18T16:18:33.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:66: setup_log_test: PGID=1.0 2024-09-18T16:18:33.940 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:69: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 20 2024-09-18T16:18:34.066 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: {} 2024-09-18T16:18:34.093 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: {} 2024-09-18T16:18:34.118 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: {} 2024-09-18T16:18:34.127 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:70: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 20 2024-09-18T16:18:34.242 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: {} 2024-09-18T16:18:34.269 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: {} 2024-09-18T16:18:34.293 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: {} 2024-09-18T16:18:34.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:71: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-max-pg-log-entries 30 2024-09-18T16:18:34.403 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: {} 2024-09-18T16:18:34.429 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: {} 2024-09-18T16:18:34.454 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: {} 2024-09-18T16:18:34.463 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:72: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 10 2024-09-18T16:18:34.582 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: {} 2024-09-18T16:18:34.608 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: {} 2024-09-18T16:18:34.634 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: {} 2024-09-18T16:18:34.643 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:73: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 20 2024-09-18T16:18:34.917 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: {} 2024-09-18T16:18:34.943 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: {} 2024-09-18T16:18:34.969 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: {} 2024-09-18T16:18:34.978 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:75: setup_log_test: touch td/repro-long-log/foo 2024-09-18T16:18:34.979 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: seq 1 20 2024-09-18T16:18:34.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:34.980 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:18:35.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:35.191 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:18:35.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:35.368 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:18:35.476 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:35.476 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:18:35.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:35.579 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:18:35.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:35.671 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:18:35.780 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:35.780 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:18:35.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:35.864 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:18:35.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:35.952 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:18:36.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:36.065 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:18:36.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:36.166 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:18:36.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:36.283 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:18:36.391 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:36.392 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:18:36.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:36.534 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:18:36.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:36.635 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:18:36.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:36.744 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:18:36.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:36.836 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:18:36.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:36.928 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:18:37.020 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:37.021 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:18:37.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:37.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-18T16:18:37.221 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:81: setup_log_test: test_log_size 1.0 20 2024-09-18T16:18:37.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-18T16:18:37.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=20 2024-09-18T16:18:37.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-18T16:18:37.222 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-18T16:18:37.319 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: 21474836501 2024-09-18T16:18:37.341 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: 42949672973 2024-09-18T16:18:37.363 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: 64424509445 2024-09-18T16:18:37.372 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-18T16:18:40.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-18T16:18:40.374 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-18T16:18:40.497 INFO:tasks.workunit.client.0.smithi047.stdout:20 2024-09-18T16:18:40.497 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-18T16:18:40.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-18T16:18:40.498 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 20 2024-09-18T16:18:40.629 INFO:tasks.workunit.client.0.smithi047.stdout:20 2024-09-18T16:18:40.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-18T16:18:40.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:18:40.740 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:18:40.741 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-18T16:18:40.741 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:18:40.741 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-18T16:18:40.851 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:18:40.852 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:83: setup_log_test: rados -p test rm foo 2024-09-18T16:18:40.966 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: seq 1 20 2024-09-18T16:18:40.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:40.967 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:18:41.065 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:18:41.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:41.066 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:18:41.182 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:18:41.183 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:41.184 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:18:41.266 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:18:41.267 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:41.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:18:41.359 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:18:41.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:41.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:18:41.443 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:18:41.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:41.444 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:18:41.535 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:18:41.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:41.537 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:18:41.628 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:18:41.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:41.630 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:18:41.737 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:18:41.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:41.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:18:41.855 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:18:41.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:41.857 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:18:41.964 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:18:41.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:41.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:18:42.056 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:18:42.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:42.058 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:18:42.140 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:18:42.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:42.142 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:18:42.266 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:18:42.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:42.268 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:18:42.375 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:18:42.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:42.377 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:18:42.459 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:18:42.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:42.461 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:18:42.560 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:18:42.561 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:42.562 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:18:42.719 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:18:42.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:42.721 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:18:42.828 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:18:42.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:42.830 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:18:42.912 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:18:42.913 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-18T16:18:42.914 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-18T16:18:43.038 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:18:43.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:92: setup_log_test: test_log_size 1.0 21 2024-09-18T16:18:43.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-18T16:18:43.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=21 2024-09-18T16:18:43.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-18T16:18:43.040 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-18T16:18:43.139 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: 21474836505 2024-09-18T16:18:43.161 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: 42949672976 2024-09-18T16:18:43.184 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: 64424509449 2024-09-18T16:18:43.192 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-18T16:18:46.194 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-18T16:18:46.194 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-18T16:18:46.317 INFO:tasks.workunit.client.0.smithi047.stdout:21 2024-09-18T16:18:46.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-18T16:18:46.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-18T16:18:46.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 21 2024-09-18T16:18:46.429 INFO:tasks.workunit.client.0.smithi047.stdout:21 2024-09-18T16:18:46.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-18T16:18:46.430 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:18:46.553 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:18:46.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-18T16:18:46.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:18:46.553 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-18T16:18:46.663 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-09-18T16:18:46.664 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:161: TEST_trim_max_entries_with_dups: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 2 2024-09-18T16:18:46.764 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: {} 2024-09-18T16:18:46.790 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: {} 2024-09-18T16:18:46.815 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: {} 2024-09-18T16:18:46.824 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:162: TEST_trim_max_entries_with_dups: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 2 2024-09-18T16:18:46.924 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: {} 2024-09-18T16:18:46.950 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: {} 2024-09-18T16:18:46.975 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: {} 2024-09-18T16:18:46.983 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:163: TEST_trim_max_entries_with_dups: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 2 2024-09-18T16:18:47.083 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: {} 2024-09-18T16:18:47.109 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: {} 2024-09-18T16:18:47.134 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: {} 2024-09-18T16:18:47.143 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:164: TEST_trim_max_entries_with_dups: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-max 4 2024-09-18T16:18:47.243 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: {} 2024-09-18T16:18:47.268 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: {} 2024-09-18T16:18:47.294 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: {} 2024-09-18T16:18:47.302 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:165: TEST_trim_max_entries_with_dups: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 20 2024-09-18T16:18:47.403 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: {} 2024-09-18T16:18:47.428 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: {} 2024-09-18T16:18:47.454 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: {} 2024-09-18T16:18:47.462 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:169: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-09-18T16:18:47.585 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:18:47.587 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:170: TEST_trim_max_entries_with_dups: test_log_size 1.0 18 2 2024-09-18T16:18:47.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-18T16:18:47.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=18 2024-09-18T16:18:47.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=2 2024-09-18T16:18:47.588 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-18T16:18:47.684 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: 21474836507 2024-09-18T16:18:47.707 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: 42949672979 2024-09-18T16:18:47.728 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: 64424509451 2024-09-18T16:18:47.737 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-18T16:18:50.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-18T16:18:50.739 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-18T16:18:50.861 INFO:tasks.workunit.client.0.smithi047.stdout:18 2024-09-18T16:18:50.862 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-18T16:18:50.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-18T16:18:50.863 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 18 2024-09-18T16:18:50.974 INFO:tasks.workunit.client.0.smithi047.stdout:18 2024-09-18T16:18:50.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-18T16:18:50.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:18:51.085 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T16:18:51.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-18T16:18:51.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:18:51.086 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 2 2024-09-18T16:18:51.196 INFO:tasks.workunit.client.0.smithi047.stdout:2 2024-09-18T16:18:51.196 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:171: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-09-18T16:18:51.313 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:18:51.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:172: TEST_trim_max_entries_with_dups: test_log_size 1.0 15 6 2024-09-18T16:18:51.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-18T16:18:51.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=15 2024-09-18T16:18:51.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=6 2024-09-18T16:18:51.315 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-18T16:18:51.412 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: 21474836510 2024-09-18T16:18:51.435 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: 42949672982 2024-09-18T16:18:51.457 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: 64424509454 2024-09-18T16:18:51.465 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-18T16:18:54.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-18T16:18:54.467 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-18T16:18:54.623 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T16:18:54.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-18T16:18:54.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-18T16:18:54.624 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 15 2024-09-18T16:18:54.736 INFO:tasks.workunit.client.0.smithi047.stdout:15 2024-09-18T16:18:54.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-18T16:18:54.736 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:18:54.849 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T16:18:54.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-18T16:18:54.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:18:54.849 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 6 2024-09-18T16:18:54.957 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T16:18:54.957 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:173: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-09-18T16:18:55.082 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:18:55.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:174: TEST_trim_max_entries_with_dups: test_log_size 1.0 12 10 2024-09-18T16:18:55.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-18T16:18:55.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=12 2024-09-18T16:18:55.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=10 2024-09-18T16:18:55.085 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-18T16:18:55.181 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: 21474836513 2024-09-18T16:18:55.204 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: 42949672984 2024-09-18T16:18:55.225 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: 64424509457 2024-09-18T16:18:55.234 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-18T16:18:58.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-18T16:18:58.236 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-18T16:18:58.359 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T16:18:58.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-18T16:18:58.359 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-18T16:18:58.360 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 12 2024-09-18T16:18:58.470 INFO:tasks.workunit.client.0.smithi047.stdout:12 2024-09-18T16:18:58.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-18T16:18:58.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:18:58.617 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T16:18:58.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-18T16:18:58.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:18:58.617 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 10 2024-09-18T16:18:58.728 INFO:tasks.workunit.client.0.smithi047.stdout:10 2024-09-18T16:18:58.728 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:175: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-09-18T16:18:58.843 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:18:58.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:176: TEST_trim_max_entries_with_dups: test_log_size 1.0 9 14 2024-09-18T16:18:58.845 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-18T16:18:58.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=9 2024-09-18T16:18:58.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=14 2024-09-18T16:18:58.846 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-18T16:18:58.944 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: 21474836516 2024-09-18T16:18:58.966 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: 42949672987 2024-09-18T16:18:58.988 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: 64424509460 2024-09-18T16:18:58.997 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-18T16:19:01.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-18T16:19:01.999 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-18T16:19:02.122 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T16:19:02.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-18T16:19:02.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-18T16:19:02.123 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 9 2024-09-18T16:19:02.260 INFO:tasks.workunit.client.0.smithi047.stdout:9 2024-09-18T16:19:02.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-18T16:19:02.260 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:19:02.371 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T16:19:02.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-18T16:19:02.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:19:02.371 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 14 2024-09-18T16:19:02.482 INFO:tasks.workunit.client.0.smithi047.stdout:14 2024-09-18T16:19:02.483 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:177: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-09-18T16:19:02.646 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:19:02.648 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:178: TEST_trim_max_entries_with_dups: test_log_size 1.0 6 18 2024-09-18T16:19:02.648 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-18T16:19:02.648 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=6 2024-09-18T16:19:02.648 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=18 2024-09-18T16:19:02.648 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-18T16:19:02.746 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: 21474836518 2024-09-18T16:19:02.769 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: 42949672990 2024-09-18T16:19:02.792 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: 64424509462 2024-09-18T16:19:02.800 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-18T16:19:05.802 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-18T16:19:05.803 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-18T16:19:05.926 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T16:19:05.926 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-18T16:19:05.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-18T16:19:05.927 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 6 2024-09-18T16:19:06.038 INFO:tasks.workunit.client.0.smithi047.stdout:6 2024-09-18T16:19:06.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-18T16:19:06.039 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:19:06.150 INFO:tasks.workunit.client.0.smithi047.stdout:18 2024-09-18T16:19:06.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-18T16:19:06.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:19:06.151 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 18 2024-09-18T16:19:06.262 INFO:tasks.workunit.client.0.smithi047.stdout:18 2024-09-18T16:19:06.262 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:179: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-09-18T16:19:06.381 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:19:06.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:180: TEST_trim_max_entries_with_dups: test_log_size 1.0 3 20 2024-09-18T16:19:06.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-18T16:19:06.383 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-09-18T16:19:06.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=20 2024-09-18T16:19:06.384 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-18T16:19:06.480 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: 21474836521 2024-09-18T16:19:06.502 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: 42949672993 2024-09-18T16:19:06.524 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: 64424509465 2024-09-18T16:19:06.533 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-18T16:19:09.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-18T16:19:09.535 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-18T16:19:09.657 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:19:09.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-18T16:19:09.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-18T16:19:09.658 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-09-18T16:19:09.770 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:19:09.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-18T16:19:09.770 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:19:09.880 INFO:tasks.workunit.client.0.smithi047.stdout:20 2024-09-18T16:19:09.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-18T16:19:09.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:19:09.881 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 20 2024-09-18T16:19:09.990 INFO:tasks.workunit.client.0.smithi047.stdout:20 2024-09-18T16:19:09.991 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:183: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-09-18T16:19:10.100 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:19:10.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:184: TEST_trim_max_entries_with_dups: test_log_size 1.0 4 20 2024-09-18T16:19:10.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-18T16:19:10.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=4 2024-09-18T16:19:10.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=20 2024-09-18T16:19:10.102 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-18T16:19:10.200 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: 21474836524 2024-09-18T16:19:10.222 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: 42949672995 2024-09-18T16:19:10.244 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: 64424509468 2024-09-18T16:19:10.253 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-18T16:19:13.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-18T16:19:13.255 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-18T16:19:13.377 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T16:19:13.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-18T16:19:13.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-18T16:19:13.378 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 4 2024-09-18T16:19:13.489 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T16:19:13.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-18T16:19:13.489 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:19:13.599 INFO:tasks.workunit.client.0.smithi047.stdout:20 2024-09-18T16:19:13.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-18T16:19:13.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:19:13.600 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 20 2024-09-18T16:19:13.710 INFO:tasks.workunit.client.0.smithi047.stdout:20 2024-09-18T16:19:13.710 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:185: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-09-18T16:19:13.811 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:19:13.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:186: TEST_trim_max_entries_with_dups: test_log_size 1.0 3 20 2024-09-18T16:19:13.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-18T16:19:13.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-09-18T16:19:13.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=20 2024-09-18T16:19:13.813 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-18T16:19:13.910 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: 21474836527 2024-09-18T16:19:13.933 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: 42949672998 2024-09-18T16:19:13.955 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: 64424509471 2024-09-18T16:19:13.963 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-18T16:19:16.965 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-18T16:19:16.966 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-18T16:19:17.087 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:19:17.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-18T16:19:17.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-18T16:19:17.088 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-09-18T16:19:17.199 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:19:17.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-18T16:19:17.200 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:19:17.310 INFO:tasks.workunit.client.0.smithi047.stdout:20 2024-09-18T16:19:17.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-18T16:19:17.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:19:17.311 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 20 2024-09-18T16:19:17.421 INFO:tasks.workunit.client.0.smithi047.stdout:20 2024-09-18T16:19:17.421 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:187: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-09-18T16:19:17.530 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:19:17.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:188: TEST_trim_max_entries_with_dups: test_log_size 1.0 4 20 2024-09-18T16:19:17.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-18T16:19:17.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=4 2024-09-18T16:19:17.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=20 2024-09-18T16:19:17.532 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-18T16:19:17.629 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: 21474836529 2024-09-18T16:19:17.652 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: 42949673001 2024-09-18T16:19:17.674 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: 64424509473 2024-09-18T16:19:17.682 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-18T16:19:20.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-18T16:19:20.684 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-18T16:19:20.809 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T16:19:20.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-18T16:19:20.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-18T16:19:20.809 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 4 2024-09-18T16:19:20.921 INFO:tasks.workunit.client.0.smithi047.stdout:4 2024-09-18T16:19:20.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-18T16:19:20.922 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:19:21.032 INFO:tasks.workunit.client.0.smithi047.stdout:20 2024-09-18T16:19:21.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-18T16:19:21.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:19:21.033 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 20 2024-09-18T16:19:21.144 INFO:tasks.workunit.client.0.smithi047.stdout:20 2024-09-18T16:19:21.144 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:189: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-09-18T16:19:21.316 INFO:tasks.workunit.client.0.smithi047.stderr:error removing test>foo: (2) No such file or directory 2024-09-18T16:19:21.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:190: TEST_trim_max_entries_with_dups: test_log_size 1.0 3 20 2024-09-18T16:19:21.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-18T16:19:21.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-09-18T16:19:21.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=20 2024-09-18T16:19:21.318 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-18T16:19:21.416 INFO:tasks.workunit.client.0.smithi047.stdout:osd.0: 21474836532 2024-09-18T16:19:21.438 INFO:tasks.workunit.client.0.smithi047.stdout:osd.1: 42949673004 2024-09-18T16:19:21.460 INFO:tasks.workunit.client.0.smithi047.stdout:osd.2: 64424509476 2024-09-18T16:19:21.469 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-18T16:19:24.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-18T16:19:24.471 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-18T16:19:24.636 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:19:24.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-18T16:19:24.636 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-18T16:19:24.637 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-09-18T16:19:24.750 INFO:tasks.workunit.client.0.smithi047.stdout:3 2024-09-18T16:19:24.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-18T16:19:24.750 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:19:24.861 INFO:tasks.workunit.client.0.smithi047.stdout:20 2024-09-18T16:19:24.862 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-18T16:19:24.862 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-18T16:19:24.862 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 20 2024-09-18T16:19:24.972 INFO:tasks.workunit.client.0.smithi047.stdout:20 2024-09-18T16:19:24.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:35: run: teardown td/repro-long-log 2024-09-18T16:19:24.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-09-18T16:19:24.973 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T16:19:24.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-09-18T16:19:24.974 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:19:24.974 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:19:24.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:19:24.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:19:24.974 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:19:25.081 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:19:25.081 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:19:25.082 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:19:25.082 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:19:25.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:19:25.083 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:19:25.084 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:19:25.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:19:25.084 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:19:25.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:19:25.089 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:19:25.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:19:25.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T16:19:25.089 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-09-18T16:19:25.103 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:19:25.103 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:19:25.103 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:19:25.103 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.506073 2024-09-18T16:19:25.104 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:19:25.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:19:25.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T16:19:25.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/repro-long-log 0 2024-09-18T16:19:25.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-09-18T16:19:25.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T16:19:25.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-09-18T16:19:25.105 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T16:19:25.105 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T16:19:25.105 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T16:19:25.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T16:19:25.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T16:19:25.106 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T16:19:25.106 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T16:19:25.107 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T16:19:25.107 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T16:19:25.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T16:19:25.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T16:19:25.109 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T16:19:25.109 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:19:25.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T16:19:25.110 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T16:19:25.110 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T16:19:25.111 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:19:25.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T16:19:25.112 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-09-18T16:19:25.113 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T16:19:25.113 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T16:19:25.113 INFO:tasks.workunit.client.0.smithi047.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.506073 2024-09-18T16:19:25.113 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.506073 2024-09-18T16:19:25.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T16:19:25.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T16:19:25.115 INFO:tasks.workunit.client.0.smithi047.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T16:19:25.115 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T16:19:25.116 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T16:19:25.141 INFO:tasks.workunit:Stopping ['osd'] on client.0... 2024-09-18T16:19:25.141 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-18T16:19:25.468 DEBUG:teuthology.parallel:result is None 2024-09-18T16:19:25.469 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-18T16:19:25.494 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-18T16:19:25.495 DEBUG:teuthology.orchestra.run.smithi047:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-18T16:19:25.597 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-18T16:19:25.598 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-18T16:19:25.609 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-18T16:19:25.610 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-18T16:19:25.661 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-18T16:19:25.669 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-18T16:19:25.669 DEBUG:teuthology.orchestra.run.smithi047:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T16:19:25.690 INFO:teuthology.orchestra.run.smithi047.stderr:bash: line 1: ntpq: command not found 2024-09-18T16:19:25.694 INFO:teuthology.orchestra.run.smithi047.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-18T16:19:25.694 INFO:teuthology.orchestra.run.smithi047.stdout:=============================================================================== 2024-09-18T16:19:25.694 INFO:teuthology.orchestra.run.smithi047.stdout:^* hv01.front.sepia.ceph.com 3 7 377 94 +265us[ +262us] +/- 58ms 2024-09-18T16:19:25.694 INFO:teuthology.orchestra.run.smithi047.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 23 -897us[ -897us] +/- 60ms 2024-09-18T16:19:25.694 INFO:teuthology.orchestra.run.smithi047.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 27 +797us[ +797us] +/- 68ms 2024-09-18T16:19:25.694 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T16:19:25.695 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-18T16:19:25.703 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-18T16:19:25.704 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-18T16:19:25.729 DEBUG:teuthology.orchestra.run.smithi047:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-18T16:19:25.795 DEBUG:teuthology.orchestra.run.smithi047:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-18T16:19:25.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T16:19:25.823 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-18T16:19:25.831 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-18T16:19:25.854 INFO:teuthology.task.internal:Duration was 9116.075431 seconds 2024-09-18T16:19:25.855 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-18T16:19:25.897 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-18T16:19:25.897 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-18T16:19:25.938 INFO:teuthology.orchestra.run.smithi047.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-18T16:19:26.271 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-18T16:19:26.271 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi047.front.sepia.ceph.com 2024-09-18T16:19:26.271 DEBUG:teuthology.orchestra.run.smithi047:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-18T16:19:26.336 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-18T16:19:26.336 DEBUG:teuthology.orchestra.run.smithi047:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-18T16:19:26.403 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-18T16:19:26.403 DEBUG:teuthology.orchestra.run.smithi047:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-18T16:19:26.553 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-18T16:19:26.563 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-18T16:19:26.563 DEBUG:teuthology.orchestra.run.smithi047:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-18T16:19:26.589 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-18T16:19:26.605 DEBUG:teuthology.orchestra.run.smithi047:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:19:26.653 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern = core 2024-09-18T16:19:26.666 DEBUG:teuthology.orchestra.run.smithi047:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-18T16:19:26.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T16:19:26.722 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-18T16:19:26.731 INFO:teuthology.task.internal:Transferring archived files... 2024-09-18T16:19:26.734 DEBUG:teuthology.misc:Transferring archived files from smithi047:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911065/remote/smithi047 2024-09-18T16:19:26.735 DEBUG:teuthology.orchestra.run.smithi047:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-18T16:19:26.808 INFO:teuthology.task.internal:Removing archive directory... 2024-09-18T16:19:26.808 DEBUG:teuthology.orchestra.run.smithi047:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-18T16:19:26.857 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-18T16:19:26.868 INFO:teuthology.task.internal:Not uploading archives. 2024-09-18T16:19:26.868 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-18T16:19:26.897 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-18T16:19:26.897 DEBUG:teuthology.orchestra.run.smithi047:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-18T16:19:26.913 INFO:teuthology.orchestra.run.smithi047.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 18 16:19 /home/ubuntu/cephtest 2024-09-18T16:19:26.914 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-18T16:19:26.938 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-18T16:19:27.016 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd} duration: 9116.075431108475 flavor: default owner: scheduled_skanta@teuthology success: true 2024-09-18T16:19:27.016 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-18T16:19:27.088 INFO:teuthology.run:pass